9.4M views · Jan 4, 2016 · 22:58 · 1 HN recommender · 4 mentions · discussed 2019–2021
Linus Tech Tips documents a major RAID failure on its Whonnock server and the attempt to recover the lost data with a data recovery firm.
Why Hacker News points people here: HN refers to it as an honest, candid account of the common story of losing precious data to storage failure.
“Actually a pretty good telling of a story all too common for those tasked with handling precious data.”HN comment, 2021
“Actually a pretty good telling of a story all too common for those tasked with handling precious data.”
Telling of a common story of losing precious data
Video for those interested: this video Actually a pretty good telling of a story all too common for those tasked with handling precious data.
Linus Tech Tips data loss story
I really like the idea behind storing error-correction codes at the application level. Disks should just store and retrieve a chunk of bits and it should be pushed up to the application level to decide whether or not those bits were what we wrote. From there, we can make the right decisions about how durable we really want a file to be. Instead, people seem to have invested a lot of effort in going the other direction; let the application have less control and let many magic layers in the middle figure out the details. We have had filesystems for decades, which pretend to map names to a list of durable block numbers. Of course, the blocks are not so durable (nor is the database), so people lose their data all the time and act like it's normal. People have a solution to that problem, though, make the block devices more durable! We can just add software or hardware RAID on top of a bunch of disks, and now suddenly what the filesystem thinks is just one block is now 3 blocks sharded across 3 disks. Some magical curator process then tells me when disks have failed (as though it can detect that with 100% reliability), and I pop in a new drive and magically everything has been repaired. Of course, in reality, the entire disk doesn't just disappear all at once; some blocks just return data with a flipped bit, or all 0s, or an occasional error. My experience in running RAID-1 arrays under Linux (for my workstation) is that I notice some errors in the logs and have to manually figure
“That video is technically hilarious though”
Story of a RAID failure and recovery through a sketchy company
I'm not sure if you're referring to this video ("All of our data is gone"), but in any case, I've heard references to ZFS in later videos so yeah, the situation's probably not quite that insane now. That video is technically hilarious though, it TL;DRs as "download Ubuntu-based USB image provided random RAID recovery company, boot USB and provide said company headless SSH to broken system"... at one point in the video I remember Linus talks to another recovery company about the process and the guy pretty much went "...O.o..." which IMO sums my own opinion up, lol
“seemingly quite genuine fly-on-the-wall coverage of temporarily losing their data to a RAID failure”
Example of the channel openly covering its own RAID failure
Sebastian and Linus Media Group seem to be pretty frank and open about a lot of this: see for instance link on putting up gurning thumbnails and navigating the YT algorithm (and that whole playlist, including a revenue breakdown link ), the Gamers Nexus video on LTT's production process link and the coverage LTT regularly lavishes on its own production setup and process, from a nearly-12m video about the awkwardness of ingesting video from their new 12K cameras link to seemingly quite genuine fly-on-the-wall coverage of temporarily losing their data to a RAID failure this video . Obviously you always have to take anyone's self-presentation with a grain of salt. But they're certainly not hiding the fact that there's a more grounded reality behind the "on" personas.
