Meta quickly detects silent data corruptions at scale
engineering.fb.com
engineering.fb.com
(Work at Meta, mostly on capacity)
You may have posted a photo 7 years ago, and statistics show that basically nobody ever revisits it. However, in case you do, it needs to be there. So these enormous buildings do basically nothing, but still need to be there.
It makes me wonder how it can go on like this. Users only keep adding content and never remove it. The income per user cannot grow forever, storage cannot get infinitely cheap, the model has to break one day?
Or maybe I'm underestimating how much space newer material needs?
And again I have been saying this since ~2015/16, we dont have any meaningful roadmap for cost reduction on storage, whether that is Hot as in NAND, Bulk as in HDD, or Cold as in Optical Disc. I dont see 2TB SSD dropping below $100 in next 5 years, or 10TB HDD below $120.
Remember when Google promise infinite Gmail storage?
Then maybe we could:
- stop encouraging users to post shit just so we can track them
- stop tracking them which requires many data points and a lot of processing power (for 0 benefit for the user or society at large)
- stop the copyright non-sense and actually use hyperlinks instead of reuploading the same content 500 times across platforms? maybe even do content-addressed storage (Bittorrent/IPFS) who knows?
So if you're designing capacity for exponential growth, the future point at which you stop experiencing exponential growth and only have to worry about roughly linear growth is a much easier problem to solve.
But the painful ones are the 'subtle' failures. Why does machine PABL12 sometimes give NaN as a result while all 99,999 machines return sensible numbers? But all the burn in hardware tests pass...
The solution was to simply exclude any machines that were outliers. Anything in the top or bottom 0.01% for any metric simply exclude that machine from future workloads.
Sure, in most cases there was nothing wrong with the hardware, but when you're spending hours debugging some fault caused by a sometimes-bad floating point unit on one core of one machine out of 100,000, you're just wasting your time. By auto-banning outliers, the machine will end up doing some other task where data consistency matters less.
It was an annoying struggle trying to raise the visibility of broken CPUs during my years at Google SRE. The SRE org and the rest of the software side of Tech Infra resisted the whole concept, even though it was well-known among platforms hardware eng. The process for taking a known-bad machine out of service involved 1) the machine being reported independently by three different teams; 2) the machine continuing to be in service for days or weeks, at the leisure of some very asynchronous automation; and 3) the machine being returned immediately to service because it passed all of the cursory checks during reinstall. Really irritating. Consequently every major service had to maintain their own private blacklist.
It's nice to see that some influential people on the software side are starting to come around, with papers like "Cores That Don't Count" etc, but man they could have been on this boat a decade ago.
If one could show that the CPU said 2+2=9, I'm sure they would yank it out right away, but "it returns 500 errors a lot" isn't very debugable. The only thing they can do is run the diag and return it to service if nothing comes up.
And your last statement is definitely not true. I can recall multiple instances of demonstrable logic errors in which the machine repeatedly returned to service. This includes all of the machines of a certain generation of a certain vendor's CPUs that were found to have latent ALU bugs, 8 years after going into service.
At that scale, it's quite likely sent to repair automatically and whoever's on call just gets a notification.
https://blog.cloudflare.com/however-improbable-the-story-of-...
> So Google found fail-silent Corruption Execution Errors (CEEs) at CPU/cores. This is interesting because we thought tested CPUs do not have logic errors, and if they had an error it would be a fail-stop or at least fail-noisy hardware errors triggering machine checks. Previously we had known about fail-silent storage and network errors due to bit flips, but the CEEs are new because they are computation errors. While it is easy to detect data corruption due to bit flips, it is hard to detect CEEs because they are rare and require expensive methods to detect/correct in real-time.
https://muratbuffalo.blogspot.com/2021/06/silent-data-corrup...
> The paper claims that silent data corruptions can occur due to device characteristics and are repeatable at scale. They observed that these failures are reproducible and not transient. Then, how come did these CPUs pass the quality control tests by the chip producers? In soft-error based fault injection studies by chip producers, CPU CEEs are evaluated to be a one in a million occurrence, not 1 in 1000 observed at deployment at Facebook and Google... The paper also says that increased density, technology scaling, and wider datapaths increase the probability of silent errors.
That is, the so-called "bitrot" phenomenon is largely mis-attributed. Bitrot doesn't happen at rest. It happens in transit.
The only correct way to test for bitrot is to read the data back immediately after it was written and the cache flushed. If it's the same as the original, we know it made it to the disk undamaged. Then re-read it again after some time. If it doesn't match, re-read immediately again, ideally using a different physical memory block. Compare again. If it doesn't match, take the disk to another machine and re-read again. If it doesn't match, only then it's an actual at-rest bitrot... OR it's a drive's firmware bug, because corrupted data must be corrected or it must not be returned at all.
Because we had the checks for it in flight. Also, more often than not these same blocks had been checked before, and found to be fine.
> The only correct way to test for bitrot is to read the data back immediately
No, the only correct way is to read it back after some time has passed. Mis-written data is not the same as bitrot.
> must be corrected or it must not be returned
Every error-correction technique has a limit to how many simultaneous errors it can correct. Beyond that, bits can be flipped in a way that seems valid but in fact is not (detectable by cross-checking with other erasure-coded fragments of the same block on other machines). Just because you haven't seen it doesn't mean it doesn't happen. As I said, and as others have said many times, with sufficient scale and time even the most unlikely scenarios become almost inevitable. Why do you persist in telling me I didn't see what I saw with my own eyes? Are you assuming that my thirty years in storage gave me less understanding or insight regarding these issues than whatever experience (if any) you have?
> No, the only correct way is to read it back after some time has passed. Mis-written data is not the same as bitrot.
Well, no. If you want to check for at-rest bitrot, you need to make sure that you've written out the correct thing. Otherwise it's not possible to tell at-rest corruption from the one that happened on the way in.
> Every error-correction technique has a limit to how many simultaneous errors it can correct.
But it can detect that the case when it can't recover. Which is why it will either produce a correct output or an error.
> As I said, and as others have said many times, with sufficient scale and time even the most unlikely scenarios become almost inevitable.
This is not an argument if it goes against how things actually work.
> Why do you persist in telling me I didn't see what I saw with my own eyes?
I am merely curious in your exact testing technique, because at-rest bitrot is vanishingly impossible, even at the exabyte scale. For it to happen, the data and its ECC (7-11% of the data size) need to be both corrupted in a coordinated way. That is exceedingly unlikely. Especially in the context of academic papers that found that on-disk corruption is nearly always clustered and is either small scale or full-sector failures.
So when you say you ran into a lot of these cases, it's only natural to ask for details. And "scale" is not a detail.
> Are you assuming that my thirty years in storage gave me less understanding or insight regarding these issues than whatever experience (if any) you have?
I have no way to tell. But given your experience, can you explain how at-rest bitrot, should it occur, can seep through the on-disk error correction? I am not talking about raid-style setups, just the banal ECC record in a disk sector [1].
[1] https://en.wikipedia.org/wiki/Advanced_Format#Overview (linking to Advanced Format, because it has a diagram)
The paper "Parity Lost and Parity Regained" assigns a probability of 1.88e−5 to misdirected writes bugs among disks, so if you have a warehouse full of disks you now have this nightmare.
That is simply not true. For any parity/ECC/FEC/erasure-code scheme carrying M data bits in N (greater than M but less than 2M) total, there must be multiple data patterns that will match the same error checks. That's just mathematics. Also, bear in mind that ECC bits can be corrupted too. This opens up the distinct possibility of something that looks like a correctable error, but the "correction" leads to a wrong result. I've seen such issues in many kinds of storage systems, from low level to high. Anyone who has actually worked in this area, instead of deriving their "expertise" from a quick scan of Wikipedia, would be utterly unsurprised by the idea that disk firmware might do such a thing, or have bugs in their ECC implementation, or not follow a spec.
Whatever the causes, whatever the merely-theoretical probabilities, the fact remains that I've seen these. I've been paged for them. I've done the analyses of possible causes. A bit pattern was written and repeatedly verified over a quite long period of time (ruling out data path issues), then at some point a different bit pattern was read and would persistently be read thereafter. How is that not real bitrot? How does it matter, beyond ruling out everything above the disk level, what the precise causes are? If you can't answer those questions, you're just posting noise.
It indeed is not. Had to reread the theory and I stand corrected, RS-style ECC can't detect errors in excess of the redundancy count.
> How is that not real bitrot?
It is and I can see how it can happen.
> How does it matter, beyond ruling out everything above the disk level, what the precise causes are?
It would've mattered if a drive could detect on-disk bitrot reliably, which was what the stats I worked with (also in exabytes, funnily enough) and the IEEE papers I read led me to believe.
For what it's worth, you won. Hats off.
You can periodically scrub the entire pool to find and even fix these issues (in a pool with redundancy)
What are some recommendations for "personal data storage" grade "silent data corruptions"?
"personal data storage" for my case is < 1TB, text / binary files (jpg, mp4).
I am looking at a wikipedia list below, and then searching through hn comments.
<https://en.wikipedia.org/wiki/Comparison_of_file_systems#Blo...> column: Data checksum/ ECC
I found many comments on ZFS, and not so much comments on dm-integrity, BlueStore/Cephfs, and others. So I am thinking of looking into ZFS, but if there are any recommendations, I would like to seek advice.
I am experimenting with Git LFS, git-annex, I like the filesystem UI better, so I am looking for filesystem like solutions.
This type of security theater can be easily bypassed by any determined attacker and thus only serves to deter honest users.
To play devil's advocate, the large amount of attackers aren't really determined. They're just fishing for easy targets. If you check the logs on a VPS you'll see an endless stream of people trying to exploit things like Wordpress 24/7 on your brand new VPS that has nothing but a html landing page.
With banks, I imagine they have a compliance check list they have to tick off to make sure that -- if and when a successful attack happens -- their insurance would pay out. If they haven't taken simple steps like blocking VPNs it could lead to the insurance company claiming negligence.
The rise of the copy-paste attacker: https://en.wikipedia.org/wiki/Script_kiddie
Known VPN-associated IP addresses were far more likely to be associated with abuse than average. Not just a little bit, but approaching 2 orders of magnitude worse in our case. It's not even close.
It's too bad for the people who need to use public VPN services for whatever reason, but until we have perfect bot/abuse detection, banning VPN, Tor, and proxy services is far and away the most effective tool for cutting down on abuse.
When asked, they cited possible abuse as a reason. But whitelisted them again after a while.
When I'm traveling I'll often pipe my traffic through a VPN on my home network. I have had some weird failures but I've usually assumed that it was due to an unreliable hotspot I'm using. Now I'm wondering if using a VPN is the real problem...
If you're tunneling through your home network it's unlikely to cause problems, unless you've been doing nefarious things from your home IP and that has also ended up on a blocklist.
Often with these things it’s just about time; it feels wrong because you’re just not used to the change yet. Maybe that will happen, but it’s been months now. Usually with these changes I change my mind quicker than that.
I think it's too soon to tell. Facebook has really negative brand recognition (from my POV), and who knows, maybe "metaverse" style online interaction is the future. (For the record I'm anti-web3 and indifferent on metaverse communities)
Their branding move was bold, yet unconvincing.
none of my life is “contained” within shitbook
I see no 3d Second Life models interacting.
In reality of course none of that exists and Meta has so far not shown how they plan to accomplish that. Worse yet, Facebook is directly responsible for making things not seamless on the Internet and in VR. So I don't have much hope (or fear) of them actually being successful in building that. But their vision is a lot broader than just Facebook in VR.
I don't think it's too soon.
I suspect it does help because a lot of hiring goes on for Quest, Instagram, Portal, WhatsApp, Workplace — you can talk to candidates about those specific products rather than make them think of grandpa posting right-wing memes on the blue app.
That being said with all of the data Facebook has from Facebook itself I would imagine Suckerberg must know that for the long term success of his different businesses they do have to distance the smaller ones a bit more than they are now - personally I am hoping we see a major fall in Facebooks value and influence and things like Oculus are spun-off to be as autonomous as possible.
I wonder if we’ll see them used for large scale applications whose correctness is critical.
I put a desk chair on Marketplace last Friday, and got 8 messages that were actually scams. These were trying to "schedule" a Fedex/DHL pickup, and would redirect me to fake branded websites that were requesting my personal details and bank account. This was so obviously fake it baffled me Meta can't detect these automatically.
I am also getting multiple message requests per week asking from hookups. These are obviously fake [1].
---
Subjectively too I also see so much less bot activity on Facebook than I do on any other social media.
There is a lot of money to be made from defrauding FB users. This monetary incentive results in criminals investing tons of effort into bypassing anti-fraud stuff. It is a non-stop effort of incremental moves on both parties that will carry on for as long as FB users remain a juicy target.