Hacker News Reader

Comment by teekert | Hacker News Reader

I work with TBs of genomics data and I am at this point pretty convinced that some of that data changes on-disk. Every now and then. It's very weird. Recently I heard an interview (on one of the Jupiter broadcasting shows) about how btrfs is exposing errors of underlying hardware using their checksumming features, I had a bit of an Aha! moment. The files in question (BAM files) have a sort of checksumming built in (called cigar strings) and I am getting more and more convinced that the discrepancies I run into are actually bit flips on the hardware. Anybody else have any experience with this? BTRFS would be a good tool to prevent such bitrot, imo...

View on HN

Depending on how you store this data (the number of physical disks involved) you may also want to explore ZFS. ZFS and BTRFS have pretty comparable features, but given how battle-hardened ZFS is, and its ever-increasing integration into the Linux ecosystem, I'm seeing less and less reason to prefer BTRFS.

Both do checksumming and regular scrubbing to help detect bitrot, but both need sufficient redundancy configured to actually be able to repair the corrupted data when it is detected.

There is exactly one distro that comes with ZFS, so it's a bit exaggeration to say it's well integrated with Linux, considering how much effort is applied by kernel developers to make it even harder to use ZFS. And this one distro that includes ZFS is just one lawsuit away from removing it again.

It's easy to use zfs on Debian and derivatives

  apt install dkms spl-dkms
  apt install zfs-dkms zfsutils-linux

I am testing OpenZFS with Ubuntu 20.04 LTS right now. OpenZFS seems pretty solid so far (I tried various scenarios of failure/corruption) and Ubuntu seems to actually care about ZFS support and its integration in Linux.

Ubuntu, Proxmox, Manjaro all have root on ZFS via the installer and there are others that I don't remember offhand.

That, this, is the most diplomatical way of saying 'just use ZFS' we've seen in a decayed. Mod parent up. BTRFS will never be a suck-zess. Use ext4 or ZFS for life.

Bit-flips are a lot more common than people think as are read errors on drives. Surprisingly, usually a bit flipping here or there doesn't cause any real problems so they tend to go unnoticed. If your working with TBs of data using a checksumming filesystem like ZFS of BTRFS would certainly be recommended, but you should ALSO make sure your using ECC RAM. Checking the integrity of the data as it comes off the disk doesn't do you much good if a bit then flips while it's in memory.

To emphasise, using ECC RAM is another step that can be used to increase the integrity of your data.

It’s like security - it’s done in layers.

Using a checksumming filesystem without ECC RAM is an improvement on not using a checksumming filesystem at all. Do not consider ECC RAM to be a requirement for ZFS / BTRFS.

It does add another layer of improvement though, so if you can, you should!

And even if you can’t, and even if you can’t provide the redundancy to enable bitflips to be resolved, still use a checksumming filesystem if you can!

DDR5 mainstreaming ECC RAM as the default is super exciting!

Err, no.

The CIGAR is not a checksum, even though you'll notice if it's corrupted, because it won't match the read length anymore. However, BAM is zlib-compressed (even if you asked samtools not to compress it). Each block therefore contains an actual checksum, and corruption will show up during decompression. Therefore, if you can decompress BAM, but the CIGAR doesn't make sense, you've got a software problem, not a hardware problem.

From my experience in beerinformatics, you got some software that... uhhh... interprets the BAM specification differently than you do. It's depressingly common.

Hmm, thank you for the information. But I'm by now pretty sure that I saw the checking process finish successfully at one moment, and not at others. It's not reproducible and all those algorithms should be deterministic, right? If not for bit flipping I really can't explain this.

It really can't be bit flipping on disk. First off, the checksum would catch it. Second, the file is compressed, so flipping a bit turns it into complete garbage. Third, BAM uses a length-prefixed coding in several places. A flipped bit in any length field would mess up the decoding.

Those bits could still flip in memory, which isn't BTRFS's fault. That's still unlikely, because you'd typically get segfaults, not silent corruption. So, it's probably bad software.

> those algorithms should be deterministic, right?

Combine bad code (samtools) with multithreading, and you will quickly become convinced that your computer is possessed by an evil spirit.

Currently not using btrfs by the way. But you are positive that BAM files should be self consistent? And when would you catch a bitflip then? What would it look like in a compressed file lake BAM? A much larger effect than just a cigar length mismatch supposedly?

Oy vey, veering off topic, but whatever...

BAM is a sequence of zlib-compressed blocks. Each of them has a checksum, and any corruption gets past it with probability 2^-32. Any sane software will give you an error. When in doubt, just decompress the stupid file using gunzip or zcat, those will verify the checksum. Don't guess, check.

But if you insist on using software that just ignores the checksum (I think, not even samtools, which segfaults(!) on misspelled file names(!) is that awful), all bets are off. Decompressing a corrupted file yields nonsense that doesn't even have the same length as the original data. BAM would also quickly get out of sync, and subsequent records would contain complete nonsense. You'd know if you saw that.

However...

> you are positive that BAM files should be self consistent?

That is a loaded question. In addition to being syntactically correct, BAM files should maintain a few variants that are hard to check. They rearely do. BWA(!) produces inconsistent BAM files, and I've seen many, many scripts in the wild that produce complete bullshit that only happens to work in a very specific setting. Like I said, you're probably using some awful software.

Thank you very much, this is very helpful!

I'd personally go with ZFS. There are already people in your field using ZFS to ensure the integrity of their data (based on past conversations here on HN).

I've used both ZFS and Btrfs extensively. Only Btrfs has lost data and behaved badly. I would highly recommend evaluating ZFS.

btrfs has eaten 100% of the data I have stored on it. 0/10 would not suggest.

ZFS has similar features and has eaten 0% of the data I have stored on it. would suggest.

n=1 is n=None ;)

I once ran into a RAID card that would write bad data if you put it on a 90-degree PCI connector (needed for a 2U server box). No errors appeared anywhere until you encountered corrupted files.

Hardware RAID falls into the "never again" category for me.

Though in this case it sounds like any PCI card that couldn't handle the degraded signals from the angle connector would have exhibited data corruption.

I have problems with USB packets getting corrupted on their way to my audio interface. Signal integrity is really underappreciated in the PC peripheral world.

I think /dev/md raid would have made that failure more noticeable, more quickly. Hardware RAID cards don’t always pass on disk errors to the kernel.

As other folks are suggesting, ECC may be suitable for you (alongside a protective FS). But if you want to purchase your next workstation with ECC support, be very careful: some motherboards manufacturers just say 'ECC' in their specs but that doesn't mean that they actually use ECC capabilities, that only means that you can plug ECC-RAM into it. Make sure the specs explicitly say something like "Corrects single-bit errors and detects double-bit errors" (you need both, some MBs support only single-bit correction).

I think ECC RAM and a good FS are complementary tools to achieve storage high reliability. One without another may not be that useful.

btrfs definitely helps catching errors, even without ECC or redundancy. An example: On a server with multiple VMs on a big consumer NVMe that was starting to get unreliable Windows reported nothing other than bluescreening occasionally (sfc and chkdsk showed no errors). Linux on the other hand reported errors during a bi-weekly scrub and in the kernel logs.

The SMART thresholds on the drive were strange too. A small but non-zero amount of read errors was not considered a health problem yet. I only realized that after the scrub reported problems.

>On a server with multiple VMs on a big consumer NVMe

How did you avoid the crippling performance penalty with storing VM disks on btrfs? The usual workaround suggested to btrfs users is to disable copy-on-write for VM images, but doing so also turns off checksumming thus disabling the very features that make btrfs, well, btrfs.

The VMs were running on raw partitions. The btrfs reporting errors was inside a linux VM.

I have a lot of video files (e.g. old camera tapes) stored on spinning hard drives. Bit flips are distressingly common, and increase with age.

My next storage setup will definitely use a checksumming filesystem.

What software do you use ? We use bwa/samtools/varscan/bcftools and we've reprocessed old data right from the fastq files a few times. So far we haven't gotten different results.

I encounter an error that I can't currently find exactly but it happens when running mismatch_profile from the RSeQC package. It reports a mismatch in the length of the cigar notation and the actual read inside a BAM file. Lately I have become 99% sure that this sometimes happens in files that did not show this error before (we reprocess a lot of data from time to time). I suspect it is bit flips because I think bit flips can easily go unnoticed, but not in this case, because the cigar notation is a shorter description of the actual read and they should thus match. The mismatch signals a change in the file that occurred without us touching it.

Hmm, yeah that is odd! Maybe its a bug in the alignment tool?

It's not reproducible and I say I'm 99% sure it comes from bit flips is because I know with 99% certainty the process did finish successfully on a file at some point but then failed several weeks later, on the same file.

Or ZFS, I wouldn't store that kind of data on other filesystem's without that kind of protection...

Reply on news.ycombinator.com