Hacker News Reader

Comment by londons_explore | Hacker News Reader

I would like to see filesystems benchmarked for robustness.

Specifically, robustness to everything around them not performing as required. For example, imagine an evil SSD which had a 1% chance of rolling a sector back to a previous version, a 1% chance of saying a write failed when it didn't, a 1% chance of writing data to the wrong sector number, a 1% chance of flipping some bits in the written data, and a 1% chance of disconnecting and reconnecting a few seconds later.

Real SSD's have bugs that make them do all of these things.

Given this evil SSD, I want to know how long the filesystem can keep going serving the users usecase.

View on HN

  nbdkit memory 10G --filter=error error-rate=1%

... and then nbd-loop-mount that as a block device and create your filesystem on top.

Notes:

We're working on making a ublk interface to nbdkit plugins so the loop mount wouldn't be needed.

There are actually better ways to use the error filter, such as triggering it from a file, see the manual: https://www.libguestfs.org/nbdkit-error-filter.1.html

It's an interesting idea to have an "evil" filter that flips bits at random. I might write that!

Suddenly I'm reminded of the time someone made a Bad Internet simulator (causing packet loss or other problems) and named the program "Comcast".

I love it.

How does this compare with dm-flakey [0] ?

[0]: https://www.kernel.org/doc/html/latest/admin-guide/device-ma...

dm-flakey runs inside the kernel so is going to be faster. With nbdkit + NBD loop mounting there's a bit of overhead going through a Unix socket, which is avoided by using ublk + nbdkit plugins, but you still have to go out to userspace for each device access.

The advantage is you can write plugins as regular code in lots of different programming languages, with a stable API and ABI: https://www.libguestfs.org/nbdkit.1.html So it's a lot easier to experiment or write plugins for odd devices.

Also our sparse RAM disk supports huge disks (up to 16 exabytes IIRC, 2^63-1 bytes whatever that is).

> For example, imagine an evil SSD which had a 1% chance of rolling a sector back to a previous version, a 1% chance of saying a write failed when it didn't, a 1% chance of writing data to the wrong sector number, a 1% chance of flipping some bits in the written data, and a 1% chance of disconnecting and reconnecting a few seconds later.

There are stories from the ZFS folks of dealing with these issues and things ran just fine.

While not directly involved with ZFS development (IIRC), Bryan Cantrill was very 'ZFS-adjacent' since he used Solaris-based systems for a lot of his career, and he has several rants about firmware that you can find online.

A video that went viral many years ago, with Cantrill and Brendan Gregg, is "Shouting in the Datacenter":

* https://www.youtube.com/watch?v=tDacjrSCeq4

* https://www.youtube.com/watch?v=lMPozJFC8g0 (making of)

A 1% error rates for corrupting other blocks is prohibitively. A file system would have do extensive forward error correction in addition to checksumming to have a chance to work with this. It would also have to perform a lot of background scrubbing to stay ahead of the rot. While interesting to model and maybe even relevant as a research problem given the steadily worsening bandwidth to capacity ratio of affordable bulk storage I don't expect there are many users willing to accept the overhead required to come even close to a usable file system an a device as bad as the one you described.

Well, it’s sort of redundant. According to [1], the raw bit error rate of the flash memory inside today’s SSDs is already in the 0.1%-1% range. And so the controllers inside the SSDs already do forward error correction, more efficiently than the host CPU could do it since they have dedicated hardware for it. Adding another layer of error correction at the filesystem level could help with some of the remaining failure modes, but you would still have to worry about RAM bitflips after the data has already been read into RAM and validated.

[1] https://ieeexplore.ieee.org/document/9251942

ZFS will do this. Give it a RAIDz-{1..3} setup and you've got the FEC/Parity calculations that happen. Every read has it's checksum checked, and when reading if it finds issues it'll start resilvering them asap. You are of course right in that it will eventually start getting to worse and worse performance as it's having to do much more rewriting and full on scrubbing if there are constant amounts of errors happening but it can generally handle things pretty well.

I don't know...

Let's say you have 6 drives in raidz2. If you have a 1% silent failure chance per block, then writing a set of 6 blocks has a 0.002% silent failure rate. And ZFS doesn't immediately verify writes, so it won't try again.

If that's applied to 4KB blocks, then we have a 0.002% failure rate per 16KB of data. It will take about 36 thousand sets of blocks to reach 50% odds of losing data, which is only half a gigabyte. If we look at the larger block ZFS uses internally then it's a handful of gigabytes.

And that's without even adding the feature where writing one block will corrupt other blocks.

I've had ZFS pools survive (at different times over the span of years):

- A RAIDz1 (RAID5) pool with a second disk start failing while rebuilding from an earlier disk failure (data was fine)

- A water to air CPU cooler leaking, CPU overheated and the water shorted and killed the HBA running a pool (data was fine)

- An SFF-8088 cable half plugged in for months, pool would sometimes hiccup, throw off errors, take a while to list files, but worked fine after plugging it in properly (data was fine after)

Then the usual disk failures which are a non-event with ZFS.

I recovered from 3 disk RAID 6 failure (which itself was organization failure driven...) in linux's mdadm... ddrescue to the rescue, I guess I got "lucky" the bad blocks didn't happen in same place on all drives (one died, other started returning bad blocks), but chance for that to happen are infinitesly small in the first place

So shrug

How do you know you got lucky with corrupted blocks?

mdadm doesn’t checksum data, and just trusts the HDD to either return correct data, or an error. But HDDs return incorrect data all the time, their specs even tell you how much incorrect data they’ll return, and for anything over about 8TB you’re basically guaranteed some silent corruption if you read every byte.

> and for anything over about 8TB you’re basically guaranteed some silent corruption if you read every byte

This is simply not true. I've been running bi-weekly ZFS scrubs on my file server which has grown from 80 TB to 200 TB of data over three years now with zero failed checksums. That is petabytes of reads with zero corruption. The oldest drives are nearing 5 years old (they were in a Btrfs Synology NAS before, again, zero failed checksums).

The wildly pessimistic numbers on the spec sheets are probably just there so they can deny warranty replacements by saying that some errors are in-spec.

Those specs aren't true, or at least the distribution is very far from uniform and they're failing to explain it even a fraction of the way. You don't get corruption that often.

Yes, I also had a "1 disk failed, second disk failed during rebuild" event (like the parent, not your story) with mdadm & RAID 6 with no issues.

People seem to love ZFS but I had no issues running mdadm. I'm now running a ZFS pool and so far it's been more work (and things to learn), requires a lot more RAM, and the benefits are... escaping me.

Are you sure the data survived? ZFS is sure and proves it with checksums over metadata and data. I don't know madm well enough to know if it does this too.

It doesn't

My son has a ZFS pool where he just realised one of the drives has half a billion errors on it and the data is still fine. New drive has been ordered!

Perhaps you ought to run to a Best buy and buy external drives and do backups ASAP.

RAID is not backup, always remember that!

It's actually all replaceable data and he's planning to recreate the pool anyway. Just insane that ZS handles it though.

This is why I always opt for ZFS.

Please, where can i read more about this. I remember bricking OCZ drive by setting ATA password, as was fashionable to do back then, but 1% writes going to wrong sector - what are these drives, fake sd cards from aliexpress?

Like, which manufacturer goes, like "tests show we cannot write more that 400kB without corrupting drive, let us ship this!" ?

> Like, which manufacturer goes, like "tests show we cannot write more that 400kB without corrupting drive, let us ship this!" ?

According to https://www.sqlite.org/howtocorrupt.html there are such drives:

4.2. Fake capacity USB sticks

There are many fraudulent USB sticks in circulation that report to have a high capacity (ex: 8GB) but are really only capable of storing a much smaller amount (ex: 1GB). Attempts to write on these devices will often result in unrelated files being overwritten. Any use of a fraudulent flash memory device can easily lead to database corruption, therefore. Internet searches such as "fake capacity usb" will turn up lots of disturbing information about this problem.

Bit flips amd overwriting wrong sectors from unrelated files being written are also mentioned. You might think this sort of thing is just cheap USB flash drives, bit I've been told about NVMe SSDs violating their guarantees and causing very strange corruption patterns too. Unfortunately when the cause is a bug in the storage's device own algorithms, the rare corruption event is not necessarily limited to a few bytes or just 1 sector here or there, nor to just the sectors being written.

I don't know how prevalent any of these things are really. The sqlite.org says "most" consumer HDDs lie about commiting data to the platter before reporting they've done so, but when I worked with ext3 barriers back in the mid 2000s, the HDDs I tested had timing consistent with flushing write cache correctly, and turning off barriers did in fact lead to observable filesystem corruption on power loss, which was prevented by turning on barriers. The barriers were so important they made the difference between embedded devices that could be reliably power cycled, vs those which didn't reliably recover on boot.

> Fake capacity USB sticks

Those drives have a sudden flip from 0% corruption to 95-100% corruption when you hit their limits. I wouldn't count that as the same thing. And you can't reasonably expect anything to work on those.

> The sqlite.org says "most" consumer HDDs lie about commiting data to the platter before reporting they've done so, but when I worked with ext3 barriers back in the mid 2000s, the HDDs I tested had timing consistent with flushing write cache correctly

Losing a burst of writes every once in a while also manifests extremely differently from steady 1% loss and needs to be handled in a very different way. And if it's at power loss it might be as simple as rolling back the last checkpoint during mount if verification fails.

Writes going to the wrong sector are usually wear levelling algorithms gone wrong. Specifically, it normally means the information about which logical sector maps to which physical sector was updated not in sync with the actual writing of the data. This is a common performance 'trick' - by delaying and aggregating these bookkeeping writes, and taking them off the critical path, you avoid writing so much data and the user sees lower latency.

However, if something like a power failure or firmware crash happens, and the bookkeeping writes never happen, then the end result that the user sees after a reboot is their data written to the wrong sector.

With this level of adversarial problems, you'd better formulate your whole IO stack as a two-player minimax game.

On the whole, I'm not sure that a FS can work around such a byzantine drive, at least not if its the only such drive in the system. I'd rather FSes not try to pave over these: these disks are faulty, and we need to demand, with our wallets, better quality hardware.

> 1% chance of disconnecting and reconnecting a few seconds later

I actually have such an SSD. It's unusable, when it is in that state. The FS doesn't corrupt the data, but it's hard for the OS to make forward progress, and obviously a lot of writes fail at the application level. (It's a shitty USB implementation on the disk: it disconnects if it's on a USB-3 capable port, and too much transfer occurs. It's USB-2, though; connecting it to a USB-2-only port makes it work just fine.)

At the point where it disappears for seconds it's a distributed system not an attached disk. At this point you have to start applying the CAP theorem.

At least in Unix the assumption is that disks are always attached (and reliable...) so write errors don't typically bubble up to the application layer. This is why losing an NFS mount typically just hangs the system until it recovers.

> At least in Unix the assumption is that disks are always attached (and reliable...)

I want to say a physical disk being yanked from the system (essentially what was happening, as far as the OS could tell) does cause I/O errors in Linux? I could be wrong though, this isn't exactly something I try to exercise.

As for it being a distributed system … I suppose? But that's what an FS's log is for: when the drive reconnects, there will either be a pending WAL entry, or not. If there is, the write can be persisted, otherwise, it is lost. But consistency should still happen.

Now, an app might not be ready for that, but that's an app bug.

But it can always happen that the power goes out, which in my situation is equivalent to a disk yank. There's also small children, the SO tripping over a cable, etc.

But these are different failure modes from some of what the above post listed, such as disks undoing acknowledged writes, or lying about having persisted the write. (Some of the examples are byzantine, some are not.)

The OS notices when disks or NFS servers ("NFS server not responding") go missing, but there's no way to generally broadcast this information to every process. And then you would have to build in error handling everywhere. What does `cat` do when it's informed that the file it's reading is temporarily unavailable? You can either block the next read() call and hang the process until the disk comes back, or return a hard error that it's gone forever. Otherwise each application has to have its own wait/timeout code. (Which is what applications dealing with HTTP have to deal with but at least when you make an HTTP connection you're aware that something could go wrong.)

NFS is a bit of a pathological case.

> but there's no way to generally broadcast this information to every process.

Sure there is: pending I/O returns EIO.

> What does `cat` do when it's informed that the file it's reading is temporarily unavailable?

Let's test it. I inserted a USB disk, mounted it, started `cat`, and yanked the disk. `cat`'s read(2) call returns EIO, and cat dies with an error.

I'd expect similar results from most processes: for applications, there is no sane, in-process way to recover from such a situation; either you can rely on the FS's capabilities, if any, or you need to build those into your files (e.g., how SQLite does).

> And then you would have to build in error handling everywhere.

Apps that presume I/O isn't fallible are broken. Disconnection is hardly the only possible error during I/O.

NFS is, again, rather pathological: the FS driver there I think just keeps the mount alive, and pends the writes until a connection is re-established. I'd argue this is terrible behavior, too: the probability with which an SRE has to debug a hung, nigh unkillable process when NFS is involved in system design is 100%.

Ackchually, the journal is not a WAL (a journal is a log of intended and completed actions, while the WAL actually stores the data before committing it to the data store, it's more advanced and can recover all data ).

But yeah, I agree that is what the journal is for. However, I don't know if the journal in "modern" filesystems works the same when a drive reconnects than when the power is cut out (the OS still being alive and all in the first case).

I would expect that it must: the OS cannot assume that the newly connected drive wasn't altered by another system in the meantime. (Or that it's the same FS, even.)

(although I grant that, a.) the FS probably has a UUID that idents it, and b.) it's only gone for a few ms — but those aren't things that, reasonably, the OS is going to look at to go "eh, we don't need to recover this newly attached disk".)

(I'm not sure I'm still grokking the differences between a WAL & a journal, to you.)

I have basically the opposite problem. I've been looking for a filesystem that maximizes performance (and minimizes actual disk writes) at the cost of reliability. As long as it loses all my data less than once a week, I can live with it.

Don't you want a RAM disk for this? It'll lose all your data (reliably!) when you reboot.

You could also look at this: https://rwmj.wordpress.com/2020/03/21/new-nbdkit-remote-tmpf... We use it for Koji builds, where we actually don't care about keeping the build tree around (we persist only the built objects and artifacts elsewhere). This plugin is pretty fast for this use case because it ignores FUA requests from the filesystem. Obviously don't use it where you care about your data.

> Don't you want a RAM disk for this? It'll lose all your data (reliably!) when you reboot.

Uhh hello, pricing ?

Depends on how much space you need.

Is there RAM that can interface with an M.2 slot rather than the standard ram slots?

CXL is kind of like that. I don't think there's specifically an M.2 form factor but it's (in hand-waving terms) RAM over PCIe.

https://en.wikipedia.org/wiki/Compute_Express_Link

Have you tried allowing ext4 to ignore all safety? data=writeback, barrier=0, bump up dirty_ratio, tune ^has_journal, maybe disable flushes with https://github.com/stewartsmith/libeatmydata

You can also add journal_async_commit,noauto_da_alloc

> maybe disable flushes with https://github.com/stewartsmith/libeatmydata

overlayfs has a volatile mount option that has that effect. So stacking a volatile overlayfs with the upper and lower on the same ext4 could provide that behavior even for applications that can't be intercepted with LD_PRELOAD

Thanks, this looks promising

Cue MongoDB memes

Probably just using it for cache of some kind

You should be able to do this with basically any file system by using the mount options `async`(default) `noatime` disabling journalling, and massively increasing vm.dirty_background_ratio, vm.dirty_ratio, and vm.dirty_expire_centisecs.

Tongue-in-cheek solution: use a ramdisk[1] for dm-writecache in writeback mode[2]?

[1]: https://www.kernel.org/doc/Documentation/blockdev/ramdisk.tx...

[2]: https://blog.delouw.ch/2020/01/29/using-lvm-cache-for-storag...

It all depends on how much reliability are you willing to give up for performance.

Because I have the best storage performance you'll ever find anywhere, 100% money-back guaranteed: write to /dev/null. It comes with the downside of 0% reliability.

You can write to a disk without a file-system, sequentially, until space ends. Quite fast actually, and reliable, until you reach the end, then reliability drops dramatically.

Trouble is you can't use /dev/null as a filesystem, even for testing.

On a related note, though, I've considered the idea of creating a "minimally POSIX-compliant" filesystem that randomly reorders and delays I/O operations whenever standards permit it to do so, along with any other odd behavior I can find that remains within the letter of published standards (unusual path limitations, support for exactly two hard links per file, sparse files that require holes to be aligned on 4,099-byte boundaries in spite of the filesystem's reported 509-byte block size, etc., all properly reported by applicable APIs).

Yeah, I've had good experience with bypassing fs layer in the past, especially on a HDD the gains can be insane. But it won't help as I still need a more-or-less posixy read/write API.

P.S. I'm fairly certain that /dev/null would lose my data a bit more often than once a week.

Not sure if this is feasible, but have you considered dumping binary on the raw disk like done with tapes?

You could even use partitions as files. You could only have 128 files, but maybe that's enough for OP?

How would you cope with losing all you data once a week?

"once a week" was maybe a too extreme example. For my case specifically: lost data can be recomputed. Basically a bunch of compiler outputs, indexes and analysis results on the input files, typically an order of magnitude larger than the original files themselves.

Any files that are important would go to a separate, more reliable filesystem (or uploaded elsewhere).

On top of other suggestions I've seen you get already, raid0 might be worth looking at. That has some good speed vs reliability tradeoffs (in the direction you want).

Some video production workflows are run on 4xraid0 just for the speed - it fails rarely enough and intermediate output is just re-created.

Can confirm. When I can't work off my internal MacBook storage, my working drive is a RAID0 NVME array over Thunderbolt. Jobs setup in Carbon Copy Cloner make incremental hourly backups to a NAS on site as well as a locally-attached RAID6 HDD array.

If the stripe dies, worst case is I lose up to one hour of work, plus let's say another hour copying assets back to a rebuilt stripe.

There are so many MASSIVE files created in intermediate stages of audiovisual [post] production.

The file system is tmpfs. Just don’t reboot :)

Kent use md-faulty to test such a scenario, you can also see here the test-CI https://evilpiepirate.org/~testdashboard/ci

If you have an evil SSD which screws up 5% (1 in 20) of your writes, well... You don't need a better FS. You need a new disk right now.

This is not the kind of frequency you need to protect against. You just need to detect it and replace the drive ASAP.

ISTM one could design a filesystem as a Byzantine-fault-tolerant distributed system that happens to have many nodes (disks) partially sharing hardware (CPU, memory, etc). The result would not look that much like RAID, but would look quite a bit like Ceph and its relatives.

Bonus points for making the result efficiently support multiple nodes, each with multiple disks.

Reply on news.ycombinator.com