Hacker News Reader

Comment by josephg | Hacker News Reader

I love Kyle's work with Jepsen. Has anyone done anything similar with filesystems? I'm imagining a filesystem mounted on a virtual device, and having the virtual device randomly suffer write faults and various other failures. And then a fuzzer asserting that all consistency requirements are still met. (All operations before a successful fsync should be stored, no lost writes or files, no unexpected garbage data, etc.)

View on HN

You can imagine that Jepsen could run against filesystems itself. Basic JVM IO would be fairly easy to get going, and the POSIX APIs could be tested by having Jepsen call (maybe via IPC?) a small C/whatever binary. I don't know what fault injection would look like though: presumably you'd want to do something down in the kernel/device drivers/hw, and that land is largely a mystery to me.

Tangentially, I would love to have a FUSE filesystem with a.) minimal build dependencies, b.) some sort of CLI interface, and c.) the ability to, say, forget to flush un-fsynced data to "disk", allowing us to simulate a power failure. There have been a bunch of research projects on this front, and they find bugs spectacularly. I bet this approach would also find errors in distributed systems, but I've yet to find one that really has the right shape for use with Jepsen.

This is probably not relevant to Jepsen, and I don't know how similar to a distributed database distributed metadata in parallel filesystems look. Anyway, some of the tests for the Lustre PFS use Linux' fault injection system. Also, Linux documents specific support for NFS fault injection, which I know nothing about, and may or may not apply to pNFS. (It probably doesn't make it any more relevant or databased, but the OrangeFS PFS uses bdb or lmdb instances for the arbitrarily distributed metadata.)

Still a layer above the filesystem, but the closest example I'm aware of is the IO error testing in SQLite's test suite:

> I/O errors are simulated in both the TCL and TH3 test harnesses by inserting a new Virtual File System object that is specially rigged to simulate an I/O error after a set number of I/O operations. As with OOM error testing, the I/O error simulators can be set to fail just once, or to fail continuously after the first failure. Tests are run in a loop, slowly increasing the point of failure until the test case runs to completion without error. The loop is run twice, once with the I/O error simulator set to simulate only a single failure and a second time with it set to fail all I/O operations after the first failure.

https://www.sqlite.org/testing.html#i_o_error_testing

Yes filesystems are another big important part when it comes to the topic of data consistency. I'm not aware of a project similar to Jepsen for filesystems but there have been research papers like the one presented in this page that explore the guarantees of popular local filesystems: https://danluu.com/file-consistency/

Of course distributed filesystems are even harder but we can see that even writing to a local file is surprisingly complex and in many (most?) apps probably not really guaranteeing proper data consistency.

Imho the whole Posix filesystem API is flawed to begin with and it would be great if a modern replacement would emerge.

We did something in the neighborhood, check it out: https://www.scylladb.com/2016/02/16/fault-injection-filesyst...

> I'm not aware of a project similar to Jepsen for filesystems

Not as elaborate as Jepsen, but there has been some work:

- Kirk McKusick's papers and work on BSD fs log semantics

https://www.researchgate.net/scientific-contributions/Marsha...

I believe the term to look for is "soft updates."

- the BSD/NeXT file test program "fstest.c", used on local and NFS (Samba), which found many bugs in popular fs using simple operations. The ZFS team also has a version.

You can Bing versions of that by using quotes "fstest.c".

- the Luster/Gluster maintainers/consulting team used to just untar emacs on their distributed fs buildouts and see how many nodes left the cluster. (They lived off DARPA funding basically, and were paid to configure and install distributed fs for US govt/military supercomputer installations, and fix the underlying bugs as found.)

- Ironically, the Ceph team did not own any commercial storage devices, so just tested on regular linux machines.

- Reiserfs 3 was the first GA log fs on linux (default on SUSE), so I was one of the earliest US users in production.

SUSE's rep called me a liar at trade show, saying "nobody uses our distro in the US. :) It worked well on email server loads, and could delete 1 million files in a directory in under 1 second. I followed the development of v4, but the "wandering logs" and "dancing trees", etc. kind of wigged me out.

https://en.wikipedia.org/wiki/Dancing_tree

Source: DBA and storage engineer.

IRON file systems https://people.cs.uchicago.edu/~haryadi/pdf/sosp05-ironFS.pd...

EIO: Error Handling is Occasionally Correct https://www.usenix.org/legacy/events/fast08/tech/gunawi.html

Evaluating File System Reliability on Solid State Drives https://www.usenix.org/conference/atc19/presentation/jaffer

Relatedly, at the Midlands Grad School 2019 conference, John Hughes did an introductory course to QuickCheck in which one of the final exercises (meant to demonstrate the use of Haskell and QuickCheck to black-box test arbitrary binaries) was to model `fopen`, `fprintf`, `fclose` and friends in C. I never actually did it, but I guess that's one of the most comprehensive ways to find out how something behaves!

Reply on news.ycombinator.com