In-Memory Filesystems in Rust
andre.arko.net
andre.arko.net
Going this route means you're going to leverage all the well tested Linux VFS code and your tests will execute with higher fidelity.
Tricky to make that kind of change to std lib now I appreciate, but it seems like an odd gap.
Complicated logic can be in pure functions and not be intertwined with IO if it needs to be tested.
Mocking IO seems like it won’t really capture the problems you might encounter in reality anyway.
You could read the whole .git in at once, and then you'd have an in-memory file-system, if you wanted to.
In any case, I agree with you: it's not about mocking.
If you parameterize everything by IO then you have to mock the IO
You basically want to test your code does something sensical with either every possible combination of errors, or with some random subset of combinations of errors. This can be automated and can verify the behaviour of your program in the edge cases. It's actually not usually that helpful to only test "we read corrupt garbage from the file" which is the thing you're describing here.
I suspect that with OSes becoming much more UNIX-like the demand for such abstraction layers shrank almost to nothing.
But the stdlib one is a bit barebones. So people created: https://github.com/spf13/afero
I think was trying to test something in Rust and I was surprised by how many people were OK with using real file's for unit testing.
It seems like a massive oversight for being able to use rust in a corporate environment.
Why does being in a corporate environment matter?
Or maybe your using drives over a network and randomly your tests will now fail becasue of things outside your control. Things like that.
That's why when writing tests you always want them to actually do io like that.
Take a look at the library mentioned,afero, and you'll see how nice it handles working with the file system in tests.
You can have everything in memory, and a whole new fs in each test
I have a Rust library to implement the UAPI config spec (a spec that describes which files and directories a service should look for config files in), and initally wanted to test it with filesystem mocks. After making some effort to implement the mock types and traits, plus wrappers around the `<F = StdFs>` types to hide the `<F>` parameter because I didn't want to expose it in the public API, I realized it was much easier to not bother and just create all the directory trees I needed for the tests.
You might find Lunchbox [1] interesting. I needed an async virtual filesystem interface for a project a few years ago (and didn't find an existing library that fit my needs) so I built one:
> Lunchbox provides a common interface that can be used to interact with any filesystem (e.g. a local FS, in-memory FS, zip filesystem, etc). This interface closely matches `tokio::fs::` ...
It includes a few traits (`ReadableFileSystem`, `WritableFileSystem`) along with an implementation for local filesystems. I also used those traits to build libraries that enable things like read-only filesystems backed by zip files [2] and remote filesystems over a transport (e.g. TCP, UDS, etc) [3].
[1] https://crates.io/crates/lunchbox
[2] https://crates.io/crates/zipfs
[3] https://github.com/VivekPanyam/carton/tree/main/source/anywh...
To be clear `File::drop()` does sync, it just ignores errors (because `drop()` doesn't have a way of returning an error). It's not really Rust specific I guess, I just don't know off the top of my head what other languages behave this way.
I've wondered for a while what it'd take to eliminate such pitfalls in the "traditional" RAII approach. Something equivalent to deleting the "normal" RAII destructor and forcing consumption via a close() could be interesting, but I don't know how easy/hard that would be to pull off.
Modern Java provides the concept of suppressed exceptions. Basically an exception can maintain the list of suppressed exceptions.
When stack unwinds, just allow finaliser to throw an exception. If it threw an exception, either propagate it to the handler, unwinding stack as necessary, or add it to the current exception as a suppressed exception. Exception handler can inspect the suppressed exceptions, if necessary. Exception print routine will print all suppressed exceptions for log visibility.
Java does not do it properly for finally blocks, instead overwriting current exception, probably because the concept of suppressed exception was introduced in the later versions and they wanted to keep the compatibility.
But it can be done properly.
Does make me wonder about the specifics behind that. I had assumed that there are some kind of soundness issues that force that particular approach (e.g., https://github.com/rust-lang/rust/pull/110975, "Any panics while the panic hook is executing will force an immediate abort. This is necessary to avoid potential deadlocks like rustc hangs after ICEing due to memory limit #110771 where a panic happens while holding the backtrace lock."; alternatively, some other kind of soundness issue?), but I don't have the knowledge to say whether this is a fundamental limitation or "just" an implementation quirk that basically got standardized. Rust' first public release was after Java 7, so in principle the precedent was there, for what it's worth.
It does not. BufWriter<File> flushes its userspace buffer (but doesn't fsync either). If you have a bare File then drop really just closes the file descriptor, that's it.
https://github.com/rust-lang/rust/blob/ee361e8fca1c30e13e7a3...
According to the standard, fstream doesn't have an explicit destructor, but the standard says "It uses a basic_filebuf<charT, traits> object to control the associated sequences." ~basic_filebuf(), in turn, is defined to call close() (which I think flushes to disk?) and swallow exceptions.
However, I can't seem to find anything that explicitly ties the lifetime of the fstream to the corresponding basic_filebuf. fstream doesn't have an explicitly declared destructor and the standard doesn't require that the basic_filebuf is a member of fstream, so the obvious ways the file would be closed don't seem to be explicitly required. In addition, all of fstream's parents' destructors are specified to perform no operations on the underlying rdbuf(). Which leaves... I don't know?
cppreference says the underlying file is closed, though, which should flush it. And that's what I would expect for an RAII class! But I just can't seem to find the requirement...
close without fsync (or direct IO) essentially is telling the OS that you don't need immediate durability and prefer performance instead.
The more general issue (not checking close(2) errors) is mostly true for most programming languages. I can count on one hand how many C programs I've seen that attempt to check the return value from close(2) consistently, let alone programs in languages like Go where handling it is far more effort than ignoring it.
Also, close(2) doesn't consistently return errors. Because filesystem errors are often a property of the whole filesystem and data written during sync has been disassociated from the filesystem, the error usually can't be linked to a particular file descriptor. Most filesystems instead just return EIO if the filesystem had an error at all. This is arguably less useful than not returning an error at all because the error might be triggered by a completely unrelated process and (as above) you might not receive errors that you do care about.
Filesystems also have different approaches to which close(2) calls will get filesystem errors. Some only return the error to the first close(2) call, which means another thread or process could clear the error bit. Other filesystems keep the error bit set until a remount, which means that any program checking close(2) will get lots of spurious errors. From memory there was a data corruption bug in PostgreSQL a few years ago because they were relying on close(2) error semantics that didn't work for all filesystems.
The interesting part, to me, was that using the vfs crate or the rsfs crate didn't produce any differences from using tmpfs or an SSD. In theory, those crates completely cut out the actual filesystem and the OS entirely. Somehow, avoiding all those syscalls didn't make it any faster? Not what I expected.
Anyway, if you have examples of in-process filesystem mocks that run faster than the in-memory filesystem cache, I'd love to hear about them.
A+ on technical prowess,
F- on being able to articulate a couple words about it on a text file.
https://blog.metaobject.com/2017/02/mkfile8-is-severely-sysc...
That was 8 years ago, and even then mkfile needed a 512K buffer size to saturate the hardware. With the 512 byte default buffer it was 8x slower than the hardware.
In addition, as others have pointed out, if you are not doing something extra to ensure things are flushed to disk, you are just measuring the buffer cache in the first place.
You might run into weird environment or permission issues when you run your tests in a ci job.
Omg, the comment about getting lectured. Relatable about anything testing related. It's happening this comment section. Someone hand waving testing away with pure functions, I'm sure someone will say pattern matching will solve all the problems thanks to the compiler.
I had expected that without the ramdisk, that file IO would have been the bottleneck for my testing, but in fact found that even with regular file IO my cpu was still the bottleneck, and multi-threading provided a massive performance boost. Completely decimating my expectations.
Well, benchmarks could be wrong or misleading. Did you make sure that the IO actually happens and that it dominates the process execution time?