Hacker News Reader

Comment by kolektiv | Hacker News Reader

It always surprised me somewhat that there isn't a set of traits covering some kind of `fs` like surface. It's not a trivial surface, but it's not huge either, and I've also found myself in a position of wanting to have multiple implementations of a filesystem-like structure (not even for the same reasons).

Tricky to make that kind of change to std lib now I appreciate, but it seems like an odd gap.

View on HN

See David R. Hanson's "A Portable File Directory System" [0][1], for example: a 700 lines long implementation of early UNIX's filesystem API that piggy-backs on some sort of pre-existing (block-oriented) I/O primitives, which means you can do it entirely in-memory, with about another 300 lines of code or so.

I suspect that with OSes becoming much more UNIX-like the demand for such abstraction layers shrank almost to nothing.

[0] https://drh.github.io/documents/pds-spe.pdf

[1] https://drh.github.io/documents/pds.pdf

Having traits in the stdlib would be nice, but there's also the type parameter pollution that results from mocking which I think is also a turn-off, as TFA says about rsfs.

I have a Rust library to implement the UAPI config spec (a spec that describes which files and directories a service should look for config files in), and initally wanted to test it with filesystem mocks. After making some effort to implement the mock types and traits, plus wrappers around the `<F = StdFs>` types to hide the `<F>` parameter because I didn't want to expose it in the public API, I realized it was much easier to not bother and just create all the directory trees I needed for the tests.

Yeah having traits for this in the stdlib would be nice.

You might find Lunchbox [1] interesting. I needed an async virtual filesystem interface for a project a few years ago (and didn't find an existing library that fit my needs) so I built one:

> Lunchbox provides a common interface that can be used to interact with any filesystem (e.g. a local FS, in-memory FS, zip filesystem, etc). This interface closely matches `tokio::fs::` ...

It includes a few traits (`ReadableFileSystem`, `WritableFileSystem`) along with an implementation for local filesystems. I also used those traits to build libraries that enable things like read-only filesystems backed by zip files [2] and remote filesystems over a transport (e.g. TCP, UDS, etc) [3].

[1] https://crates.io/crates/lunchbox

[2] https://crates.io/crates/zipfs

[3] https://github.com/VivekPanyam/carton/tree/main/source/anywh...

Go has a basic FS abstraction in the standard library: https://dev.to/rezmoss/gos-fs-package-modern-file-system-abs...

But the stdlib one is a bit barebones. So people created: https://github.com/spf13/afero

This is one nice thing about go.

I think was trying to test something in Rust and I was surprised by how many people were OK with using real file's for unit testing.

It seems like a massive oversight for being able to use rust in a corporate environment.

> It seems like a massive oversight for being able to use rust in a corporate environment.

Why does being in a corporate environment matter?

Permissions are usually more strict then a hobby projects. So you can't just be writing and leaving things all over the file system without the possibility of failure.

Or maybe your using drives over a network and randomly your tests will now fail becasue of things outside your control. Things like that.

That's why when writing tests you always want them to actually do io like that.

Seems like you could just put your test files in a tmpfs, if leaving files laying around is an issue.

You need to mount and unmount fikesystems to do that.

Take a look at the library mentioned,afero, and you'll see how nice it handles working with the file system in tests.

You can have everything in memory, and a whole new fs in each test

The problem is it only tests trivial properties of the API. That might be enough for some things, but it’s not a panacea.

This is something I'm hopeful will fall out accidentally from Zig's new IO interface. If everyone doing IO has the implementation injected, mocks (fault injection, etc) become trivial, along with any other sort of feature you might want like checking S3 if you don't have a sufficiently recent local copy.

Mocking file system or network seems counter productive.

Complicated logic can be in pure functions and not be intertwined with IO if it needs to be tested.

Mocking IO seems like it won’t really capture the problems you might encounter in reality anyway.

It's not always about mocking (in my cases it hasn't been). Sometimes it is about multiple "real" implementations - a filesystem is itself an abstraction, and a very common one, it seems like it would at least sometimes be useful to be able to leverage that more flexibly.

I have a little system that takes a .git and mounts it as a fuse filesystem. Every commit becomes a directory with a snapshot of the project at that point in time.

You could read the whole .git in at once, and then you'd have an in-memory file-system, if you wanted to.

In any case, I agree with you: it's not about mocking.

Some examples where it would be useful: Exposing a zip file or exe-embedded data as a filesystem, or making an FS backed by 9P, WebDAV or SFTP.

Same like mocking databases; yes they make your tests run faster and more “pure”, but you’re suddenly not really testing reality anymore.

Sure, but they're not mutually exclusive approaches. Having tests that run in a couple of seconds thanks to low I/O and cheap scaffolding setup/teardown can be a valuable addition to slower, comprehensive system and integration test suites.

And you can spot differences between your understanding of reality (test mockups) and how it behaves on the real filesystem. (In this case.)

Fault injection is much easier if you can mock IO. And you aren't really testing your software if you're not injecting faults.

My point is, you can give gibberish data to a processing function if it just takes in-memory data instead of being parameterized by “IO”.

If you parameterize everything by IO then you have to mock the IO

It's not just about gibberish data. It's read failures, write failures, open failures, for a variety of reasons including permission errors, disk errors, space errors, etc.

You basically want to test your code does something sensical with either every possible combination of errors, or with some random subset of combinations of errors. This can be automated and can verify the behaviour of your program in the edge cases. It's actually not usually that helpful to only test "we read corrupt garbage from the file" which is the thing you're describing here.

Reply on news.ycombinator.com