FUSE-based file system to inject faults | Hacker News Reader

FUSE-based file system to inject faults

github.com

83 pointstodsacerdoti16 comments

Nice. I think a reasonable addition to something like this might be a "truncated I/O" failure mode, i.e. make the read/write syscalls succeed, but only read/write less than what was requested.

This is kind of a pet peeve of mine. I've seen way too much FLOSS as well as proprietary C code that simply assumes return value >0 means everything was processed, or assume -1 means unrecoverable error and don't even bother to check for e.g. EINTR.

I'm curious just how bad some programs might fall on their nose when they hit such an error condition.

Wonder if there’s some way of encoding this as a much higher level, such that you could wrap any difficult handling logic inside of a library and simplify the handling of errors for us mere humans.

  #include <stdio.h>

GNOME's GIO library has a function that reads all requested bytes in a single call, until EOF or an error condition is reached: https://developer.gnome.org/gio/stable/GInputStream.html#g-i...

But don't you still need to switch on all of the possible errors to figure out what to do?

I guess I'm thinking about something that's more like a strategy that you tell the system. E.g. if it's retryable, so do this many times or up to this many seconds. If it's not possible to write (out of space, device no longer exists, internet is down), then hand me back a string I can display to my user about the error. Etc. Something that is more akin to all the cases that you'd have to handle being done inside of a library call, with easy predictable ways to work with very few possible error(s) that get returned.

It could certainly be taken further along these lines; I was mentioning it because of the "short read" scenario, which is... not quite an error, but can be the cause of one, as st_goliath writes above.

One could add a side channel to tell it to fail certain calls. I'm doing a similar thing at work---I have a mock DB that runs with a side channel for the testing framework to tell the mock DB what to pass and what to fail (I should say I'm not testing the DB, but a program that talks to the DB).

Samba actually ran into this with io_uring. It can return less that the client asked for for read/write. We were returning this to the client and the clients... didn't cope well (even though the protocol is supposed to allow this :-).

The Samba client code worked fine though :-).

Here's a block device with injectable faults: https://libguestfs.org/nbdkit-error-filter.1.html You can either have them injected randomly, or switch injection on and off at runtime. I did a talk about it at FOSDEM: https://archive.fosdem.org/2019/schedule/event/nbdkit/

I did something similar when investigating how SQL Server behaved under certain failure cases of the file system (specifically how it behaved when seeing lost writes). It's a really useful technique for validating behaviors under error conditions.

Neat. I've done something similar with an LD_PRELOAD library for a dameon's integration test suite. This might be a cleaner way to go about it.

LD_PRELOAD covers the entire kernel API, so you could do the same thing for networking and .. threads.... and memory. _that_ would be a remarkably effective chaos monkey.

note that LD_PRELOAD doesn't work for things written in Go because they have their make their own syscalls instead of going through glibc

Seccomp bpf would work as an alternative in this case: https://www.kernel.org/doc/html/latest/userspace-api/seccomp...

Slightly related, there is a proof of concept FUSE based system that mutates files being read. [1] The purpose is when fuzzing with a simple bash harness, it is not high performance but has a fast set up time.

1. https://github.com/TACIXAT/FuzzyFileSystem

We have an internal Samba VFS module that can do the same thing for SMB testing. It's really useful !

Reply on news.ycombinator.com