How Distributed Systems Respond to Degraded Hardware

Intercepting system calls seems like the easiest way (I expect most of these failures are in I/O). Use LD_PRELOAD to easily hook the libc wrappers, for (almost) anything C-based.

Deeper simulation is possible, but this should get you the low-hanging fruit.

Use ptrace to wrap the system calls for things that aren't C-based (though this is a little more involved than LD_PRELOAD).

There are several commercial users in the project list: https://www.emulab.net/projectlist.php3

https://github.com/mefellows/muxy looks interesting, albeit for certain test types. I'll be trying it out in the near future.