Fuzzing Raft for Fun and Publication

I'm very excited by this work. As they say, testing for distributed systems leaves a great deal to be desired, but there work is a great step!

I was excited for FoundationDB, who put a great deal of focus on testing[1], but they were acquired into mystery by Apple. To give you an idea of how well tested they were, Aphyr of "Call Me Maybe" fame didn't test them "because their testing appears to be waaaay more rigorous than mine"[2]. FoundationDB ran Jepsen internally anyway[3].

DEMi is a really nice tool that has me excited. I've worked with similar test systems for non-blocking IO models and they're hugely useful. For non-blocking IO, you can artificially accelerate time as you can leap to the next interrupt, speeding up fuzzing substantially (see: Flow from FoundationDB in [1]).

@ikneaddough, the demi-applications repo notes Spark, have you tested it so far? I quite like Spark but had some very temperamental issues at scale.

[1]: https://www.youtube.com/watch?v=4fFDFbi3toc

[2]: https://twitter.com/aphyr/status/405017101804396546

[3]: https://web.archive.org/web/20150325003511/http://blog.found...

This is super interesting - especially the tool they used. It looks like it interposes itself at the RPC level, unlike Jepsen which intervenes at the network level. That lets it create reproducible test cases in a distributed system, which it then minimizes so that they can be understood more easily. Very cool.

https://github.com/NetSys/demi http://www.eecs.berkeley.edu/~rcs/research/nsdi_draft.pdf

I am working on an implementation of Raft and that's a gold mine of issues to look for. It's also relieving to see testing tools for DSs coming up.

DEMi is written very clearly, and it's a pleasure to read the code: https://github.com/NetSys/demi . Definitely challenges the research "spaghetti code" stereotype.

Really cool. :-)

Next, they should fuzz IronFleet: http://research.microsoft.com/apps/pubs/default.aspx?id=2558...