Automated Failure Testing

techblog.netflix.com

42 points·hepha1979··4 comments

I love failure testing, this is a great article. Molly looks pretty interesting. Netflix is remarkably bold for enabling their failure tests in production.

Shameless plug for libfaultinj [1], a fault injection library. It's remarkably simple (and still not quite as feature-rich as I'd like). You can use it in a unit-testing context to make sure you do something sane for things like file-not-found, disk-full, or extraordinary-delay-on-open. Or you can use it in a larger-scale integration style testing with random likelihood of failures popping up.

https://github.com/androm3da/libfaultinj

> Netflix is remarkably bold for enabling their failure tests in production.

This is almost a best practice these days, isn't it? I mean, once your reliability engineering teams are mature enough from a visibility and tooling standpoint to be able to identify and prevent issues. Of course, this all assumes a relatively robust architecture from the get-go...

Maybe a bit off topic, but I couldn't find any info on the origin of the name "Molly". Am I guessing correctly that it has something to do with this[1] anecdote?

[1]: http://catb.org/~esr/jargon/html/M/molly-guard.html

That's what I immediately thought of as well.

Presumably, Molly (Netflix's tool) is the kid that goes around and switches everything off. As in, it intentionally causes the problems Molly Guards protect from.

Reply on news.ycombinator.com