required reading for anyone using distributed systems: http://aphyr.com/tags/Jepsen

in particular:

http://aphyr.com/posts/284-call-me-maybe-mongodb

http://aphyr.com/posts/315-call-me-maybe-rabbitmq