Aphyr and jepsen are a blessing both to breakdown software like these as to illustrate complex distribute system concepts with real world evidence. The dist sys class provided by Aphyr is amazing (https://github.com/aphyr/distsys-class)