The following link shows how we do Chaos Engineering in TiDB, an open source distributed database:

https://www.pingcap.com/blog/chaos-practice-in-tidb/

Regarding the Fault Injection tools we are using:

- Kernel Fault Injection, the Fault Injection Framework included in Linux kernel, you can use to implement simple fault injections to test device drivers.

- SystemTap, a scripting language and tool diagnose of a performance or functional problem.

- Fail, gofail for go and fail-rs for Rust

- Namazu: a programmable fuzzy scheduler to test a distributed system.

We also built our own Automatic Chaos platform, Schrodinger, to automate all these tests to improve both efficiency and coverage