Turmoil, a framework for developing and testing distributed systems
tokio.rs
tokio.rs
Overall, my experience has been positive. When it works, it's great. A pattern I've been following is to have a single fixture that returns a simulation of the system under a standard configuration, for example N replicas of an atomic register, so that each test looks like: 1. Modify the simulation with something like`turmoil::hold("client", "replica-1")`. 2. Submit a request to the server. 3. Make an assertion about either the response, or the state of the simulation once the request has been made. For example, if only some replicas are faulty, the request should succeed, but if too many replicas are faulty, the request / simulation should time-out.
One of the things I have found difficut is that when a test fails, it can be hard to tell if my code is wrong, or if I am using Turmoil incorrectly. I've had to do some deep-dives into the source in order to fully understand what happens, as the behavior sometimes doesn't line-up with my understanding of the documentation.
We've discussed improving the tracing experience, and even adding visualizations, but it hasn't been prioritized yet.
I came across Microsoft's Coyote project [2] a while back, which seems similar to this, though for C#. Has anyone here tried that out? If so, what have your experiences with it been like?
[1] Also known as any network ever.
I designed a series of workshops to teach these ideas internally at Microsoft. You can find the source code used in the workshop at https://github.com/microsoft/coyote-workshop - the repo can use better README files but sharing it nonetheless in case the setup helps inspire your own use of Coyote.
So first off, distributed systems use different software for different tasks. These will not be/were not written in Rust, and even if they were, they may not use Tokio (which is a library I could go into a rant about as well...).
Second, the very premise of the library, namely that it simulates the components and serializes everything into a single deterministic thread, is already wrong. Distributed system issues are difficult because they are non-deterministic, the faults happen rarely, and they exhibit Heisenbug behaviour: the more diagnostic you add to the system, the more likely you are to serialize the event sand therefore hide the bug. This library starts off with serialization, effectively hiding most if not all issues that can happen in a real system. What is the point of this?
So how would an actually useful distributed testing framework look like? It would be almost the exact opposite of turmoil: 1. It would handle an actual distributed system with different components written in different languages 2. It would inject payloads that are likely to tease out races/consistency violations 3. It would inject failures at various layers (network, storage etc) to test how the system recovers 4. It would provide useful diagnostics, and a way to reduce the failure scenarios to the extent possible (again, we're talking about non-deterministic failures, so this is not easy)
Obviously, that's not always true. Having a way to do deterministic testing in a single process is helpful.
> Distributed system issues are difficult because they are non-deterministic, the faults happen rarely,
You do not want non-determinism in your tests just because the real world non-determinism. How else are you going to write repeatable tests of scenarios involving concurrency?
You can build fuzzing on top of a deterministic simulator, allowing you to reproduce failures. This is a strategy that FoundationDB has used to great effect.
In the async scenarios as with Tokio and Flow(which I just learned about, cheers), the atomicity extends to the scheduler yields, which is already pretty good. It cannot test e.g. atomic memory operations/memfences etc but it can test in a fairly fine-grained way.
However, my original point still stands: you can only do this if you control all of the system within your particular scheduler. In real world scenarios you are more likely to encounter:
1. Some database written in C
2. Some storage layer maintained by AWS/Azure/GCP
3. Some messaging layer written in Go
4. Failover modes of the Kubernetes/Hashicorp stack
5. Your node.js webserver named "backend"
6. Some performance-sensitive component written in Rust
This sort of testing just does not apply
It is, but then you're not testing a distributed system.
> You do not want non-determinism in your tests just because the real world non-determinism. How else are you going to write repeatable tests of scenarios involving concurrency?
You can't, but you can get close. E.g. with fuzzing you can generate the scenarios with some ordering constraints, so if you replay the scenario a couple of times you are likely to reproduce the failure. It's a balancing act, too much serialization and your tests are useless, too little serialization and the state space becomes too large.
Folks interested in this space might also be interested in the system I spend most of my time working on: Shadow. It also performs deterministic simulation of a network of hosts, but it intercepts network and system interactions at the syscall level via seccomp. As such it can work with binaries compiled from ~any language, usually without any code modification or special compilation. https://shadow.github.io/
I found myself struggling with this in python with asyncio, is there something similar in that space?
Every time I wish I was using rust
> Testing distributed systems is hard. Non-determinism is everywhere (network, time, threads, etc.)
I assume that, at a very high level, what it does is: introduce random delay between components, give them slightly different fake times when they ask for system time, put them in different threads & perhaps introduce (additional) pre-emption etc., and anything else to simulate 'etc.'.