Testing Distributed Systems with Deterministic Simulation (2014) [video] | Hacker News Reader

Testing Distributed Systems with Deterministic Simulation (2014) [video]

youtube.com

54 pointssensible1238 comments

My experience is that this is difficult without the system under test being designed to operate deterministically.

So for example: the system clock needs to be replaceable with a fake, or the current time needs to be injected, or perhaps both.

Another example: Go channels. These are a real PITA to surface in a way that allows a test harness to drive them deterministically.

A while ago I built a simulator testing harness for an early version of the Knative autoscaler[0]. I eventually ran aground on the difficulty of keeping up with changes, but it may be rearchitected and repurposed to assist Kubernetes HPA development instead.

FoundationDB's approach is to pretty much bake optional determinism into the language with some macro magic. It makes me wish that were easier to do in other languages.

[0] https://github.com/pivotal/skenario

In the FDB simulator case, every event has a future time at which it will fire, and the clock advances to that time when the event is dequeued.

This also has the side benefit of artificially “speeding up time”, which is useful when many actions happen based on long timeouts.

I used the same approach, which I learned from DES textbooks. I wrote up the design here: https://github.com/pivotal/skenario/blob/master/docs/concept...

WebAssembly allows deterministic execution (with caveats: https://github.com/WebAssembly/design/blob/master/Nondetermi...) of any code. I think it is a perfect platform for such type of testing.

I used to have a similar opinion, that it requires a special language to leverage such power. But recently I've discovered that Rust, with its latest advancement in async/await design together with swappable runtime, can provide a decent story for building determinism tests like described in the video. Yes of course one still needs to mock all the IO as well as clock part, but the deterministic core is already there, and with a carefully designed runtime and surrounding library, we might have pretty good support in Rust for this.

Disclaimer: I'm never a Rust zealot, I do understand Rust has its tradeoffs, and it's not the panacea for every problem. In fact I do have side projects which are perfectly suited and written in Go. I'm just saying Rust's design turns out to be suitable for such a deterministic testing structure, and Rust's target for system programming, can also benefit A LOT from this style of testing.

And also a shameless plug: I do have some initial work exploring this area: https://github.com/xxuejie/diviner. It's still quite rough and a lot of work is needed but I do believe this is something that is worth exploring.

after the jepsen/mongodb post that hit front page nearly at the same time, it seems like the foundationdb team has the exact opposite values regarding resilience and data safety guarantees. I cant wait for the first results of the couchDb rewrite on foundationDb

Discussed a bit at the time: https://news.ycombinator.com/item?id=8345030

This is similar to https://github.com/microsoft/coyote/

Reply on news.ycombinator.com