[1]: https://skiplabs.io
Very cool to see a team use Jepsen for super early pre-release testing of the system.
I wonder if you wish you had waited for the runtime to be a bit more stable, or you feel this was already well worth the effort, even with some of the identified failures being in "known incomplete" areas? (I could see either side of the argument - waiting longer might give you more valuable failures, but testing early gives you a chance to catch problems before they become baked into the foundation and become more difficult to fix...)
Another tool that feels like sci-fi to me any time I hear a mention of it, is Antithesis [1] - written by the people who built FoundationDB. Could be another interesting integration to investigate in the future to help bulletproof the language runtime?
I would suggest against this kind of integration test when the data model or API are in constant flux, because then you have to re-write or even re-design the test as the API changes. Small changes--adding fields or features, changing HTTP paths or renaming fields--are generally easy to keep up with, but if there were, say, a redesign that removed core operations, or changed the fundamental semantics, it might require extensive changes to the test suite.
The plan is to engage with Jepsen again once we have a system that passes the current suite, expand the test surface even further, and continue iterating until we are satisfied with the results. There won't be a public release before that is true.
Working with Jepsen also sparked a couple other interesting ideas, like building a Python language fuzzer to ensure that many shapes of Python programs work as intended in Capela. That's something we would love to do in the future.
Re: Antithesis - absolutely. Kyle mentioned them to us and we think it will be a very interesting product for us to adopt to further ensure we're delivering a reliable product.