Debugging in the Multiverse
antithesis.com
antithesis.com
The closest I've seen is Qemu record/replay, but that's very slow (no KVM acceleration, no multicore), and broken in current Qemu versions (replayed system just gets stuck).
Essentially, it involves a series of sci-fi concepts, and then showing the kind of program (in modified perl) that someone might write to take advantage of those capabilities.
Perhaps the language is too small a vantage point to really get into what’s happening when debugging.
I don't know to what degree this is true for other language teams but one thing I've observed is that language designers, compiler people, VM people, and IDE/debugger people have more distinct cultures than you might expect. That can make it hard to ship features that cut across those domains. I think we've gotten a lot better at doing that kind of holistic design on the Dart team, but it took years of team-building to get there.
I sometimes wonder if this sort of determinism is the sort of thing that is either designed in from the start of a system/PL, or you need near-hardware level control (like Antithesis).
No language will prevent you from misimplementing the specs, but languages can be designed in such a way that it easy to trace back why the button is green and not red.
It seems like those who are the most serious about debugging are from the video game industry. They get all the cool stuff with time travel, hot reload, etc... So much that I expected to see something about video games, and was surprised it wasn't.
And Antithesis is (theoretically) one solution to that, which is very neat!
(I know modern games are also often operating in a distributed environment, but to generalize very broadly, there’s a lot more happening “in one place”)
If the spec is written in the language itself, then some languages certainly will.
See Lean, Rocq, Isabelle, etc
A little bit. The big thing that others are missing is that it's basically impossible for a PL to accomplish this. Antithesis is basically recording all the state including I/O, network I/O, all RNGS (including the OS) and the big one which everyone has trouble with which is time. So basically you don't need to set up your code and how it interfaces with its environment to be deterministic - you can run within a deterministic container instead which flips the problem on its head and makes it much easier. I'm sure there are tradeoffs. A noteable one is how expensive and slow this approach is vs making your code deterministic. But given how basically no one bothers to make their code deterministic and this is a drop-in solution for scenarios like that, it's really worth it. Additionally, unlike approaches like rr which offer similar capabilities, this is even more generic & not dependent on adding support for every OS interface (e.g. rr doesn't support io_uring yet but I believe antithesis would since it's running at the VM level)
(I work at Antithesis, if youre interested in chatting more once this thread has gone cold come join discord.gg/antithesis)
I don't really see a fit for the automated testing product in our stack at the moment, but I would love to use a time traveling hypervisor that I can hop into whenever I'd like.
Currently, it seems your pricing is pretty focused on the automated testing service. Do you have pricing or plans that offer just the deterministic dev environment?
If you need something else, reach out and ask us about it, because we have a few of them in the pipeline.
Is this kind of stuff only possible in an Antithesis Environment?
10 bucks says you get complaints for not implementing the "real world" feature.
> I’ve been involved in too many production outages and emergencies whose aftermath felt just like that. Eventually all the alerts and alarms get resolved and the error rates creep back down. And then what? Cordon the servers off with yellow police tape? The bug that caused the outage is there in your code somewhere, but it may have taken some outrageously specific circumstances to trigger it.
So practically, if a production outage (where I think "production" means it cannot be in a simulated environment, since the customers you're serving are real) is caused by very specific circumstances, and your production system records some, but not every attribute of its inputs and state ... how does one make use of antithesis? Concretely, when you have a fully-deterministic system that can help your investigation, but you have only a partial view of the conditions that caused the bug ... how do you proceed?
I feel like this post is over-promising but perhaps there's something I just don't understand since I've never worked with a tool set like this.
I think you're right that the framing leans towards providing value in prod issues, but we left out how we provide value there. I think you're also right that we're just used to experiencing the value here, but it needs some explanation.
Basically this is where guided, tree-based fuzzing comes in. If something in the real world is caused by very specific circumstances, we're well positions to have also generated those specific circumstances. This is thanks to parallelism, intelligent exploration, fault injection, our ability to revisit interesting states in the past with fast snapshots, etc.
We've had some super notable instances of a customer finds a bug in prod, recalls its that weird bug they've been ignoring that we surfaced a month ago, and then uses this approach to debug.
The best docs on this are probably here: https://antithesis.com/docs/introduction/how_antithesis_work...
> When we succeed at this, we collect huge volumes of logs “just in case” they provide some crucial clue, incurring equally huge storage costs.
The 'packets from the past' section says we can just retroactively decide what we should have recorded.
Doesn't that mean we're effectively recording everything always? What's the cost of this? Or is all of this under the assumption that we never have to debug something that happened outside of the simulation environment, e.g. in response to an actual in-bound request from a customer? If this is just saying we can afford to save everything in our development environment ... well in that context recording the logs probably wasn't a "huge storage cost" either, right? Or am I missing something basic here?
In fact, we just made a radical upgrade to this functionality. Expect a blog post about that soon.
This is just a standard replay engine from what I can tell.
It also is not in any sense a replay engine. We don’t need to record anything except the inputs!
This is exactly a replay engine (or I guess you could say replay engines are deterministic simulators). How do you think you replay a recording except with a deterministic execution system that injects the non-deterministic inputs at precise execution points? This is literally how all replay engines work. Furthermore, how do you think recordings work except by recording the inputs? That is literally how all recording systems designed to feed replay engines work. The only distinction is what constitutes non-determinism in a given context. At the whole hypervisor level, it is just I/O into the guest; at the process level, it is just system calls that write into the process; at the threading level, it is all writes into the process. These distinctions are somewhat interesting at a implementation level, but do not change the fundamental character of the solution which is that they are all a replay engine or deterministic simulator, whatever you want to call it.
[0]: https://www.usenix.org/conference/enigma2016/conference-prog...
[1]: https://qira.me/
We don't run this on your production server, but in the same simulation that we use to find your bugs. See also: https://antithesis.com/product/how_does_antithesis_work/
If this is prod, your job is going to be finding what combination of these things caused it this time.
How does this tooling deal with that?
The "onto" direction doesn't really matter.