Hacker News Reader

Comment by aftbit | Hacker News Reader

How does this deal with non-determinism from the outside world? For example, let's say one of my tests is flaky because it asks an external service to give it some data, and that external service is flaky in what it returns?

Or what if my bug is caused by bitflips in failing memory, that lead to impossible control flow paths being hit? Think something like:

    if x != 0:
        return 1/x

Failing with an error because x is 0.

Not hypothetical scenarios, both real bugs I've had to troubleshoot in my career.

View on HN

Bitflips are an interesting form of fault-injection that we could add:

https://antithesis.com/docs/applications/reliability/fault_i...

If you know somebody who will pay money for us to prioritize this feature, let me know! Otherwise, I'm sure we'll get to it eventually. We have all kinds of crazy ideas for new faults.

Communication with the outside world is something that we obviously have to ban. This means that all of the inputs to your system are being provided by our platform, and that any dependencies have to be mocked, or run inside the hypervisor with you.

In practice this can cause friction for people with a ton of dependencies. Some of the most common things we've already mocked (for example we have an entire fake AWS that we can run in there with you), and if your dependency is one of our existing customers, we can probably work something out...

How do y'all provide the fake AWS? Is it built in-house or are you running something like LocalStack?

i think they mention in a previous blog post with warpstream that they use localstack

The product does not appear to be a record-replay debugging product, it appears to be a precise fault injection test generation product.

In a record-replay debugging product, you want to reproduce the execution of your system to translate what occurred in reality into the debugging lab.

In this product, the goal appears to be creating a deterministic environment so that you can precisely inject non-determinism/faults to probe the response of your product to the environment.

The former is about analyzing bugs your test found, the latter is about creating tests to produce bugs. In that context, your question is unrelated to the product. It does not seek to reproduce flaky tests, it seeks to help invent new and exciting flaky tests.

You're sort of right, but we're actually sort of both. We use determinism to do controlled fault-injection and to explore your program's state space, but we can also use it for very powerful debugging. Look for future announcements about this.

Certainly. Once you have a deterministic environment, the ability to replay execution is basically free.

In fact, what has been described is basically the replay engine in a record-replay system which “runs” the recorded execution of a non-deterministic system in a deterministic mode to replay the exact same execution. As such, it retains the same benefits available to any replay. The key here is that you can “bypass” the record step since you are already running in the replay engine from the start.

Reply on news.ycombinator.com