Comment by spinningD20 | Hacker News Reader

Comment by spinningD20 | Hacker News Reader

Parent Full thread

The other issue I would point out is that building a database, while impressive with their quality, is still fundamentally different than an application or set of applications like a larger SaaS offering would involve (api, web, mobile, etc). Like the difference between API and UI test strategies, where API has much more clearly defined and standardized inputs and outputs.

To be clear, I am not saying that you can't define all inputs and outputs of a "complete SaaS product offering stack", because you likely could, though if it's already been built by someone that doesn't have these things in mind, then it's a different problem space to find bugs.

As someone who has spent the last 15 years championing quality strategy for companies and training folks of varying roles on how to properly assess risk, it does indeed feel like this has a more narrow scope of "bug" as a definition, in the sort of way that a developer could try to claim that robust unit tests would catch "any" bugs, or even most of them. The types of risk to a software's quality have larger surface areas than at that level.

There's a lot of assertions that I throw into business applications that would be very useful to test in this way. So I don't think this only applies to testing databases.

Also, when properties are difficult to think of, that often means that a model of the behavior might be more appropriate to test against, e.g. https://concerningquality.com/model-based-testing/. It would take a bit of design work to get this to play nicely with the Antithesis approach, but it's definitely doable.

Just to clarify, I am definitely not saying this is only useful or only applies to databases.

The point was more that, I don't see how this testing approach (at the level that it functions) would catch all of the bugs that I have seen in my career, and so to say "all of the bugs" or even "most of the bugs" is definitely a stretch.

This is certainly useful, just like unit tests, assertions, etc are all very useful. It's just not the whole picture of "bugs".

Yes, there are plenty of non-functional logic bugs, e.g. performance issues. I think this starts to drastically hone in on the set of "all" bugs though, especially by doing things like network fault injection by default. This will trigger complex interactions between dependencies that are likely almost never tested.

They should clarify that this is focused on functional logic bugs though, I agree with that.

I’d go so far as to say it is actually impossible to do UI testing in some kind of web based product unless it came from the browser makers themselves.

I’d settle for decent heap debugging.

Reply on news.ycombinator.com