Hacker News Reader

Comment by grebc | Hacker News Reader

[flagged]

Carl (author) here. This is a poor reading of the blog post. Notably, I built a general-purpose workload that simply exercises the write/checkpoint machinery. The exact same workload we use to break all kinds of transactional systems. This generic workload found the bug without any special knowledge of the bug, due to how Antithesis works.

I think the issue is that you knew there was a bug to be found in the write and checkpoint interactions, which then determined which workload you built.

My understanding is that this workload is very uncommon: the Tailscale blog says they used a custom unusual configuration to have many checkpoints like this. So without knowing the bug, it seems unlikely one would build this workload and then find the bug. If that makes sense.

Edit: just want to say that you being able to repro it is awesome, but that the overall claim seems a little overstated to me.

I can see how the post comes across that way. I may need to edit it to somehow be clearer about how I approached the workload development. I'll refer you to my other replies to comments for more information on my approach:

https://news.ycombinator.com/item?id=49278424 https://news.ycombinator.com/item?id=49278521

Also appreciate the nice words at the end :) I'm feeling a bit ganged up on.

I do feel like you're being ganged up on as well. IMO this is a very cool thing that Antithesis is able to find and reproduce this bug in this specific subsystem with a relatively generic workload.

I guess what would have been an even more cool thing is "we ran some more general testing with Antithesis, and it found five other bugs". Have you thought along those lines or explored something like that? There have to be other, similar bugs lurking in SQLite :)

I’m sorry you’re feeling ganged up on, I thought it was a nice article and cool you were able to repro the bug with your tools.

Like some others have mentioned, one of my earliest thoughts was “how much of a hint was the LLM given about the bug?” I think if the prompt used was stated clearly/verbatim near the beginning of the article, that would probably dispel a good amount of the criticism.

I read your ad.

It states you replicated the bug once the SQLite team fixed it, and published it.

Not sure what’s difficult about replicating behaviour when it’s spelled out for you.

this is a deliberate mischaracterisation of the content of the article

You’re mischaracterising an ad, that’s also funnily enough full of it.

Okay, so use the same methodology to find a new, unknown bug? That'd be actually impressive.

As-is, this is just 20/20 hindsight with concerns about leading the AI on through the prompt hand-waved away. Come on.

As the post (and linked repo) pretty clearly indicate, zero info about the existence of the bug went into this work.

Oh btw, we also found some other ones… Stay tuned!

So you found this before SQLite published the fix?

Reply on news.ycombinator.com