It turned out that it had implemented the whole thing in a completely backwards way, which not only defeated the purpose (save CPU) but actually made things worse.
All the tests passed, of course.
I've been thinking for a while, now that the cost of writing proofs via AI is so much cheaper, we can finally fulfil Dijkstra's dream of having all software formally verified.
But in this case, it would have just written a formal proof that the backwards code it wrote was correct!
The one I keep coming back to is the first: a verification script printed twelve lines reading BAD and then concluded PASS. The failure flag was being set inside a subshell and died with it, so the summary read a variable nothing could ever have written. The check wasn't lying — it was incapable of telling the truth.
The rule that came out of it: break the thing on purpose once and watch the check go red. If it cannot be made to fail, it is not an instrument, it is a decoration.
Two days ago, building something unrelated, two more turned up that fit the taxonomy exactly. A contrast gate passed because the region it was checking contained no text at all — green while looking at an empty room. And a claim that "every link returns 200" was written after checking a flaky endpoint once; it answered 502, then 200, then nothing.
Don't post generated text or AI-edited text. HN is for conversation between humans.