Hacker News Reader

Comment by rafterydj | Hacker News Reader

I see this get mentioned a lot but I still am skeptical that AI can generate tests we can trust more than any other code we know we cannot trust.

Yes tests are conceptually isolated and that helps, but I've personally seen unit tests get generated that are semantically incorrect - that is, they test the structure of the code (e.g. they can check function output types and values), but they can't know _why_ the unit tests need to be there, so the really really helpful tests never get generated. Not to mention the obvious issues with generated tests only testing is x = x, or needless redundant tests for the same thing, or them essentially testing basic features of the language.

View on HN

You have to iterate on the tests, review and validate them, just like any other code, and if you generate a whole project's tests all at once the quality is abysmal, of course. I've been using a lot of old school data-driven testing techniques, where the harness is just code I review, and the data itself is e.g. json files and drives the system.

I actually have a public (AGPL) example here: https://github.com/pgdogdev/pgdog/tree/main/integration/sql - pgdog is particularly testable since it is trying for complete transparency, so you have a perfect oracle in hand via base postgresql, but it demonstrates the concept at least.

Then this falls into the exact same pit the OP mentioned, either you need to blindly trust that the LLM is generating tests that actually work, or you need extensive test coverage for your tests to ensure that your tests are actually testing.

It turns out that you don't actually need tests for your tests, because the code provides a baseline truth for the tests. You do, at some point, have to be epistemically sound enough to actually look for correctness in either the code, behavior, or tests. We unfortunately haven't fully unlocked completely solipsistic value generation yet.

This is also part of why I like end to end tests that use actual UI flow, so I can watch it go by in slow mode before letting it loose fully automated.

Maybe it's because I haven't had my coffee yet, but I cannot understand what you are saying.

What do you mean by "be epistemically sound enough"?

You are using it as if to say "if your code is grounded in sound abstractions, you'll be fine and tests will therefore generate successfully" but preface that claim with "the code provides a baseline truth for the tests". The latter does not follow from the former, and it also does not lift the burden of responsibility away from the programmer - which is where my doubts on test generation stem from in the first place.

Additionally, what is "completely solipsistic value generation"?

You reference it like a perk in a skill tree, but to my ears "generating completely solipsistic values" seems like a way of describing AGI with a philosophical wording instead of just saying AGI.

I mean that your code has to accomplish something in the real world that is verifiable on a human level. It has to let customers get something done, or trade resources via a market, or something. That requires that it have some basis in reality that provides a ground truth about whether the system is working or not, and that's what gives you feedback that drives your tests and design.

You: >>> You want comprehensive tests at every level, far more than is reasonable for a human to build or maintain

Also you: > You have to iterate on the tests, review and validate them

Yes, "maintain" is not quite the same as "review", but the line is veeery fine. I find it really tiring to review masses of tests that an agent spews out.

Especially because I know what it has a tendency to write irrelevant/vacuous/useless tests. It's insane the amount of times I have told Codex to "write a test that reproduces the reported bug, SEE THE TEST FAIL, then implement a fix", only for it to guess an irrelevant test, not run it to see it fail, and implement a code change that has nothing to do with either the test or the actual bug.

Which is why test generation has to be carefully guided as well, and this is something at which I've incidentally been fast. Ultimately it's a constant battle between LLM handholding and doing things yourself.

I don't even care about tests being correct as you can still verify them even when tedious. What I care is that, more often than not, the shape of the solution is not fixed. Having unit tests for those can be extremely costly as when the changes happens, you have to change all the tests.

I've been burned by this in my honeymoon period with unit testing (pretty much the reason it ended). These days, I prefer broader scope of testing, especially user-facing part. The users may be other developers or end users. I only do unit testing for tricky algorithms or math formulae.

I want all the layers of the pyramid, eventually, but the top layers matter the most. I can't count the number of times my paranoid "make sure that customers can successfully pay us" end to end test suite has prevented the money faucet from being shut off. I install one perennially at any company I work at and they always pay for themselves surprisingly quickly.

I’ve been involved in B2B (so no payment flow). But it’s basically the same with an handful of integration tests for common workflows. They run fast and mostly serve a canary to ensure that we are not crippling some use cases. When a bug hits us, a test case is added/modified for it.

They’re mostly a reflection of the current requirement of the project.

AI shouldn't write tests. At least not all of them. Definitely not e2e's. The tests should be guardrails to constrain agents. This way, the author of code matters less.

Reply on news.ycombinator.com