Hacker News Reader

Comment by almondfestival | Hacker News Reader

I'm sure this method has evolved and/or been supplanted over the last 15 years, but one thing that struck me reading this is how much the dynamics of unit test coverage have changed in recent history, with AI-generated commits containing 10x as many unit tests (many of them kind of silly and tautological) as in the olden days. Gonna need to update some of those coefficients in their CRAP1 formula... Or maybe test coverage has/will become too noisy a parameter to use at all.

View on HN

Humans have been hand-crafting artisanal test cases of equal banality for years, often to meet some metric no one is verifying deeper than an automated report etc. I'm more inclined to believe the null hypothesis here, that the formula doesn't need any change and the signal is still equally valid. If anything changed, the signal is probably more useful now, since you'll never catch a good AI writing test cases that do literally nothing except call a function in a try catch with an empty catch block. (Models may have done that a few years ago but I haven't witnessed that behavior in a long time.) I've witnessed myriad examples of this in pre-AI production code.

Anecdotally, I’ve found that codebases that enforce code coverage metrics often have worse behavior coverage than ones that don’t.

It’s a classic example of Goodhart’s Law in action. Code coverage metrics only measure what percentage of code the test suite causes to run. But it’s very, very easy to write tests that run code without actually confirming that it produces correct output for all possible inputs. And it’s very, very easy to assume that a module with 90+% code coverage also has 90+% behavior coverage, and then become complacent about reviewing the suite for proper behavior coverage.

Djikstra seemed like he was mostly against testing. But only because he was for proofs. A unit test is a single example. The real way to demonstrate the absence of errors is to prove they aren’t there (vis a vis axioms and assumptions).

But most developers don’t have the mathematical sophistication nor the time.

It’s not that unit testing is useless. Just good to know what their limitations are and to plan your testing strategy accordingly.

You also have the problem of potentially having to re-verify everything by hand for every little change. Maybe fine for the kinds of projects Dijkstra was working on, but less practical in a business setting.

Tools like QuickCheck and Hypothesis are an interesting middle ground, though. I strongly prefer them over standard-issue unit testing for verifying algorithm implementations.

Hundred percent. All about trade-offs.

Although proof techniques such as proof repair have come a long way, it’s still impractical for a lot of scenarios.

TLA+ is great for systems design and such. Quick check style tests are awesome and a very low bar to clear from unit tests.

Yeah. And TLA+ can confirm that the design is sound, but it can’t confirm that the implementation conforms to the design. QuickCheck style tests can’t solve that problem, but perhaps they can mitigate it.

I think it might be possible to write a TLA+ parser and generate QuickCheck tests from it that will exercise invariants at least.

> Or maybe test coverage has/will become too noisy a parameter to use at all.

It already is, ive banned unit tests via ci checks from our codebases, they were not particularly useful before LLMs and now they are a net negative.

We require int and some e2es and that does all that units do and more.

I sometimes use unit tests to fill in gaps. I just caught a function with almost full coverage except for an if statement.

Playwright e2e tests are harder to maintain than pytest code, so I wrote a test just for that one case, and while I was at it, a few other input strings that seemed like they could be problems.

If the function had no test coverage at all, I probably would have tried to get it with an integration test, because that would mean the callers also weren't getting tested.

The occasional unit test is fine. But you do not need for example `test_helper_test.go` that is just ridiculous.

Its gotten so bad I decided to throw the baby out with the bathwater so to speak and focus on the high value places instead of trying adjudicate every individual usage

Are they slow and brittle?

I like having both. Unit tests are a decent indicator for local development. Fast, quick to iterate on. Integration tests are slow as molasses and I can’t iterate with them when it takes 20 minutes to setup the suite and run everything. Too slow. But still useful as you say.

They don't flake, generally can run a few hundred in 3-4 mins. Iteration times are definitely key.

Reply on news.ycombinator.com