Hacker News Reader

Comment by leeoniya | Hacker News Reader

> "no no, it has full test coverage"

i don't have enough fingers (and toes) to count how many times i've demonstrated that "100% coverage" is almost universally bullshit.

View on HN

Codex is freakin hot-to-trot to churn out test coverage for every single thing it implements, and some of it is very esoteric and highly prescriptive (regexes for days) BUT .. after a while, it dawned on me that LLM-driven test coverage is less about proving “code correctness” (you’re better off writing those tests yourself alongside them), and more about just trying to ensure that whatever gets bolted on stays bolted on. For better or worse, obviously, since if you bolt on trash, trash you shall have.

There's a very old paper by Cem Kaner about the meaninglessness of "100% coverage" where he included an appendix where he enumerates 101 different possible types of code coverage: https://www.researchgate.net/publication/243782285_Software_...

Wholeheartedly agree, but in fairness, I trust the tests of the best AI models more than those of the average human developer. There's a lot of people around that combine high diligence with complete intellectual laziness, producing tons of useless tests.

Actually no, cancel that. I realise now that I trust AIs more than the average developer, period. At this point they do produce better code than most people I've dealt with.

Reply on news.ycombinator.com