Hacker News Reader

Comment by tivert | Hacker News Reader

> 75% of TestGen-LLM's test cases built correctly, 57% passed reliably, and 25% increased coverage.

The problem I have with LLM generated tests is that it seems highly likely that they'd "ratify" buggy behavior, and I'd think that'd be especially likely if the code-base already had low test coverage. One of the nice things about writing new tests by hand is you've got someone who can judge if it's the system being stupid or if it's the test.

At a minimum they should be segregated in a special test folder, so they can be treated with an appropriate level of suspicion.

View on HN

Writing tests is indeed a great opportunity for finding bugs.

But a codebase with good test coverage allows you to safely perform large scale refactorings without having regressions and that's useful property even if you have bugs and the refactoring preserves them faithfully.

The risk of using a tool that generates tests designed to encode the current behaviour is that you may be lulled in a false sense of safety, while all you've done is to encode the current behaviour, as advertised.

Perhaps this problem can be just solved by not calling these tests "tests" but something like "behavioural snapshots" or something like that (cannot think of a better name, but the idea is to capture the idea that they were not meant to encode necessarily the correct behaviour but just the current behaviour)

https://en.wikipedia.org/wiki/Characterization_test

aka snapshot tests.

I'm a huge fan. But sometimes it's hard to produce the snapshots.

Sometimes you could replace snapshots with checksums. In other words, they're just alerts that code changed, or make changing code unnecessarily tedious.

Unit tests are dead weight during refactoring. Integration tests are useful.

Code coverage metrics are most easily met by writing unit tests. Unit tests are tedious to write.

If you have a robot writing unit tests, I do not want to see your codebase. Refactor all day long. I’m not going near it.

It is not at all uncommon for me to refactor the internals of a unit (ie, a class) in order to add a feature. Unit tests are extremely useful here.

Meta is specifically writing unit tests, where probably every part of the system except the code being tested is mocked.

Those tests tend to have very low value with testing significant refactoring, as the code the tests covers often won't exist anymore.

> One of the nice things about writing new tests by hand is you've got someone who can judge if it's the system being stupid or if it's the test.

This is an instance of a more general problem, which I call "the problem of unwanted change". If you have an automated system which can change itself, how do you know a change is actually intended/correct or merely a symptom of a bug, failure or imperfect knowledge the automation has?

That's why I think human supervision is always needed to an extent, to determine what scenario has occured.

This happens in all sorts of systems. And people tend to think they can solve this with just another layer of automation like here. Testing was originally invented as a way to check if the program works correctly. If you automate it, you will face the same problem, just with a bigger code (in the form of tests rather than assertions).

On the other hand, in a codebase where test coverage is poor and average tenure of engineers is ~1yr, a significant impediment is getting the initial test scaffold set up; maybe I don’t know how to build factories for all the incidental inputs required to test my code, but I do know how the code itself should behave.

If LLMs can help you to scaffold a test, and make it easy for you to write the business logic validation, I think that would be a big win.

On the other hand, if the generated tests are like most UTs, they will be over-coupled to the implementation and therefore slow down development. You might even see folks start deleting all the tests and regenerating them as part of a big diff, if it’s too hard to fix individual test cases that are failing due to coupling rather than any logic issue.

In sufficiently large systems there is some value in test that just detect changed behavior, even if the behavior is buggy. Parts of the code probably rely on the bugs and accidentally (or intentionally) fixing them can lead to more severe problems.

Of course these kinds of tests are no replacement for tests that check actual requirements.

> Parts of the code probably rely on the bugs

The combination of pieces is what should be tested then, not the behavior of each individual piece “just in case something needs it that way”. You’ll never be able to change anything with that approach. And why shouldn’t bugs be fixed anyways?

It’s not that bugs shouldn’t be fixed it’s that they should be fixed intentionally, and with awareness of what the fix might break so you can make it backwards compatible if necessary.

> LLM generated tests is that it seems highly likely that they'd "ratify" buggy behavior

For a new project or a project under active development, I agree that auto generating tests is probably a bad idea. But there are countless legacy systems with low coverage that are in maintenance mode. For those, generating tests that verify the current behavior is super useful. It lets someone make a change, and see that everything else stayed the same.

> you've got someone who can judge if it's the system being stupid or if it's the test.

but why couldn't this be done even with the llm generated test cases?

In that case, I think the point is the difference between what is LLM "assisted" tests (like say Copilot) vs. LLM "owned" tests.

If a test is worth having, it’s worth writing by hand.

Throw away mandatory code coverage tech debt instead of adding artificially-intelligent tech debt on top.

The best I can see a use for something like this is more like a linter than a test writer. “Robot, find weird things in the code and bring them to me for review.”

Then you write the tests yourself.

But this tool doesn’t know what is weird and what isn’t. What you’re talking about is something more like a fuzzer or static analyzer.

Yeah, but that means you can really live Hyrum's Law.

Keeping them separated would also improve future training.

I mean if I can use it to generate coverage for a method then prune the tests it will still save me hours.

Might also be useful when you want to refactor a legacy system or try to figure out the input space for a module, library or method.

Reply on news.ycombinator.com