Automated Unit Test Improvement Using Large Language Models at Meta

Only goes to confirm my view of them that the language is deficient

They knew that people would write coverage tests for getters and setters, and calculated that eventuality into their minimums.

Various people objected to this, pointing out that 100% test coverage tells you nothing about whether the tests are any good. Our lead (wisely, IMO) responded that they were correct - 100% tells you nothing - but that _any other percentage_ does tell you something.

- stryker-mutator (C#, Typescript)

- pitest (Java)

- mutatest (Python)

The result was a header with:

    static const unsigned ONE = 1;
    static const unsigned TWO = 2;
    static const unsigned THREE = 3;
   ... 

To me that reads like your team fucked up at a very fundamental level, as they both failed to take into account the whole point of automated tests and also everyone failed to flag those nonsense tests as a critical blocker for the PR.

Unless your getters aand setters are dead code, they are already exercised by any test covering the happy path. Also, a 80% coverage target leaves out plenty of headroom to leave out stupid getter/setter tests.

A team that pulls this sort of stunt is a team that has opted to develop defensive tricks to preserve their incompetence instead of working on having in place something that actually benefits them.

https://en.wikipedia.org/wiki/Perverse_incentive#The_origina...

It comes up a bit on hacker news https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

I have never seen such poorly written application in my 6 years of experience (and I am not only talking about style, stuff was absolutely utterly broken, while they had no clue what's wrong).

I hate Sonar with passion. It only ever should be used to report vulnerabilities, not telling me to rename variables or that i "should refactor this code duplication!" I already have a fucking backlog with Jira tickets, don't tell what i am supposed or not to do, and when i am supposed to do it.

But oh boy mangers love this stupid power burner

Classic

I attended many TOC conferences in the 90s and early 2000s. Eli Goldratt was famous for saying "Tell me how you'll measure me, and I'll tell you how I will behave."

The problem is, the ugly, careless code extremely very well tested: 95% code coverage. My replacement had 100% code coverage... but by being far shorter, my PR couldn't pass tests, as total coverage went down, not up. The remaining code in the repo? A bunch of Swing UI code, the kind that is hard to test, and where the test don't mean anything. So facing the prospect of spending a week or two writing swing test, the dev lead decided that it was best to keep the old code somewhere in the repo, with tests pointing at it, just never called in production.

Thousands of lines of completely dead, but very well covered code were kept in the repo to keep Sonar happy.

A big problem is making it mandatory with huge bureaucracy to avoid its stupidness. Just last week I was battling yet another code quality tool they made mandatory: it was complaining that my res.status(200).json() wasn't setting up HSTS headers. And then I tried setting it up manually, it kept complaining, app.use(helmet()), same thing. Apparently it wanted me to write the whole backend code into a single file for it to stop complaining. And of course, HSTS is much more elegantly and automatically handled by the ingress or load balancer itself.

I could have spent a week or two flagging it as a false positive and explaining what is HSTS for upper management to approve it. I ended up just adding a res.sendJson(data, status = 200) to the prototype of the response object. Which is obviously stupid, but working in a bureaucracy-heavy sector has made me realize how much of bad software is composed of many such bad implementations combined.

Something I've learned along the way as well. A few times in my career I will end up working under a manager that insists only work "that can be explicitly measured" be performed. That means they forbade library upgrades, refactors, things like that because you couldn't really prove an improvement in customer metrics or immediate changes in eng productivity.

I've also been at companies that follow that mantra more broadly and apply it to eng performance reviews. The entire culture turns into engineers focusing on either short term gains without regard for long term impact, or gaming metrics for meaningless changes and making them look important and impactful.

Important but thankless work gets left behind because, again, it's work that is not "immediately measurable." End result is a bunch of features customers hate (eg dark patterns), and a rickety codebase that everyone is disincentivised fix or improve.

My personal favorite was tests that a team member introduced for an object that had all the default runtime parameters in it. Think like how long should a timeout be by default, before getting overridden by instance specific settings. This team member introduced a test that checked that each value was the same as it was configured to.

So if I wanted to update or add a default, I had to write it in two places, the actual default value, and the unit test that checked if the defaults were the same as the test required.

Career QA here. E2E are the absolute highest pevel of test, that take the most time to implement, have the most dependencies, and tell you the least about what is actually wrong.

If you think finding what broke is painful with full suites of unit/integration tests under the E2E suite is bad, throw those out or stop maintaining them at all. Let me know how it goes.

Ideally, the code coverage tool would have heuristics to detect trivial getter/setter methods, and filter them out, so adding tests for them won't improve code coverage. Non-trivial getters/setters (where there is some actual non-trivial logic involved) shouldn't be filtered, since they should be tested.

Although, there is room for debate about what counts as trivial. Obviously this is trivial:

    public void setUser(User user) {
       this.user = user;
    }

    public void setUser(User user) {
       this.user = Objects.requireNonNull(user);
    }

    public void setOwners(List<User> owners) {
       this.owners = List.copyOf(owners);
    }

However we might think about this differently if we flipped it to "our standard is 20% completely untested".

Uncoverage communicates the value much better.

With Java at least, it seems to drive people to do things like not catch exceptions. It's hard to inject errors so all the catch blocks are hit.

Also makes people not want to switch to use record classes, as that feature removes class boilerplate that is easy to cover.

but why couldn't this be done even with the llm generated test cases?

Might also be useful when you want to refactor a legacy system or try to figure out the input space for a module, library or method.

This is an instance of a more general problem, which I call "the problem of unwanted change". If you have an automated system which can change itself, how do you know a change is actually intended/correct or merely a symptom of a bug, failure or imperfect knowledge the automation has?

That's why I think human supervision is always needed to an extent, to determine what scenario has occured.

This happens in all sorts of systems. And people tend to think they can solve this with just another layer of automation like here. Testing was originally invented as a way to check if the program works correctly. If you automate it, you will face the same problem, just with a bigger code (in the form of tests rather than assertions).

But a codebase with good test coverage allows you to safely perform large scale refactorings without having regressions and that's useful property even if you have bugs and the refactoring preserves them faithfully.

The risk of using a tool that generates tests designed to encode the current behaviour is that you may be lulled in a false sense of safety, while all you've done is to encode the current behaviour, as advertised.

Perhaps this problem can be just solved by not calling these tests "tests" but something like "behavioural snapshots" or something like that (cannot think of a better name, but the idea is to capture the idea that they were not meant to encode necessarily the correct behaviour but just the current behaviour)

Of course these kinds of tests are no replacement for tests that check actual requirements.

For a new project or a project under active development, I agree that auto generating tests is probably a bad idea. But there are countless legacy systems with low coverage that are in maintenance mode. For those, generating tests that verify the current behavior is super useful. It lets someone make a change, and see that everything else stayed the same.

Throw away mandatory code coverage tech debt instead of adding artificially-intelligent tech debt on top.

The best I can see a use for something like this is more like a linter than a test writer. “Robot, find weird things in the code and bring them to me for review.”

Then you write the tests yourself.

If LLMs can help you to scaffold a test, and make it easy for you to write the business logic validation, I think that would be a big win.

On the other hand, if the generated tests are like most UTs, they will be over-coupled to the implementation and therefore slow down development. You might even see folks start deleting all the tests and regenerating them as part of a big diff, if it’s too hard to fix individual test cases that are failing due to coupling rather than any logic issue.

Something with low coupling, high cohesion, and low complexity should be easy to unit tests.

Of course, this doesn't seem like it's going to replace engineers, but it'll help the organization out for relatively low cost.

> 75% of test cases built correctly, 57% passed reliably [implying test cases by context], and 25% increased coverage [same implication]

The actual report talks about test classes, where each class has one or more test cases.

> (1) 75% of test classes had at least one new test case that builds correctly.

> (2) 57% of test classes had at least one test case that builds cor- rectly and passes reliably.

> (3) 25% of test classes had at least one test case that builds cor- rectly, passes and increases line coverage compared to all other test classes that share the same build target.

Those are two very different statements. They even have a footnote acknowledging this:

> For a given attempt to extend a test class, there can be many attempts to generate a test case, so the success rate per test case is typically considerably lower than that per test class.

But then in their conclusion they misrepresent their findings again, like the abstract:

> When we use TestGen-LLM in its experimental mode (free from the confounding factors inherent in deployment), we found that the success rate per test case was 25% (See Section 3.3). However, line coverage is a stringent requirement for success. Were we to relax the requirement to require only that test cases build and pass, then the success rate rises to 57%.

There's almost always more business logic to what a unit should do than it's types though. Depending on the language, the type system can only encode so much of that logic.

Consider the opposite: can't the compiler generate implementations from types and interfaces? In most cases, no. LLMs are filling some of that gap though because they can use some surrounding context to return the high probability implementation (completion) from the interface or type definition.

[1] https://en.m.wikipedia.org/wiki/Fuzzing