Using LLMs to enhance our testing practices

> Using an autocomplete to "bang it out" seems foolish.

Based on my own experience, I find the widespread scepticism on HN about AI-assisted coding misplaced. There will be corner cases, there will be errors, and there will be bugs. There will also be apps for which AI is not helpful at all. But that's fine - nobody is saying otherwise. The question is only about whether it is a _significant_ nett saving on the time spent across various project types. The answer to that is a resounding Yes.

The entire set of tests for a web framework I wrote recently were generated with Claude and GPT. You can see them here: https://github.com/webjsx/webjsx/tree/main/src/test

On an average, these tests are better than tests I would have written myself. The project was written mostly by AI as well, like most other stuff I've written since GPT4 came out.

"Using an autocomplete to bang it out" is exactly what one should do - in most cases.

Fully agreed.

It's bad enough when human team members are submitting useless, brittle tests with their PR's just to satisfy some org pressure to write them. The lazy ones provide a false sense of security even though they neglect critical scenarios, the unstable ones undermine trust in the test output because they intermittently raise false negatives that nobody has time to debug, and the pointless ones do nothing but reify architecture so it becomes too laborious to refactor anything.

As contextually aware generators, there are doubtless good uses for LLM's in test developement, but (as with many other domains) they threaten to amplify an already troubling problem with low-quality, high-volume content spam.

Mostly agree.

My first thought when I read this post was: Is his goal to test the code, or validate the features?

The first problem is he's providing the code, and asking for tests. If his code has a bug, the tests will enshrine those bugs. It's like me writing some code, and then giving it to a junior colleague, not providing any context, and saying "Hey, write some tests for this."

This is backwards. I'm not a TDD guy, but you should think of your test cases independent of your code.

I subscribe to the concept of the "pyramid of tests" - lots of simpler unit tests, fewer integration tests, and very few end-to-end tests. I find that using LLMs to write unit tests is very useful. If I just wrote code which has good naming both for the classes, methods and variables, useful comments where necessary and if I already have other tests which the LLMs can use as examples for how I test things, I usually just need to read the created tests and sometimes add some test cases, just writing the "it should 'this and that'" part for cases which weren't covered.

An added bonus is that if the tests aren't what you expect, often it helps you understand that the code isn't as clear as it should be.

Pretty much this and I prefer the opposite. "Here's the new test case from me, make the code pass it" is a decent workflow with Aider.

I get that occasionally there are some really trivial but important tests that take time and would be nice to automate. But that's a minority in my experience.

> "More tests" is not the goal - you need to write high impact tests, you need to think about how to test the most of your app surface with least amount of test code.

Are there ways we can measure this?

One idea that I’ve had, is collect code coverage separately for each test. If a test isn’t covering any unique code or branches, maybe it is superfluous - although not necessarily, it can make sense to separately test all the boundary conditions of a function, even if doing so doesn’t hit any unique branches.

Maybe prefer a smaller test which covers the same code to a bigger one. However, sometimes if a test is very DRY, it can be more brittle, since it can be non-obvious how to update it to handle a code change. A repetitive test, updating it can be laborious, but at least reasonably obvious how to do so.

Could an LLM evaluate test quality, if you give it a prompt containing some expert advice on good and bad testing practices?

There is an art to writing tests especially getting absraction levels right. For example do you integration test hitting the password field with 1000 cases or do that as a unit test, and does doing it as a unit test sufficiently cover this.

AI could do all this thinking in the future but not yet I believe!

Let alone the codebase is likely a mess of bad practice already (never seen one that isn't! That is life) so often part of the job is leaving the campground a bit better than how you found it.

LLMs can help now on last mile stuff. Fill in this one test. Generate data for 100 test cases. Etc.

Great point on focusing on high-impact tests. I agree that LLMs risk giving a false sense of coverage. Maybe a smart strategy is generating boilerplate tests while we focus on custom edge cases.

>Sometimes I spend more time on the test code than the actual code (probably normal).

This seems like the kind of thing that should be highly dependent on the kind of project one is doing, if you have an MVP and your test code is taking longer than the actual code then it is clear the test code is antagonistic to the whole concept of an MVP.

Detecting regressions is the goal. If LLMs can do that for free to cheap, that’s good. It doesn’t have to be complicated.

Totally agree, especially about the need for well-architected, high-impact tests that go beyond just coverage. At Loadmill, we found out pretty early that building AI to generate tests was just the starting point. The real challenge came with making the system flexible enough to handle complex customer architectures. Think of multiple test environments, unique authentication setups, and dynamic data preparation.

There’s a huge difference between using an LLM to crank out test code and having a product that can actually support complex, evolving setups long-term. A lot of tools work great in demos but don’t hold up for these real-world testing needs.

And yeah, this is even trickier for higher-level tests. Without careful design, it’s way too easy to end up with “dumb” tests that add little real value.

This latest update to Sonnet is super impressive.

We are really already past the point of being able to discuss these matters though in large groups.

The herd speaks as if all LLMs on all programming languages are basically the same.

It is an absurdity. Talking to the herd is mostly for entertainment at this point. If I actually want to learn something, I will ask Sonnet.

All this makes me think making software engineers redundant is really the "killer app" of LLM's. This is where the AI labs are spending most of the effort - its the best marketing after all for their product - fear sells better than greed (loss aversion) making engineers notice and unable to dismiss it.

Despite some of the comments on this thread, despite it not wanting to be true, I must admit LLM's are impressive. Software engineers and ML specialists have finally invented the thing which disrupts their own jobs substantially either via large reduction in hours and/or reduction in staff. As the hours a software engineer spends coding diminishes by large factors so too especially in this economy will hours spent required paying an engineer will fall up to the point where anyone can create code and learn from an LLM as you have just done. Once everybody is special, no one is and fundamentally employment, and value of things created from software, comes from scarcity just like everything else in our current system.

I think there's probably only a few years left where software engineers are around - or at least seen as a large part of an organization with large teams, etc. Yes AI software will have bugs, and yes it won't be perfect but you can get away with just one or two for a whole org to fix the odd blip of an LLM. It feels like people are picking on minor things at this point, which while true, for a business those costs are "meh" while the gains of removing engineers are substantial.

I want to be wrong; but every time I see someone "learning from LLM's", saving lots of time doing stuff, saving 100's of hours, etc I think its only 2-3 years in and already its come this far.

You should NEVER modify generated code. All of our generated code is pretended with a big comment that says "GENERATED CODE DO NOT MODIFY. This code could be regenerated at any time and any changes will be lost."

If you need to change behaviour of generated code you need to change your generator to provide the right hooks.

Obviously none of this applies to "AI" generated code because the "AI" generator is not deterministic and will hallucinate different bugs from run to run. You must treat "AI" generated code as if it was written by the dumbest person you've ever worked with.

I agree. Adding unit tests without a good reason comes at a cost.

Refactoring is harder, especially if it's not clear why a test is in place. I've seen many developers disable tests simply because they could not understand how, or why, to fix them.

I'm hopeful that LLMs can provide guidance in removing useless tests or simplifying things. In an ideal future they may even help in formulating requirements or design documentation.

Just ask it to do both.

Author here: Yes, there are certain functions where writing good tests will be difficult for an LLM, but in my experience I've found that the majority of functions that I write don't need anything out of the ordinary and are relatively straightforward.

Using LLMs allows us to have much higher coverage than if we didn't use it. To me and our engineering team, this is a pretty good thing because in the time prioritization matrix, if I can get a higher quality code base with higher test coverage with minimal extra work, I will definitely take it (and in fact it's something I encourage our engineering teams to do).

Most of the base tests that we use were created originally by some of our best engineers. The patterns they developed are used throughout our code base and LLMs can take these and make our code very consistent, which I also view as a plus.

re: Complacency: We actually haven't found this to be the case. In fact, we've seen more tests being written with this method. Just think about how much easier it is to review a PR and make edits vs write a PR. You can actually spend your time enforcing higher quality tests because you don't have to do most of the boilerplate for writing a test.

I do use LLMs to bootsrap my unit testing (because there is a lot boilerplate in unit tests and mocks), but I tend to finish the unit tests myself. This gives me confidence that my tests are accurate to the best of my knowledge.

Having good tests allows me to be more liberal with LLMs on implementation. I still only use LLMs to bootstrap the implementation, and I finish it myself. LLMs, being generative, are really good for ideating different implementations (it proposes implementations that I would never have thought of), but I never take any implementation as-is -- I always try to step through it and finish it off manually.

Some might argue that it'd be faster if I wrote the entire thing myself, but it depends on the problem domain. So much of what I do is involve implementing code for unsolved problems (I'm not writing CRUD apps for instance) that I really do get a speed-up from LLMs.

I imagine folks writing conventional code might spend more time fixing LLM mistakes and thus think that LLMs slow them down. But this is not true for my problem domain.

The answer to this is code review. If an LLM writes code for you - be it implementation or tests - you review it before you land it.

If you don't understand how the code works, don't approve it.

Sure, complacent developers will get burned. They'll find plenty of other non-AI ways to burn themselves too.

All of the points you raise I find common in human written tests.

I'd certainly be interested to read more about your experience!

Thanks for the suggestion -- I'll take a look into adding this!

Coverage metric can tell if lines of code were executed, but they can't tell if execution result was checked.

I believe that's called mutation testing. Using an LLM to perform the mutation sounds like a great idea