In my experience, if you just ask an LLM to write tests, it'll write you a ton of boilerplate happy path tests that aren't wrong, per se, they're just pointless (one fun one in react is 'the component renders').
How do you plan to handle this?
You're right, this deserves more attention, and is a valid problem going forward with this app. And I had this problem when just started building, it either generated XSS tests for any user input validation method (even if it used other validators) or just 1 single test case.
For now I attempt to strictly limit the amount of tests for LLM to generate.
This is achieved with "Planner" that plans the tests for each function before any generation happens, that agent is instructed to generate a plan that follows the criteria:
- testCases.category MUST be one of "happy_path" | "edge_case" | "error_handling" | "boundary".
And it is asked to generate 2-3 tests for each category. While this may result in the unnecessary tests, it at least tries to limit the amount of them.
Going forward I believe the best approach would be to tune and tweak the requirements based on the language/framework it detects.
You could automate this, but you'll end up getting rid of useful tests and keeping weird useless ones until the AI gets better at nuance and large codebases.
Guess I need to make it generate negative tests?
Which is actually probably a solid idea for this exact use case.