Improving AI-generated tests using mutation testing

blog.senko.net

1 point·senko··2 comments

First, I have some advice if you are open to it. Apply a spell/grammar checker to your post before sharing it with other people. Your post has several typos and many people will stop reading after the first or second typo. “The author didn’t care enough to proofread, why should I bother reading.”

> Remember, this is important: do not look at the tests. If you let them into your context

This is a bad idea. You are trusting the LLM’s ability to follow instructions. Worse, depending on your harness the LLM might not even be able to follow these instructions. The harness may indiscriminately place code into the context in a way that is uncontrolled by the LLM.

A better idea is to modify your harness so that certain files are excluded from the context.

> I asked another AI to carefully review the tests and identify those that don't make sense.

Test validation is an entire area of research and I’m yet to be convinced that this is a task for LLMs.

> Apply a spell/grammar checker to your post before sharing it with other people.

Hey, at least you know it isn't LLM generated! :) Thank you - I usually do, here I obviously didn't. Appreciate the callout.

>> do not look at the tests. If you let them into your context

> This is a bad idea. You are trusting the LLM’s ability to follow instructions

Oh I'm bot, I manually checked what it was doing. I might set up the ignore explicitly if I turn this into a repeatable procedure.

Reply on news.ycombinator.com