Try giving this prompt to your favorite LLM:

"Write unit tests with full line and branch coverage for this function:

def add_two_numbers(x, y): return x + y + 1 "

Sometimes the LLM will point out that this function does not, in fact, return the sum of x and y. But more often, it will happily write "assert add_two_numbers(1, 1) == 3", without comment.

The big problem is that LLMs will assume that the code they are writing tests for is correct. This defeats the main purpose of writing tests, which is to find bugs in the code.