I tried having it write some tests today. It got very close to what I want, but picked a stupid set of input values (two fields that look independent that should only be used with related values). I thought about "how do I explain this" and then just went in and fixed it myself.
How is it easier to write "Okay, go back to testBlah and change xxx to yyy" versus clicking on XXX in the IDE and typing YYY by hand? Maybe if you had 500 faulty tests and were forbidden from using search-and-replace for some reason.
It makes sense when code generation is the limiting factor, but I end up with a lot of changes where the actual code delta is smaller than the necessary prompt to convince the bot to produce it.