I think it really, really depends on the language. I haven't been able to make it work at all for Haskell (it's more likely to generate bullshit tests or remove features than actually solve a problem), but for Python I've been able to have it make a whole (working!) graph database backup program just by giving it an api spec and some instructions like, "only use built in python libraries".

The weirdest part about that is Haskell should be way easier due to the compiler feedback and strong static typing.

What I fear most is that it will have a chilling effect on language diversity: instead of choosing the best language for the job, companies might mandate languages that are known to work well with LLMs. That might mean typescript and python become even more dominant :(.