I've honestly never had that success. LLMs for me more often get in the way than actually help anything. It may be that it's because I deal mostly in very large code bases with their own nuances.
The code base I have I would love to be able to just give some AI free reign and learn the structure since a lot of it is fairly repetitive; I know it would be so easy to say "hey add X just like Y" and it would be able to do it easily.
My more experienced/senior colleagues all say roughly the same. It’s great help for our juniors though. They learn a lot and are more capable on their own with the AI assistance.
It’s improving all the time though, so I’m not writing it off at all. I am developing an evaluation suite so I can keep watching the progress in a systematic way..
Sounds like something that should be published on github
Code mods like upgrading from python 2 -> 3 could also become possible.