I'm just not worried about this, LLMs don't ship.
But if you’re asking for something you don’t know how to do you might end up with junk and not even know it.
I think there's a whole meta level of the actual dynamic between human<>LLM interactions that is not being sufficiently talked about. I think there's, potentially, many secondary benefits that can come from using them simply due to the ways you have to react to their outputs (if a person decides to rise to that occasion).
> And in that process I've found is where the real magic happens
It might be good way to learn if there's someone who's supervising the process, so they _know_ that the code is incorrect, and tells you to figure out what's wrong and how to fixes.
If you are shipping this stuff yourself, this sounds like a way of deploying giant foot-guns into production.
I still think it's a better to learn if you try to understand the code from the beginning (in the same way that a person should try to understand code they read from tutorials and stackoverflow), rather than delaying the learning until something doesn't work. This is like trying to make yourself do reinforcement learning on the outputs of an LLM, which sounds really inefficient to me.
What I find (being in the latter category) is most LLM code output falls on the spectrum of “small snippets that work but wouldn’t have taken me long to type out anyway” to “large chunk that saves me time to write but that I have to thoroughly check/test/tweak”. In other words, the more time it saves typing the more time I have to spend on it afterwards. Novices probably spend more time on the former part of that spectrum and experienced devs on the latter. I suspect the average productivity increase across the spectrum is fairly level which means the benefits don’t really scale with user ability.
I think this tracks with the main thing people need to understand about LLMs: they are a tool. Like any tool simply having access to it doesn’t automatically make you good at the thing it helps with. It might help you learn and it might help you do the thing better, but it will not do your job for you.