At my job I review a lot of code, and I write code as well. The only type of developer an LLM’s output comes close to is a fresh junior usually straight out of university in their first real development job, with little practical experience in a professional code-shipping landscape. And the majority of those juniors improve drastically within a few weeks or months, with handholding only at the very start and then less and less guidance. This is because I teach them to reason about their choices and approaches, to question assumptions, and thus they learn quickly that programming rarely has one solution to a problem, and that the context matters so much in determining the way forward.

A human junior developer can learn from this tutoring and rarely regress over time. But the LLM’s all by design cannot and do not rewire their understanding of the problem space over time, nor do they remember examples and lessons from previous iterations to build upon. I have to handhold them forever, and they never learn.

Even when they use significant parts of the existing codebase as their context window they’re still blind to the whole reality and history of the code.

Now just to be clear, I do use LLM’s at my job. Just not to code. I use them to parse documents and assist users with otherwise repetitive manual tasks. I use their strength as language models to convert visual tokens parsed by an OCR to grasp the sentence structure and convert that into text segments which can be used more readily by users. At that they are incredible, even something smaller like llama 7b.