No this isn't mindless reinventing --- this is finally having clear empiric evidence for something that we knew the entire time. This is a huge relief!
Next up is demonstrating the AI is more productive with better programming languages.
Next up is demonstrating the AI is more productive with better programming languages.
Key parts of the abstract:
> This raises a question the programming-language community has not previously had reason to ask: should error-message detail be calibrated differently for AI agents than for humans?
> We investigate this question through a controlled experiment using Shplait, an ML-style statically typed language. We construct a suite of programs containing a single deliberate type error each, and measure how often an AI agent repairs them under ablation: a detailed error context using the unification stack; a proximate error location; a minimal type error; and a dynamic (test suite) error only. An automated oracle uses a test suite to classify each repair attempt as a type error, semantically incorrect, or semantically correct.
> We find concrete evidence that more detailed error messages generally improve an agent's ability to fix type errors. We also find that the presence of a type system appears to help more than only test suite failure reports.
(It's an interesting question though - my prediction would be that AI is best in the languages it consumed the most examples of, and I reckon the size of the StackOverflow site for each language would be a rough proxy for that).
This is hard to do given the current reliance on model weights in its programming language knowledge. You’ll get your best results for python or some variation of python. But
I would be the first person to give up python for kotlin or some other more natively typed higher performing language. But the python bias is so far too big to ignore.
That actually would be pretty exciting!
But, wouldn't AI be biased toward more popular languages, since those will by nature of their popularity provide more sources for training material?
Even still, the AI could demonstrate which algorithms or maybe patterns and techniques are more productive, in the context of the popular languages.
As an example, 110v American outlet plugs are almost certainly not the most efficient way to power devices and appliances; but, because they are so standard and are good enough, using them massively eases the burden of making, buying and selling powered appliances in the US. Similar story with USB C.
On the other hand, having a bunch of different languages is an advantage. People can pick the best tool for the job, and experiment with new language features.
True, and to my point, "popular" isn't necessarily "better".
> because they are so standard and are good enough, using them massively eases the burden of making, buying and selling powered appliances
Yes, but LLMs can do the heavy lifting when analyzing the "better programming languages" for productivity. There will be a bias toward popularity in their training sources, though.