Why do we bother with programming languages today? Why not have the LLMs just write assembly code and skip the human readable part? We are not reviewing it anymore anyway.
1) Higher level code is easier for LLMs to review and iterate upon. The more the intent is clear from the code, the easier it is for humans and LLMs to work with.
2) LLMs get stuck or fail to solve a problem sometimes. It is preferable to have artifacts that humans can grok without the massive extra effort of parsing out assembly code.
3) Assembly code varies massively across targets. We want provable, deterministic transformation from the intent (specified in a higher level language) to the target assembly language. LLMs can't reliably output many artifacts for different platforms that behave the same.
4) Hopefully, we are still reviewing the code output by LLMs to some extent.
1.5) Having a compiler in the loop that does things like enforcing type constraints (and in the case if Rust in particular, therefore memory safety guarantees) is really useful both for humans and LLMs.
A very big practical reason is also that assembler code would eat context like no other.
The counter-argument, and one that matches my experience is working at a lower level is actually beneficial for LLMs since they can see the whole picture and don’t have to guess at abstractions.
Programming languages are tools for thinking. It's not clear that assembly code has the right abstractions to encourage the kind of thinking that programming large systems requires. After all, human intelligence found assembly insufficient and went on to invent better languages for thinking, why should artificial intelligence, trained on human intelligence, be any different? Maybe AI in the future will have its own languages for thinking, but assembly is likely not that.
It’s just a matter of different workflows for different users and application.