The real issue is prompt instability (chaos). A one word change to a prompt/spec will produce a drastically different program. Until that is solved there’s no world where we just check in the prompt and almost no one ever has to worry about the code.
So in other words, determinism (or lack thereof) is the hard problem!
Granted that is not the rigorous definition of determinism. Also some existing compilers are non-deterministic with such a definition, namely when "undefined behaviour" would be triggered. That's precisely the problem with UB.
You’re agreeing with my point though which is that LLMs aren’t higher level compilers. LLMs aren’t abstraction, they are delegation.
There are certainly useful things you can do with that.
But give that team to a PM who can’t read the output and anything beyond a todo app would collapse under the chaos.
They are much much better at writing plausible looking code or providing plausible sounding answers that look correct but aren’t than any junior developer. They are much better at writing subtle bugs that won’t show up for days and will take hours to debug than any junior developer is.
You can’t mentor them into being a senior developer.
Junior developers (human) can kind of get by without testing, at least in the short term. The LLM can never get by without testing unless it’s some simple one off logic.
They’re also very bad at understanding what edge cases are useful to test and what edge cases can be ignored.
I had a situation just the other day where the LLM produced tests for scores of edge cases, but a managed to leave out a specific sequence that caused a very difficult to diagnose bug. I’m assuming because it decided that sequence wasn’t possible.
But it did include dozens of redundant cases that no sane human ever would have.
I just went through this a couple of weeks ago. My test cases were ok but missing some basic edge cases. I asked it to enumerate the edge cases but that didn’t work well either. I then asked it to list near positive and near negative edge cases, and that worked well for my problem. For bugs I found in the code that weren’t covered, I asked the LLM to add them and/or mark them critical to cover (the case might had been there already but I only let it generate 3 tests with a limited size data set, so not all cases would be covered). That worked well, although I’m sure I could do better (probably by generating more tests, but many current approach is slow because each artifact is generated in a context isolated agent, so I have to speed that up first).