This was very thoughtful essay; I've had similar lines of thinking myself. How will we debug auto-generated AI code?

That said, on that topic, the essay overlooks one key point and line of reasoning about debugging, one derived from Kernighan's Law. (Kernighan as in the K in AWK + K&R C...)

The law states: "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it."

If Kernighan's law is "true" in some rough sense (as I have long agreed with), then we have a potential solution to the "AI debugging" problem... ask the LLM to make the code four times simpler than it needs to or write code with a 4x dumber model. Then a smarter model (or us) can debug it. Right?