Is entropy increasing or decreasing the longer agents work on a code base? If it's decreasing, no matter how slowly, theoretically you could just say "ok, start over and write version 2 using what you've learned on version 1." And eventually, $XX million dollars and YY months of churning later, you'd get something pretty slick. And then future models would just further reduce X and Y. Right?
Maybe they just need to keep iterating.
I am an avid user of LLMs but I have not seen them remove entropy, not even once. They only add. It’s all on the verge of tech debt and it takes substantial human effort to keep entropy increases in check. Anyone can add 100 lines, but it takes genuine skill to do it 10 (and I don’t mean code golf).
And to truly remove entropy (cut useless tests, cut useless features, DRY up, find genuine abstractions, talk to PM to avoid building more crap, …) you still need humans. LLM built systems eventually collapse under their own chaos.
I think your analogy is quite fitting!