AI is very optimized to producing code, but not there (yet? ever?) in making products. Show me a serious software made entirely with AI. There's no AI SolidWorks, AutoCAD, MacOS.
And there lies a contradiction: AI is making it easy to start and produce copious amounts of code that looks okay-ish. We start releasing a product and notice subtle changes. We did not internalize our understanding through it. There's a million lines of smart looking code, so we don't know where's what. We ask to AI to "make the button blue" https://opusfived.dev/ and the cycle begins.
I am working on a large software at work, trying a "I didn't even look at the code" approach with GPT 5.6 Sol. It started really promising but by now (and about $10k in tokens) it's a complete clusterfuck. Yes I use GSD and code graph etc.