Taking a visual proof-of-concept and turning it into a real product with integration to an existing complex system requires the developer to re-do a-lot of the work. And reading code; especially AI code someone-else wrote, is a miserable experience.
If the answer is it's reassuring because it's not happening yet (with your proviso for if ever) then fine.
Doesn't sound very compelling to me but if that's the answer then fair enough.
Trust-able in particular is a growing issue i feel. Very polished looking AI made feature-rich software can have some atrocious bugs in the simplest of parts. Some features may not have been used at all since they were created. Testing never catches everything (even when written); humanity has probably been saved from countless billions of bugs from shower-thoughts of lunch discussions. (AI doesn't take showers). When AI made tests to verify AI made code based on the instructions of a single sleep deprived human using very fuzzy and ambiguous commutation media; can we trust that the software does what we expect it to do at all?
We had this discussion recently at work. "we spend X on our accounting and offer writing system; can we just replace it?". The answer was roughly; "yes, but would you trust it not to accidentally send us in an investigation with the IRS?". If we have to do it properly enough to trust, then AI ends up not being used for much more than user interface. (and that's still a glorified spreadsheet compared to most complex software systems). There is rather fascinating case of a UK cost tracking system convicting 900 sub-postmasters of theft (a few to prison and at-least one to self-inflicted death) over the span of 15 years because a software system was "perfect, tested, and not making any mistakes in its calculation"
https://en.wikipedia.org/wiki/British_Post_Office_scandal
---
Not to say AI is useless or inconsequential; we're wasting a lot of time writing low-consequence boilerplate for UI, code interfaces, API schemas, error handling and data parsing. If they don't work we notice, so they're perfect for AI. But when a "project manager" makes a "prototype app"; it sounds a-lot like a interactive prototype in javascript for the UI; closer to a modern-day figma design than a functional product.
I keep hoping for better answers to this. Your longer answer is frankly just as disappointing.
"AI doesn't take showers" was particularly unconvincing. An army of AIs can ruminate on a codebase indefinitely. They can explore a possibility space at scale in a number of dimensions beyond most humans. How do you think these math problems are being solved? Code is largely verifiable and testable, most of the things that are hard to test is with human interaction. But humans will increasingly be out of the loop.
Most apps are not very useful. Most startups fail. Most companies are copying each other. There's a reason AI is so good.
Trust will be solved with mitigations. There will be inventions in this regard like breakthroughs in formal verification and adversarial LLMs to hold the automations accountable. At a certain point AIs will be trusted more than humans. AI has no free will. Humans go rogue all the time. You can't wipe their brains or reset them or turn them off. How many AIs have shot up their workplace? I just am kind of baffled by people who say AI can't be trusted or controlled as if humans can be?
The question is about outcomes. Costs and benefits. It seems obvious to me within 10 years unless you are within the top 1% of software engineers an AI will make more economic sense than you on a basic cost/benefit breakdown. Maybe a few of the top 10% hang around to certify certain critical paths so if the AI makes a big enough goof you are around to have the rope be hung around your neck.
The end state of capital is automation of everything. The humans actually doing interesting things aren't making "prototype apps" or being "project managers" they are building these AIs or coming up with ways to engineer UBI and a future where knowledge workers cease to exist without the entire economy collapsing.
I'm sorry but it's just not a very good answer. Thanks for trying though.
> breakthroughs in formal verification
this happens to be quite an interest of mine, and can quite confidently say; hah, no. Formal verification of software is such a disproportionately high difficulty compared to writing software; and there is duck all training data. Your lucky if the LLM knows the syntax or standard library.
I'm curious what you work with given your dismissal of software engineering as a displine existing within 10 years.