No way any regulated industry is going to just trust some AI to write the code.
Imagine trying to get a medical device through FDA approval with firmware written by a bot.
Imagine trying to get a medical device through FDA approval with firmware written by a bot.
The FDA would require extensive efficacy trials and an inventory of side-effects, the same as other devices. The FDA isn't doing computer engineering, it's measuring outcomes and risks.
If an LLM generates vulnerable code that evades detection and makes it into production, and that vulnerability is exploited, who is responsible? Presumably, the humans who were tasked with reviewing and approving the code.
As the famous quote states, it is easier to write code than to read it. An LLM would be great for improving developer tooling to generate boilerplate code, etc., but I see tremendous risk for any firms attempting to let an LLM design and build any substantial non-trivial pieces of system code. And the effort required to review the code is at least on the same order of magnitude as it would be to write the code (beyond the trivial “make me an HTTP controller or HTML form for these operations” and the like)