> they can always shrug and say "well, AI makes mistakes." Error budgets? Failure modes? Test sets? All of those can be handled later.

This is just the complete opposite in my experience. Tests are the first thing the AI writes, especially in low coverage or unknown domain situation.

These systems have been trained for "generations" to oneshot problems. The first thing they do is create a mini harness to verify their solution is correct.

Majority of issues with AI assisted development are misaligned requirements or poorly stated goals