The ultimate goal of this technology is to replace people -- if it's a tool and corporations still need to employ people to use the tool, then the valuations make no sense. So ultimately, you're going to need to hand the car keys to these agents more and more: here's access to production (we fired the devops people), here's access to all our internal messages and IP, here's all our source code (we fired the engineers).

While I think the discussions around alignment sound more like science fiction than science, it also hasn't been "solved", and the only "control" you have over what the agent does is by gating access. Even if you think the AI itself is aligned, handing complete control of your entire enterprise and all your data to a private unaccountable for-profit enterprise should give you a lot of pause.

The idea that you can create entire jobs where humans just act as reviewers or gate keepers seems very unlikely to me, just knowing human psychology. If the AI is right 99% of the time, and catastrophically wrong %1 of the time, but I've been condition to always hit "accept" because it's usually right, there's very little chance I'm going to catch that 1%. "yes, continue with the database migration". "yes, continue with the database migration." x 1000. (At button click 500, the user stopped reading). "yes, go ahead and delete the database. Shit, wait!!!"