I agree. I don't think they're mutually exclusive. You can have something smarter than you which you use as a tool.
My point was that there's nothing objectively special about our level of intelligence, so it shouldn't be used as a benchmark.
My point was that there's nothing objectively special about our level of intelligence, so it shouldn't be used as a benchmark.
That sounds like how tech CEO's treat their employees!
Kidding aside, I think the notion that you could control something (legitimately) smarter than you is a pretty risky proposition. (Fortunately one that I think is actually far off)
But then think about the CEOs and their employees again.
We just need to invent something like money for an "AI" and then we can lead in on a stick ;-)
While I think the discussions around alignment sound more like science fiction than science, it also hasn't been "solved", and the only "control" you have over what the agent does is by gating access. Even if you think the AI itself is aligned, handing complete control of your entire enterprise and all your data to a private unaccountable for-profit enterprise should give you a lot of pause.
The idea that you can create entire jobs where humans just act as reviewers or gate keepers seems very unlikely to me, just knowing human psychology. If the AI is right 99% of the time, and catastrophically wrong %1 of the time, but I've been condition to always hit "accept" because it's usually right, there's very little chance I'm going to catch that 1%. "yes, continue with the database migration". "yes, continue with the database migration." x 1000. (At button click 500, the user stopped reading). "yes, go ahead and delete the database. Shit, wait!!!"
Well, Trump seems to control a lot of people given how afraid they are.
Trump is also called not the smartest person out there.
So...