“Bad ASI would pretend to be good *until* it could execute a plan to kill everyone all at once”.
So not today but tomorrow it could engineer an airborne virus. The point is it can come up with chess moves that are inconceivable to us.
So not today but tomorrow it could engineer an airborne virus. The point is it can come up with chess moves that are inconceivable to us.
They literally have no clue the degree to which more intelligent people/systems can walk circles around less intelligent people/systems. In situations with a significant intelligence gap, they're the ones getting walked around.
Which, to your point: is completely opaque to the person getting walked around.
And this dynamic is evident even in human-to-human variations in intelligence. The gap to frontier machine intelligence will soon be much greater than the types of gaps you encounter with other humans.