We don’t know how human intelligence works. We don’t have designs or even a philosophy for AGI.
Yet, the Bostrom view is that our greatest invention will just suddenly "emerge" (unlike every other human invention). And that you can have "AGI" (hand-wavy) without having all the other stuff that human intelligence comes along with, namely consciousness, motivation, qualia, and creativity.
This is how you get a "paperclip maximizer" – an entity able to create new knowledge at astounding rates yet completely lacking in other human qualities.
What has us believe such a form of AGI can exist? Simply because we can imagine it? That's not an argument rooted in anything.
It's very unlikely that consciousness, motivation, qualia, and creativity are just "cosmic waste" hitching a ride alongside human intelligence. Evolution doesn't breed inefficiencies like that. They are more likely than not a part of the picture, which disintegrates the idea of a paperclip maximizer. He's anthropomorphizing a `while` loop.