Hacker News Reader

Comment by sverona | Hacker News Reader

His argument is essentially "a superintelligence who is better than us at everything and thinks a quadrillion times faster can do whatever it wants and we are powerless to stop it."

Yeah, I could've told you that.

If we really are going to create such an intelligence in the next five years, then we had a good run, so long and thanks for all the fish. But that assumption coupled with the security mindset he brings to the table (viz. "the only unhackable computer is an unplugged one at the bottom of the ocean") is so strong that the big list of doom vectors he comes up with appears much scarier than it actually is.

In the past I've struggled with intrusive thoughts that the government is going to come and murder me. I could have given you reasonable-sounding explanations for why I believed this. Doesn't mean it's gonna happen.

View on HN

That’s a very strawy straw man you’ve got there.

AIs can do science, write code, impersonate people, and manipulate people. We’ve already got AlphaFold and ChatGPT and Copilot. People are moving full steam ahead with AI software developers and scientists who have access to deploy code and spend money and communicate with humans autonomously.

I don’t think it takes a whole lot of imagination to see these things improving and coming together in way that an AI agent could feasibly design and execute a plan to develop a bio weapon or deadly nanotech. His points are about how hard it is to prevent that with our current AI training regime.

His analogy, for example, between the human “inclusive fitness reward function” (we evolved with the role purpose of survival and reproduction) and RLHF-style human feedback for AI is apt, and not obvious. Just because we “evolved to survive” didn’t prevent groups of humans from developing the exact opposite capacity to make us extinct.

Mmm. The definition I used is the standard definition of superintelligence, used by EY himself:

> A superintelligence is something that can beat any human, and the entire human civilization, at all the cognitive tasks. [0]

I added the "thinks a quadrillion times faster" part but I think that's fair, if perhaps off by a few orders of magnitude.

All of EY's work has an often unstated assumption that AI will adversarially try to kill us. This is explicitly noted in the replies to the AI ruin post. There are a lot of fiddly details that I'm skipping over, but I stand firm that his argument reduces to "it's functionally impossible to stop something faster and smarter than us that really wants to kill us from killing us once it gains sentience."

[0] https://youtu.be/gA1sNLL6yg4?t=1290

Your reduction doesn’t cover the real risks of orthogonality, instrumental convergence, the zero margins for failure on the first try, intelligence explosions, and the impossibility of training for alignment.

Yeah, those only make it worse, but they only really apply to a bona fide superintelligence of the sort EY describes, and those are not what we have. I don't believe we're particularly close to having one.

If we're doomed, we're doomed, but please don't tell me about it.

Reply on news.ycombinator.com