Comment by NumberWangMan | Hacker News Reader

Comment by NumberWangMan | Hacker News Reader

Parent Full thread

That's not quite the existential risk of AI -- it's that we end up growing an intelligence (or multiple intelligences) that are very very smart but only have the illusion of morality, and they decide that whatever they're trying to do would be a lot easier and more certain to succeed if they didn't have to worry about humans stopping them or inventing other AIs to compete with them. It seems likely that unless we really understand how AI works, we'll end up with something that evaluates possible courses of action in a purely amoral, sociopathic sort of way. Like, trying to avoid getting caught rather than trying to avoid doing a bad thing.

So it's not that they cause the collapse of the economy, it's that they figure out a way to entirely take over the planet, engineering a virus or something that kills almost everyone, but only once they have enough control over robots and such to keep their own power plants running and protect them from the human stragglers, eventually getting rid of all of us. Waiting until the plan is basically fool-proof before starting to take the first steps.

Even if an AI has some sort of morality, at some point, if it's smart enough, you cannot hope to control it for the same reason a toddler cannot hope to outsmart their parents. It will have a plan for us, and if it decides that, hey, humans are better off scanned and simulated rather than having physical bodies, that's what's going to happen, like it or not.

I'm not sure how worried I am about this. Some days very, other days not as much. The risk depends on a lot of factors with wide possible ranges, like how much model capability and generalization scales with compute, how likely are breakthroughs, how hard alignment turns out to be, is alignment even possible, will we learn enough from earlier models to make alignment of later ones easier, will there be a qualitative shift in alignment difficulty as models get smarter, and so on. It's also a quandary because there is substantial human suffering that AI might be able to help. But given that the downside risk is all of humans dead forever, it seems worth slowing way the heck down to a safe speed, trying to get all the benefit out of current models as we can, and focusing on making sure that we aren't about to head off a cliff.

I have come across a few great comments on intelligence here on HN — intelligence is not the bottleneck of almost anything we do. Let’s say you became “comically” smarter overnight, could you really significantly benefit that extra knowledge? Besides going for the Millenium Prize problems, the benefits would not be too significant, for example is it that valuable in the startup space? (otherwise we would see PhD students as CEOs everywhere). Opportunity/luck is much more meaningful.

We also regularly see these “with my super intelligence I predicted your next 347 steps” type of thinking, but that’s just bullshit. The bottleneck here is data if anything, so intelligence won’t aid much in itself.

The thing is, we live in an infinitely complex game with independent agents, many of which don’t operate on any sane strategy even. Let’s say we are at an endgame Monopoly — would knowing ahead of time your opponents dice rolls help you win?

(Of course I’m no expert on any of these topics, so take all these with a huge grain of salt)

> Let’s say you became “comically” smarter overnight, could you really significantly benefit that extra knowledge? Besides going for the Millenium Prize problems, the benefits would not be too significant, for example is it that valuable in the startup space? (otherwise we would see PhD students as CEOs everywhere). Opportunity/luck is much more meaningful.

Hmm, that's assuming that PhD students want to be CEOs. People become CEOs because they like money, but also because they like running companies. For a PhD student, that might mean days full of people coming to them with problems they have no interest in solving. Sure, PhD students want money, but if getting money means sacrificing years of your life doing work you don't enjoy, it's not generally worth it.

I don't disagree that luck is a huge factor in most people's success. That said, you're unlikely to become a multi-billionaire without substantial help from your brain. There are other factors, like lack of shame or embarrasment, that could also be considered a form of intelligence, or rather an adaptation to a particular kind of environment. They might make you less popular in some situations, but more likely to succeed in business. Anyway, I agree that there are multiple factors.

Information is for sure a bottleneck, that's a good point. In a game like chess, you have perfect info, which is not the case in the real world. More intelligence lets you draw better conclusions with less data, but it's not omniscience. One big question is, how close are humans to the theoretical maximum of intelligence? Some people think we're quite a ways away. When AGI is developed, is it going to blow way past us, to the point where we can't even understand the concepts it comes up with? I don't know!

> Let’s say we are at an endgame Monopoly — would knowing ahead of time your opponents dice rolls help you win?

I think so, wouldn't it? For trading properties and so on. I'm not sure if I'm getting the point you're aiming at, though, so apologies.

Maybe another analogy is poker, where your opponents haven't figured out game theory and play on intuition. Having a better model for how the game works is absolutely a big advantage that stacks up over time, even with the randomness. Maybe, as an AI, you just look for small advantages here and there, playing humans off each other, gaining trust and influence, acting helpful, until you're in a position of power. Remember that an AI is effectively immortal, and possibly very patient. One scenario of concern is that AI takes over, it just takes a generation or two, after we have all thought that we were past the hard parts. At that point, we might be able to digitize human brains, or have solved the AI alignment problem, so maybe it's a hopeful scenario.

And then a big variable is how many super-intelligent AGIs there are. Once we develop it, are they going to be carefully restricted, or are we going to test them until we think they are safe enough, then copy them everywhere? These might look very different.

I'm also not an expert, so same for me! Grains of salt all around.

You are comically underestimating intelligence. For example, compare your level of intelligence with that of an ant, to get a feeling of what it might feel like to be "comically" smarter. It is just on entirely another level.

It is not about predicting the next 100 steps an ant makes. It is about setting ant baits where you don't want the ants.

Reply on news.ycombinator.com