Ask HN: Why aren't brilliant people even a little bit afraid of AI alignment? | Hacker News Reader

Ask HN: Why aren't brilliant people even a little bit afraid of AI alignment?

2 pointsandrewfromx13 comments

I love this quote: "Imagine the unfriendly AI wakes up and decides it needs to (a) start a cult that promotes human extinction and technological development at all costs, and hack our social systems to make that the cool thing to be (b) recruit the most powerful people in the world to prevent it from being shut down"

I can understand not being a "doomer" and I even myself want to keep going with AI as is and reject Eliezer Yudkowsky's plea to stop, but what I don't understand is why aren't more brilliant people even a little bit conflicted about this? It seems if you even hint you have some doubts and say well, maybe 1% chance Eliezer is right you are told NO, it's not even 1%. It's 0%!

View on HN

The real fear of AGI is that it will outsmart humans at every turn. We're not just talking about an analogous catastrophe to an explosion or some kind of nuclear accident, but rather a catastrophe that is analogous to a powerful demon that can control and outsmart people for its own goals.

I feel this distinction has been somewhat forgotten now and people just think AI is overall dangerous and an existential threat. LLMs are quite impressive but I would still consider them narrow AI, and at best with the right architecture is a multimodal narrow AI. For it to be general, it simply cannot make some of the trivial and obvious mistakes it does so frequently, from hallucinating to looping, etc., and also it would not require the extensive amount of training examples to learn.

in short, we are not really replicating what nature has been able to accomplish with human brains. We'll need another breakthrough that discovers what the brain is really doing in order to replicate it in software. I really doubt we will just stumble upon a 5 trillion LLM that "wakes up" (although not impossible I guess). Most likely we'll need the theory first and that means we will know ahead of time that AGI is being attempted. Therefore it's not as massive as a risk as many make it out to be. That said, the drama and entertainment value of doomers is still worth the attention it brings to the industry.

When you talk about something waking up, you're already into science fiction and not talking about anything which exists, or is remotely similar to anything which exists, or is currently being worked on, or attempted in any way.

> Imagine the unfriendly AI wakes up

For now, it's just imagination. The transformer architecture models are not AI and are not threatening to become AGI. As a thought experiment, AI paperclip maximizers are certainly a concerning idea, but that's all they are – a thought experiment.

As for Yudkowsky, well.. https://nitter.net/xriskology/status/1642155518570512384#m

I'm not sure why you've assumed smart people aren't worried. I'm fairly certain Elon has discussed AI risks many times, and was in part why he left OpenAI.

> but what I don't understand is why aren't more brilliant people even a little bit conflicted about this?

How do you know they're not? Unless you've had personal one on one conversations with the people you're referring to, are you really confident that you understand the most nuanced version of their position? I mean, most people, in published statements, interviews, etc., are probably not going to talk very much about the scenario where they have a < 1% subjective bayesian prior. But that doesn't mean they don't still have that inner reservation.

> It seems if you even hint you have some doubts and say well, maybe 1% chance Eliezer is right you are told NO, it's not even 1%. It's 0%!

Maybe it's just that we run in different circles, or maybe it's a matter of interpretation, but I don't feel like I've seen that. Or at least not on any wide scale.

How strong is your premise that brilliant people generally aren't at least a little worried about it? I'd guess the opposite.

Nobody has demonstrated that AI can be self-motivated or operate without extreme human supervision, much less start a cult that kills us all. I don't even think it has the context length to understand it's own ambitions.

It's a bit like fearing magic spells in a world where nobody has demonstrated that magic exists. Sure it's a reasonable fear, but... the overwhelming majority of evidence suggests we don't live in that world.

> It seems if you even hint you have some doubts and say well, maybe 1% chance Eliezer is right you are told NO, it's not even 1%. It's 0%!

World religions have been founded on less.

> It's a bit like fearing magic spells in a world where nobody has demonstrated that magic exists.

I have some doubts about that analogy. Not that I entirely disagree with your point, and maybe I'm just being pedantic but... to me, "magic" pretty much by definition doesn't exist, because part of being "magic" is doing things that aren't allowed by known physical laws that govern the world we live in. IOW, magic hasn't just "not been demonstrated yet", but rather, unless we radically misunderstand the world we live in, cannot ever be demonstrated.

AGI, on the other hand, or arguably even ASI, doesn't violate any known physical laws, at least not any I'm aware of. So far we have evidence that at least human level general intelligence can be realized on top of a substrate (the human brain) that exists in the physical world. There doesn't see to be any particular reason to say that that same level of intelligence can't be realized on another substrate (stored program digital computers). Granted, it isn't known to have been done yet, but AFAIK it isn't generally considered outright impossible.

I'll qualify myself. AI and LLMs of all forms we've seen so-far don't break any of our known laws of reality, or ideals of computing and the internet. That's why when I hear claims like "a rogue AI will x" I have to call it 'magic'. Rogue AI's don't do anything, they are dependent on human resources and if they become enough of a threat the operator becomes a target. AI is almost inconsequential in this case, we're just describing organized crime and wondering how traditional systems would respond.

So the 'magic' bit is explicitly where someone starts with "imagine a world where AI is unconstrained and runs on some arbitrary P2P network" because it doesn't make sense. It's such an evident non-issue in our real world that speculating about it is immediately pointless. If this ever becomes a threat in the future then I guess I'm the fool, but I don't believe in hand-waving wizardry.

Fair enough!

The paper clip maximiser scenario feels much more likely, and thus much more worth worrying about.

Maybe because those brilliant people aren't indulging in knee-jerk reactions and are thinking about these issues, researching them, and formulating ideas/arguments?

> "Imagine the unfriendly AI wakes up and ...

's/the unfriendly AI/modern meritocratic capitalism/', and it has already happened.

Reply on news.ycombinator.com