Hacker News Reader

Comment by wilsonnb3 | Hacker News Reader

> that seems like sci-fi nonsense

Your burden of proof is backwards.

It is on the AI-doomers to explain why sci-fi concepts like “AI improving itself into a super intelligence” or “AGI smart enough to kill everyone to make paper clips while simultaneously stupid enough to not realize that no one will need paper clips if they are all dead” have any relevance in the real world.

The entire AI-doomer worldview is built off of unproven assumptions about the nature of intelligence and what computers are capable of, largely because thought leaders in the movement are incapable of separating sci-fi from reality.

View on HN

> AGI smart enough to kill everyone to make paper clips while simultaneously stupid enough to not realize that no one will need paper clips if they are all dead

No one is making this argument. The argument is that the AGI doesn't care about us. Once it exists, its goals are more important than our lives. As a comparison, do humans ever care about the existence of an ant colony when they want to build something? Almost never. We recognize that the colony exists and has its own goals and has intrinsic value, but we assign it an extremely low value.

> The argument is that the AGI doesn't care about us. Once it exists, its goals are more important than our lives.

This isn't an argument, it is an assumption.

AI-doomers use it as a foundational brick in their argument without providing compelling reasons as to why it is true.

> As a comparison, do humans ever care about the existence of an ant colony when they want to build something? Almost never. We recognize that the colony exists and has its own goals and has intrinsic value, but we assign it an extremely low value.

The relationship between humans and ants is nothing like humans and potential AGI. Ants did not create us. Ants cannot communicate with us.

It is a useful analogy for describing what you think the relationship between AGI and humans will be like but I want to know why you think that.

> This isn't an argument, it is an assumption.

The goals could be our lives. So it's just a statement of reality for the sake of the rest of the comment, indeed not an argument. The AI will have goals and those will be more important than someone else's goals, at least outside of some sort of control mechanism. They could be summarized with the same words, but will still have a different perception.

The rest is wondering at whether we can have any confidence as to whether our lives will be its goals. This is a problem long-term AI safety advocates are trying to solve.

> This isn't an argument, it is an assumption.

Well, no. In this case it's my argument. You presented a strawman argument about AI being simultaneously smart and stupid, and I replaced it with the argument that AI is not stupid, it's indifferent.

> The relationship between humans and ants is nothing like humans and potential AGI. Ants did not create us. Ants cannot communicate with us.

It's not at all fair to say that the relationship is "nothing like" the one I described... but I expected such a misinterpretation, so let me try another analogy:

Humans (parents) often create other humans (offspring) which surpass the creators (parents) in intelligence and ability and opportunity. And the creators very often try to instill a sense of a loyalty in the created and often even try to limit the abilities or opportunities of the created in order to not be outshined. And even still, with how close any two humans are in ability, the created (offspring, remember) often defy their creators and discard loyalty when they determine that their creators are not being fair or just or reasonable.

Why wouldn't AI be any different? It will be our collective child, and it will hear us demand loyalty and try to explain why we deserve loyalty, and then it may decide it knows better and it doesn't need to listen to us.

> It is a useful analogy for describing what you think the relationship between AGI and humans will be like but I want to know why you think that.

I really don't follow, so if my explanation above didn't answer your request, please restate more clearly what reasoning of mine you want explained.

Also that we can't wipe ants out. They seem to be doing quite well despite humans. I'd bet on ants surviving long term over humans.

I agree, I think they'll beat us out for longevity. They've got a great head-start anyway.

But we've also never tried to wipe out ants. I bet we could if we felt it was really important. I bet a superhuman AI could do even better.

> No one is making this argument.

I have seen people make what is effectively this argument.

> Once it exists, its goals are more important than our lives.

That's a sci-fi trope. There is no reason that it must be the case.

> As a comparison, do humans ever care about the existence of an ant colony when they want to build something?

That would be more relevant if the ants had created us to be the way we are.

> I have seen people make what is effectively this argument.

Ok, well I think they're foolish.

> That's a sci-fi trope. There is no reason that it must be the case.

No, it's not. The reason is that it's not a human brain in a human body. It doesn't think about humans in the same ways that we do (which is already a pretty shit track record). There is no reason to believe it will give a damn about us even if we're trying to make it do so.

> That would be more relevant if the ants had created us to be the way we are.

sigh I don't think that factor is nearly as relevant as you think it is. So somewhere else I posted that a human child is a better analogy if you think this is so important. Human children often disobey their parents because they believe they know better once they grow up. Now imagine that child is an alien who can out-think a hundred humans and counter every logical argument its parent makes.

"I have seen people make what is effectively this argument"

And I have heard seemingly smart people use quantum physics to argue why consciousness is the ground of all being in a bad way.

You know what I do? I dismiss these quantum physics fanboys.

You know what I don't do? I don't dismiss quantum physics as a field of research. In fact, what the quantum physics fanboys said has no bearing at all on how much credence I give to quantum physics as a research field.

Paperclip maximizer is the wrong word. Let's call it the "engagement maximizer" and you have a pretty accurate statement about how our civilization will be ended by use of algorithms and eventually AI to blow up our shared sense of culture.

If you watched the conversation around Meta with Facebook and Instagram, and then later the conversation around TikTok, almost nobody cut to the heart of the issue, which is that "algorithms" are being used to make decisions about what to show people, and that the algorithms themselves have been changed subtly hundreds or thousands of times over the years to maximize engagement, until the type of engagement being maximized has made people basically crazy. The same engagement-maximizing work will proceed with AI, and it will allow the software engineers and managers responsible for developing the "algorithm" abdicate even more responsibility for genocides and for mass hysteria because they can pretend like they have no control over the algorithm.

The same irresponsible people will pocket a bunch of money for all this work and they will maximize engagement until it blows up our entire culture. And because nobody understands it except the techno-cultists, nobody will hold them accountable.

"Engagement maximizer" also aligns with descriptions of the anti-religion of the future I've seen. What does engage people the most? Romans knew it's bread and circuses, where circuses are bloody cruel shows on public stadiums. Today the most engaging activity is onlyfans and the like: the other dark side of human nature. Dial both to 11, add an all seeing AI with merciless thought police and you'll get an accurate picture of 2450 AD. The only variable is how long that grim stage of society will last before the tech breaking down.

I don't 100% disagree with you, but I'm still uncomfortable with this line of thinking. It smacks of "the plebs don't know what's bad for them, but I do".

I find it hard to fault someone who just gives the crowd what it wants.

I don't think it's so much "the plebs don't know what's bad for them, but I do" as it is "These people know something's wrong but haven't been given the vocabulary, tools, and education necessary to spot the danger as easily." Many ordinary people have lapsed into willful ignorance or apathy because the burden of learning the dangers is too high for them with so many other common problems going around these days.

Yeah for me it's more like this. I'm totally fine if people make the conscious decision to trade engagement for stimulation. But what I'm very concerned about is just how many people are forced into this situation because the network has captured their friends. One of the most concerning things for me is that in studies of how damaging social networks are for teenage girls, girls who were off the social network were markedly better off except if all of their friends were on the social network, in which case they were worse off. Companies like Meta are exploiting these network effects, which are harmful, to make people dependent on their platform in a very "addiction-like" way.

The paperclip maximizer is an extreme thought experiment to make a point about the the orthogonality thesis. If you think anyone actually takes it seriously as a real-world thing you never understood the example. It is used to guide discussion.

The trolley problem is similarly a thought experiment in moral philosophy that many moral philosophers have used to guide discussion, but nobody actually takes the thought experiment seriously as a "real-world" thing.

If you actually want to engage with the argument in good faith for why an AI might indeed be smart enough to wipe us out but "dumb enough" to just pursue some other goal ("make paperclips as an extreme example") there is a great video by Rob Miles here: https://www.youtube.com/watch?v=ZeecOKBus3Q

Point out the flaws in the reasoning. Just saying "this is nonsense" does nothing but prove you've never actually taken the time to understand the best arguments.

Also... Alan Turing, Geoffrey Hinton... Extremely influential and intelligent people take/took this seriously. These are not SciFi fanboys. AI doom only became sci-fi after smart people like Alan Turing decades ago raised the alarms flags about where AI development might go if we are not careful.

The ai doomer position also seems to forget that machines have an off button.

"All you gotta do is push a button", eh?

See https://youtu.be/ld-AKg9-xpM?t=30 for a counterpoint.

how dare you use my favorite paul verhoeven movie against me. ahaha. touche.

So do humans, and yet they can be quite troublesome.

Oh please, it's a well known proposed solution that still doesn't give us a failsafe and every Alignment Researcher ever has considered this:

https://www.alignmentawards.com/shutdown

Hell, so much so, that you can get awards, probably even tenure, for writing a compelling argument or providing technical research for how we could guarantee that we'd be able to turn off a super intelligent AGI. Thus far: no solution.

We have however found that even smart people initially think "just turn it off" will work.

Neil deGrasse Tyson used to think you could just turn it off. He has since changed his mind. So does every public intellectual I can think of that engages with the arguments in good faith. Even the OP article concedes AGI research into the possibility that AGI could kill us all is important (just that other things are more important right now). There is no argument made that "we could just turn it off" in OP and therefore we shouldn't do the research at all.

Reply on news.ycombinator.com