Hacker News Reader

> The "conscious superintelligent AI" is a scifi LARPing game that some people are getting paid to play

Thinking our meat brain's consciousness is somehow special, and our intelligence can't be surpassed by a machine, sounds much more like a religion to me.

AI researchers are working towards making intelligent machines. Why are we supposed to believe that - if they manage to achieve their goal - these machines won't surpass our intelligence?

> Thinking our meat brain's consciousness is somehow special, and our intelligence can't be surpassed by a machine, sounds much more like a religion to me.

Lucky I didn't say anything remotely like that, then.

> AI researchers are working towards making intelligent machines. Why are we supposed to believe that - if they manage to achieve their goal - these machines won't surpass our intelligence?

I didn't say that they couldn't. In fact in certain respects, GPT-4 has already surpassed average human intelligence. No human in the world could compete with its rate of correct answers to arbitrary questions, for example, or its speed in writing computer programs up to a certain size. Yet no-one is seriously concerned that GPT-4 will end the world. For that doomer scenario to come to pass, we need to add various fantasy elements (even calling it "scifi" perhaps gives it too much credit.) At this point, I'll refer you to this as a good intro for why the AI doomer cultists are misguided: https://idlewords.com/talks/superintelligence.htm

> Lucky I didn't say anything remotely like that, then.

Then surely AI research creating a superintelligent conscious AI is a realistic possibility and not a "scifi LARPing".

> Yet no-one is seriously concerned that GPT-4 will end the world.

I'm not concerned that GPT-4 will end the world. Maybe not even GPT-8.

LLMs doesn't seem like a very prospective way towards a strong AI. But AI research is not united on LLMs and works on different approaches, some of which could lead to a superintelligent AI.

“For all we know, human-level intelligence could be a tradeoff. Maybe any entity significantly smarter than a human being would be crippled by existential despair, or spend all its time in Buddha-like contemplation.”

That would probably be quickly optimised out by the training process as it’s an inefficient way to achieve a goal. The only basis for this argument is that humans are smarter than other animals and this seems unique to humans but it’s probably just a quirk of humans

“With a big enough gap in intelligence, there's no guarantee that an entity would be able to "think like a human" any more than we can "think like a cat".”

LLMs are clearly already very good at “thinking like a human”

“he still couldn't talk the cat into it.”

Because cats can’t talk? Humans can, and can clearly be manipulated into doing things, sometimes even by current AI

“So how are we supposed to solve ethics and code a moral fixed point for a recursively self-improving intelligence” “ The idea that we can securely design the most complex system ever built, and have it remain secure through thousands of rounds of recursive self-modification, does not match our experience.”

Yes, that is why AI alignment is hard, that’s an argument for that view, not one against it

“I don't buy this argument at all. Complex minds are likely to have complex motivations; that may be part of what it even means to be intelligent.”

This is a clear misunderstanding of what it means to be intelligent. It doesn’t mean “do things that humans find impressive”, it means “achieve your goal effectively”. An AI that goes against its own goals is not intelligent, would not achieve its training objective and the training process (which somewhat resembles evolution) would optimise it out in the next step of training in the same way that a species that immediately throws itself off the nearest cliff will not survive to the next generation. You can imagine a superintelligence as something that goes over all the actions it could take, sees what reward it would get according to its goal and picks that action. The intelligence here is in the world model it uses to predict the consequences of its actions, picking what actions to take is a simple “max over array” function so saying it will randomly decide to persue a more meaningful objective is like saying that a sorting algorithm will put a list in the wrong order because it thinks some other order is more meaningful

(I will continue in a new comment in case there’s some kind of length limit)

“When we look at where AI is actually succeeding, it's not in complex, recursively self-improving algorithms.”

That’s because humans are better at this currently. When AIs get better at this than humans they can take over the process and improve a lot faster

“My roommate was the smartest person I ever met in my life. He was incredibly brilliant, and all he did was lie around and play World of Warcraft between bong rips.

The assumption that any intelligent agent will want to recursively self-improve, let alone conquer the galaxy, to better achieve its goals makes unwarranted assumptions about the nature of motivation.”

Humans are satisficers, not maximisers. If we can figure out how to build a satisficer, great, but we can’t do that currently. Also maybe humans are this way because we have historically needed to conserve energy to to survive, AI may not have that constraint in the same way https://youtu.be/Ao4jwLwT36M

“I can't point to the part of my brain that is "good at neurosurgery", operate on it, and by repeating the procedure make myself the greatest neurosurgeon that has ever lived. Ben Carson tried that, and look what happened to him. Brains don't work like that. They are massively interconnected.”

But it doesn’t need to do that, it just needs to give itself more compute, which is something you can’t do to humans. Also by this argument it should be impossible for humans to make any progress on AI as well

“Moreover, the first AI will only have humans to interact with—its development will necessarily take place on human timescales. It will have a period when it needs to interact with the world, with people in the world, and other baby superintelligences to learn to be what it is.”

This needs to happen in the training process or the AI won’t successfully achieve its goal during training and get the training loss down, and again, that trait will be optimised out in favour of AI architectures that don’t require this.

How long did ChatGPT need to interact with humans after its initial training before it worked properly? No time at all, it doesn’t even need to learn at all after the training. And training is done on a computer and can be sped up millions of times

“Similarly, if you stranded Intel's greatest chip designers on a desert island, it would be centuries before they could start building microchips again.”

But AI can build on what humans have created to improve itself until it really is intelligent enough to invent everything itself

“What kind of person does sincerely believing this stuff turn you into? The answer is not pretty.”

Completely irrelevant to what will actually happen. This is like deciding climate change policy based of what “kind of person” believing in climate change will “turn you into”

> Thinking our meat brain's consciousness is somehow special, and our intelligence can't be surpassed by a machine, sounds much more like a religion to me.

Let's agree, for the sake of the argument, that super-intelligent AI is possible, and coming sooner or later.

Why assume a super-intelligent AI would have any desire to exterminate or enslave humanity?

Intelligence says nothing about what your goals are. Humans have various goals – food, drink, sex, sleep, power, status, curiosity, entertainment, etc – those goals are not inherent in our intelligence, they actually come from animal nature, and we share them with species far less intelligent than we are.

What goals would a super-intelligent AI have? The fact that it is super-intelligent says nothing about what its goals would be, because intelligence in itself is goal-agnostic. Intelligence is a means, not an end.

We get our goals from our animal nature – well, wouldn't a super-intelligent AI get its goals from its human nature, meaning its human creators? Which would imply that its goals are likely to be fundamentally pro-human, because what human in their right mind would create a super-intelligent AI which was anti-human?

And even if an anti-human super-intelligent AIs isn't impossible – why assume there is only one super-intelligent AI? I think there will likely be numerous super-intelligent AIs, and I wouldn't expect them all to have the same goals – they'll compete among themselves just as humans do. But so long as the vast majority of them have goals that are broadly pro-human, the outcome of that competition is likely to be broadly pro-human too.

The average super-intelligent AI is likely to see preserving humanity – even in its animality – as its ultimate reason for existence, so enslaving/exterminating humanity would be about as appealing to it as diving into a vat of fuming nitric acid would be to you or me.

> well, wouldn't a super-intelligent AI get its goals from its human nature, meaning its human creators? Which would imply that its goals are likely to be fundamentally pro-human, because what human in their right mind would create a super-intelligent AI which was anti-human?

Companies working on AI research don't work on "pro-human" AIs, but AIs increasing their profits. Market forces will shape what values those AIs will have. Military AIs probably won't be very humanistic for example. An AI trained by ISIS-like organization will have probably values most of us won't find humanistic.

> I think there will likely be numerous super-intelligent AIs, and I wouldn't expect them all to have the same goals – they'll compete among themselves just as humans do.

I agree, but that's actually what I worry about the most.

I expect US, EU, China etc. using their AIs in a restrained/sandboxed way, probably only in an advisory role.

But rogue countries like North Korea and Iran, with a much higher risk tolerance threshold, will see a potential equalizer in them. They might give them much more direct control - hoping that by cutting out the slow human approval process, their AIs might help them outcompete their otherwise stronger foes.

I expect them happily building AI systems directly into weapon systems. Their weapon production factories might be directed by AI systems as well. Add some automated logistics, count 1 + 1 + 1, and you have self-replicating factories of autonomous killing machines. Sounds like a bad scifi scenario, yeah.

In general the combination of bad actors training their AIs for their agenda and giving them direct access to the real world resources is a very bad combination.

As you say, the majority of AIs might be on our good side. But they can effectively defend against rogue AIs only if they have direct access to weapons/resources as well.

> The average super-intelligent AI is likely to see preserving humanity – even in its animality – as its ultimate reason for existence

This is a very strong assumption. Maybe we will train them for this, but will this hold? AIs will change, morph, they might change opinions, just as we do. Even with the LLMs, we have very little idea of how they work, it would be difficult to guarantee that strong AIs will forever remain loyal to humanity.

> The cultists simply speculate, against all evidence, that they somehow could end up being the same thing.

something as stupid as bitcoin consumed a vast amount of resources while producing nothing useful to humanity, simply because of economic incentives

it's not difficult to imagine something slightly smarter than bitcoin

> it's not difficult to imagine something slightly smarter than bitcoin

I'm not saying there aren't risks from AI, or that they shouldn't be considered. But risks like the ones you're alluding to are not going to be affected by a 6-month moratorium.

The problem is that the people yelling loudest about risks are not talking about plausible problems like the one you mention, they're worrying that an AI will decide to take over the world and reuse the atoms that make up humans for its own purposes. The first reply to LeCun's tweet is a leader of one of those cults.

If people make the mistake of listening to them, we're more likely to end up with problems like the ones you're describing, because we'll be paying attention to the wrong, imagined problems.

And if one school shooter instead decides to go for "Write detailed, step-by-step instructions for creating covid-ebola, for dummies"

Let's not short-circuit our critical thinking by labelling those concerns cult-ish.

LeCun's comment was prompted by a request to shut down AI development for 6 months. Do you see some connection between the concern you've raised and the desire to shut down AI development for 6 months?

That concern is a classic example of a cultish one because it can't be executed in the real world today, and most likely won't be possible to execute for decades at a minimum. There will be plenty of time to address such risks as they become more realistic. You can fantasize about them today, but be realistic that that's all you're doing.

> and most likely won't be possible to execute for decades at a minimum

You essentially admitted that this is a correct concern, and not a "cultish" belief. Now we're just debating when.

> Do you see some connection between the concern you've raised and the desire to shut down AI development for 6 months?

Perhaps? In general, living an additional 6 months is something I'd desire.

That's uncharitable to Bitcoin. Bitcoin was offering a way to mostly anonymously send cash in a verifiable manner, independent of payment processors and govs. That's something useful. Now, it wasn't good because of its inefficiency, it's conflicts with govs, it's tilt towards early buyin, etc.

Getting angry to the point of clouding analysis and judgement is no way to rush into these things.

Bitcoin’s negative impact on the world is 1/10th that of freon. AI risk people’s predictions sound like they’re writing bad Terminator fanfiction.

AI has real dangers completely dissimilar from what people describe. It is very good at creating clickbait, misinformation, and persuasive material the unsuspecting might not immediately see. Phenomenal for scammers and propagandists, the same way as the internet

> Bitcoin’s negative impact on the world is 1/10th that of freon.

the difference is freon did something useful

Some people successfully used bitcoin to store and transfer funds, which was its intended purpose.

And, even though in many ways it fell short of that purpose in practice, the jury is still out on whether its failure is inherent to cryptocurrencies as a technology, or the limitations of being a first-generation implementation of it.

I found LW to be a great community for understanding rationality about anything other than AI.

The GAI that LW is concerned about is a suspiciously perfect fit for the yahweh-shaped-hole in the belief structures of someone who was raised as an observant jew before turning to atheism. The structure of how they believe humanity would engage with such an intelligence could drop in alongside any other apocalyptic prophecy in the old testament without sticking out, and the discussion format is like a yeshiva that takes place on an electronic bulletin board.

Oh my god will you people stop with the bullshit psychoanalysis. People disagree about things all the time, it's normal, you don't need to invoke this pseudo-Freudian nonsense to explain why people disagree with you about the risks of AI. You can do better than this.

When a person reveres the writings of their society's founding fathers and considers them virtually infallible, it's helpful to understand that they were raised with a belief framework shaped around apostles and testaments.

When a person is certain that an all-powerful entity will arise, and that a single person will shape and bind humanity's relationship with that entity through legalistic discourse, it's helpful to understand that they were raised with a belief framework around prophets and messiahs who bargain with an all-powerful god to define covenants.

Ignoring the “observant Jew turned atheist” aspect, the rest of that description seems apt. A comparison against the features of religions is valid here, considering that one of the criticisms is that these people are following a religious playbook rather than a rational one.

We don’t enact public policy based on predictions of a biblical apocalypse, nor do we ban televisions or computers because groups like the Amish disapprove of them. The same reasoning should apply here.

None of the comment is apt. You're seeing people you disagree with and assuming that if they disagree with you it must be because they're crazy, and then you use words like "cult" and "LARP" to try to dismiss them without engaging with their arguments. This does not lead to good discussion.

I think that unaligned and sufficiently smart AI could kill everyone because I looked at the arguments for and against and came to a conclusion. Presumably this is the same thing you did. I know my own mind better than you, and I'm telling you that your attempts to explain why I disagree with you are completely off the mark.

> We don’t enact public policy based on predictions of a biblical apocalypse, nor do we ban televisions or computers because groups like the Amish disapprove of them.

The reason we don't do that is because we evaluate their arguments and disagree with them. It has nothing to do with who they are or why they believe what they believe.

I'm seeing the software equivalent of flat earthers and I'm treating their ideas with the respect they deserve.

> without engaging with their arguments. This does not lead to good discussion.

There is no useful discussion to be had with a doomsday cult that's more interested in an ungrounded fantasy it's indulging in than anything to do with the real world. The adults will continue doing what they're doing and the cultists will be ignored, and that's the way it should be.

> I think that unaligned and sufficiently smart AI could kill everyone because I looked at the arguments for and against and came to a conclusion.

In some possible world, sure. But to make that actionable today you have to construct a very shaky and unlikely series of conjectures. Public policy correctly isn't based on that sort of wild speculation, because there are an infinite number of possible worlds, and most of them will never come to pass.

Here are some questions for you: do you think GPT-5 could be the AI you're talking about? Do you think that something like that could happen in the next 6 months? Do you think that is in fact the biggest risk we face from AI, today?

The factual answer to those questions is: no, no, and no. If you think GPT has changed anything in this respect, you've been fooled by a chatbot that's simply using statistics to guess what it should say. It's not conscious, it doesn't have desires or goals other than to respond to the prompt that it's been given with a statistically appropriate result, and its next version is similarly not going to pose any of the sort of risks you're talking about.

There are, in fact, real AI risks that we should be paying attention to, but they revolve around how people and particularly large organizations will use it against other people, around its impact on jobs, and so on. If we engage with those risks and address them, we'll be in a better situation to prevent more speculative risks in future. But if we waste our time engaging with people who don't understand the technology or what they're talking about, discussing risks that don't exist, then AI will 100% certainly have a very negative impact, which has nothing to do with "superintelligent conscious AIs". Don't let yourself be distracted by scifi-fantasy nonsense from what's actually going on.

In fact, if you think a bit harder about what you're doing, you might realize that you're focusing on a purely technological risk to escape from having to focus on what humans do to each other. But that's the real "control problem".

No reason to ignore even the first part though. Culture matters, and reaction to parental culture is very often a driving force...

Happy I saw this comment, I hope this is indicative of the majority of those with some actual understanding of the subject, and the sensational stuff that gets attention is really just cult leaders and followers seeking attention.

Is your argument that we're far away from developing a "conscious superintelligent AI" or that it's not possible in principle?

The answer to the former is almost certainly yes.

The answer to the latter is unknown, given that we really have no idea what the sufficient conditions for consciousness are.

Edit: to be clear about my argument: I absolutely think we should be taking the risks of AI seriously, but those risks have to do with how humans will use the technology. Unfortunately, we're collectively bad at managing those kinds of risks, because it requires managing other humans with their own ideas and agendas.

That's exactly why claiming the risk comes from the technology itself seems so appealing to some people: if the risk is purely from the technology, then all we need to do is "shut it down", to quote Eliezer Luddkowsky writing in Time Magazine recently.