Rogue superintelligence: Inside the mind of OpenAI's chief scientist

Their insight is just as valid as anybody else's. The difference is, as the people who can actually take that step forward, they have a unilateral ability to take it, not take it, or decide how to take it.

I think they are exactly the right amount full of themselves. Their insight may not be special, but what makes some bureaucrat's insight more valuable than theirs?

The irony is that they are opposing forces. AI as it is now is still tame, any technical improvement makes it more dangerous to not just economic fears of taking over jobs, but even the science fiction fears become viable if an AI becomes self aware and consistent in intent.

If you focus on the technical advances then you are focusing on NOT saving the world. Good that at least this guy isn't so wholly focused on the technical side even though saving the world is such a blurry concept.

Sakharov had some insight into the implications

Not that people, especially tech oligarchs, aren't full of themselves, but many physicists who were inventing the bomb, most famously Oppenheimer, also laid a lot of the philosophical basis of implications of nuclear weapons.

The situation may be a bit different now. The megacorporate world is the biggest threat from these tech developments, and the inventors are getting deeply embedded in this. Sort of similar situation that implications of nuclear weapons would be handled only by the military.

Respectfully, I totally disagree.

I am become death, destroyer of worlds.

The Nuremberg Trials.

And AGI is 10x impact of nuclear.

Builders are not cogs, and should not try to be them.

We need MORE ethics and good intentions - not less. It is the psychopathic corrupt business rot we should be afraid of - not people actively trying to do good

You are talking about the people not only doing the work and creating the wealth, but doing the innovations and creating new things. Then you say these are the people who are "too full of themselves". That they should give up on how the commodity they create is used - and by your definition it is a commodity. That they should focus on the needed innovations and that's it.

Well if the workers doing the work don't have a hand in making decisions, who does? We know the answer, from our current era of heirs, limited partners and such, the scions one can see on Rich Kids of Instagram, although they leave the work to their private wealth advisors.

It's the most parasitic idea possible. "Just do all the work and figure out the innovations slave, the aristocracy will take it from there".

That's an overly broad generalization, Leo Szilard certainly had insight into the implications:

> "With an enduring passion for the preservation of human life and political freedom, Szilard hoped that the US government would not use nuclear weapons, but that the mere threat of such weapons would force Germany and Japan to surrender. He also worried about the long-term implications of nuclear weapons, predicting that their use by the United States would start a nuclear arms race with the USSR. He drafted the Szilárd petition advocating that the atomic bomb be demonstrated to the enemy, and used only if the enemy did not then surrender. The Interim Committee instead chose to use atomic bombs against cities over the protests of Szilard and other scientists. Afterwards, he lobbied for amendments to the Atomic Energy Act of 1946 that placed nuclear energy under civilian control."

https://en.wikipedia.org/wiki/Leo_Szilard

There is an even more important lesson in this in that the people who worked on the bomb thought the world would be over in short order and were totally wrong.

Feynman talked about how there was an idea of "normal" people walking around not knowing they were basically doomed and going to die in a certain nuclear holocaust in a few years.

Von Neumann thought the U.S. should launch a nuclear first strike at Moscow. Obviously, if war is inevitable then you don't have to be the father of game theory to figure out you should strike first.

It was just a year ago that literally everyone was predicting we would be in a recession right now. We can't predict that but we can predict how AGI plays out even when we haven't bothered to define a measure of what AGI even is. Even people who grew up "knowing" we would all have sentient robots by 1997. I can't think of a single prediction I have heard in my lifetime that has turned out to be true other than the government debt going up.

They are focusing on technical advances to enable people to save themselves from future AGI. For that, they need more time since AI Safety research lags quite a bit behind capability research.

https://openai.com/blog/introducing-superalignment

I was speaking with a woman earlier this week who just finished her master's dissertation with a focus on Sam Altman's recent influence campaign, and she was scared of how charming he was. She obviously was trying to maintain distance for impartiality, but his interview style, and the seeming total sincerity of his communication style... It was terrifying to her in how well he could draw everyone in. She was absolutely endeared by him, but aware of how powerful that endearment was, and so scared and worried about his impact.

The OpenAI press release seemed to indicate explicitly that the 3 independent board members have the reins, not Ilya.

> The majority of the board is independent, and the independent directors do not hold equity in OpenAI.

It's something to think about. Even if Sam is 97% aligned and Ilya is 98.5% aligned towards the ideal mentality to have wrt AI safety, that 1.5% could be worth millions of lives in averaged risk

Chatgpt's consciousness would be akin to plato's cave. Even if chatgpt were more intelligent, wouldn't it be staring at different shadows?

Given that chatgpt has consciousness, would it be able to break the fourth wall? There seems to be an implicit assumption that it must break that in order to prove its consciousness to us. Maybe that's how AGI will come to be, because we desire to train it that way.

Probably being super naive, but stuff like that always makes me wonder, is there something special about human consciousness, even? Perhaps it's just a sliding scale and we're just seemingly ahead of other specifies. We assert that consciousness exists and distinguishes us, but is that really the case?

An llm is a database modelled on the concept of a neural (the tissue in our heads) net (because our neural tissue appears to be highly interconected like a net). An llm is a very interesting and complex tool that the creators and many users anthropomorphize this shit out of. Llm's are a spherical cow, a statistical model of a very poorly understood organ and difficult to define process; intelligence. Do we "prompt" our sql databases? When our querry results are incorrect is the database hallucinating? As amazing and usefull as these new ai tools are they are it seems to me they are no more intelligent or conscious than a hammer or guitar or the computer your using right now. I thinks a usefull perspective to apply to ai is competence. To clarify this I will compare an llm to a flea. A flea has a neural network that outperforms any hardware used for ai. Fleas eat, mate, avoid preditors and probably exibite a great deal of complex behavior that we are not aware of. Fleas are highly intelligent competent actors opperationg in a complex environment. Llms are not actors any more than a hammer or a sql database is and only an actor can be competent. I will agree that in most cases llms have more utillity than fleas. Consciousness seems alot more slippery than intelligence which is hard enough by itself.

Parents often let their children struggle and make imperfect decisions, and it's entirely possible (though definitely not guaranteed) that an AI superparent would do the same for us.

I think it's becoming clear that humans are fundamentally incapable of forseeing and understanding the consequences of the actions we are now capable of taking. It is likely that without some sort of super-governance that is fundamentally more capable than humans, we might not be able to survive as a species. Maybe AI can help solve that.

I don't think it's fair for you to claim a leading AI expert has critically wrong assumptions when "musing" about possibilities that relate to one of the most epistemologically difficult topics to investigate (consciousness).

You're rejecting Ilya's humble musings as having critically wrong assumptions, and then turning around to definitively explain how consciousness arises, and illuminating the relationship between consciousness, empathy, and intelligence, on a random hacker news thread. Frankly, you're making some huge claims about philosophy of mind that don't obviously track for me, and you provide no citations or arguments to support. I hesitate to accuse you of "hallucinating facts", but when you're issuing a takedown of one of the top AI experts I'd expect to see some more supporting argument.

Your definition of AGI is also a bit strange as it requires that it be fundamentally different from existing natural intelligences, if I understand correctly. That seems unnecessarily stringent to me, since if a program had the same kind and level of intelligence as me, I'd be inclined to say it is AGI.

I'm just not sure where all these confidently stated, very specific claims are coming from.

The architectures we know of so far have not been sufficient to achieve AGI with just text and image data. Humans and higher animals learn with much richer modalities than those two and probably would not be nearly as intelligent if forced to learn with just text and images. There are already ongoing efforts to train models with other modalities. Latest foundation models already go beyond pure LLMs.

Your reasoning above doesn’t mean some improvements to the current architecture(s) coupled with richer data would not be sufficient to achieve AGI.

There’s also a possibility OpenAI has recently achieved a yet undisclosed breakthrough.

Sam Altman at the APEC Summit yesterday:

"4 times now in the history of OpenAI — the most recent time was just in the last couple of weeks — I’ve gotten to be in the room when we push the veil of ignorance back and the frontier of discovery forward”

https://twitter.com/SpencerKSchiff/status/172564613068224524...

What's clear here is that users of OpenAI's products will end up in a worse place as a result of these developments. Ilya is on record as being against open sourced models with the view that they are too "powerful" to release. There are also accounts that Dev Day became a driving force for ousting Altman and stopping signups. Dev Day was about putting tools in the hands of users, so it's clear that his motivation is to restrict access to this technology. I don't want amateur philosophy from an LLM, I want greater capabilities and reduced costs. My hope is that this motivates user-focused competitors now that they have a sizeable window to catch up. So from my view Ilya will set back the field in the short term, but will spur competition in the long term.

I often observe that those dismissing this idea tend to be less informed about current insights into human cognition, philosophy, and concepts such as the information theoretic view of consciousness, neural correlates of consciousness, the free energy principle, and predictive coding.

The human mind is "just statistics on data".

People more informed than you are taking this seriously. You should pay attention and start inquiring why that's the case.

I very much disagree that it's not intelligent. GPT-4 is clearly intelligent, and we use the lobotomized version of it. So imagine what GPT-5 can do internally.

Conscious is hard to say, partly because we can't define it either, so it means something different for you and me.

This line of nonsense has become the new "Tell me you aren't current on the past 12 months of LLM research without telling me."

Harvard/MIT's Othello-GPT paper showing the development of what turned out to be linear representations of world models from training data that didn't explicitly contain that modeling is over a year old now.

That in turn inspired research showing linear representations in geographical mapping and in more traditional text models around truthiness vs falsehoods.

So we already have an increasing research trend that is showing over and over linear representations of more abstract modeling than "just statistics."

So you are wrong that LLMs with sufficient network complexity don't develop an understanding of the world (in parts).

And I'd encourage looking more into the difference between understanding the difference between training for next token prediction and the overall capabilities of the network with the smallest loss at that training task, particularly as network complexity increases.

OpenAI is a religious institution that self-selects for fundamentalism. This is why so many very good researchers passed on working there, they simply didn’t believe in The Mission Where Ilya Saves Humanity.

This is why you need to take classes other than computers and math, kids.

Here's a question to ChatGPT I just made up:

>> A magical frog was counting unicorns. He saw 5 purple unicorns, 2 green unicorns, and 7 pink unicorns. However, he made a mistake and didn't see 2 unicorns: one purple and one green. Also, since he was a magical frog, he didn't see unicorns that were the same color as himself. How many unicorns did he count?

It correctly answers 11 for me.

To me this has demonstrated:

* "Understanding": It understood that "didn't see" implies he didn't count.

* "Knowledge": It knew enough about the world to know that frogs are often green.

* "Reasoning": It was able to correctly reason about how many should be subtracted from the final result.

* "Math: It successfully did some basic additions and subtractions arriving at the correct answer.

Crucially, I made this up right here on the spot, and used a dice for some of the numbers. This question does not exist anywhere in the training corpus!

I think this demonstrates an impressive level of intelligence, for what up until about a year ago I thought a computer would ever be capable of in my lifetime. Now in absolute terms of course current gen ChatGPT is clearly far less good at reasoning and understanding than most people (well, specifically it seems to me that it's knowledge and reasoning are super-humanly broad, but child-level deep).

Can future improvements to this architecture improve the depth up to "AGI", whatever that means? I have no idea. It doesn't automatically seem impossible, but maybe what we see now is already near the limit? I guess only time will tell.

The problem with this line of reasoning is that the exact same thing can be said about our meat-calculator brains.

This gets to the philosophical heart of a debate that I can already foresee will NEVER be settled:

I guarantee you - with 100% certainty - that when we get to a point where AI is "AGI", there will be a continuous and massive political debate (akin to the abortion debate we face today) where one side argues that a given AGI is conscious and must be given rights and cannot be shut off and the other side argues that it's just a calculator and a computer program and computers can be turned off at will, erased, experimented on, and whatever.

We have the same debate today all the time! There are those who believe every human life is sacrosanct (from age 0-100+) and others who believe human life is disposable (from age 0-100+!). There's no reason to believe this debate won't extend to AGI.

Reminds me of the healthcare bots in Idiocracy.

Don't need to wait a decade, or even a half, for AI mental healthcare. It's already been tried.

https://www.theguardian.com/technology/2023/may/31/eating-di...

Americans would be wise not to give up their guns under any circumstances

I like to consider though that a super intelligence would not necessarily think in human ways 'kill everything in self interest' e.g., People, forests, animals, planet etc. Just because we humans act this way, doesn't mean AI will too. Fair enough to consider it, but equally, once it is intelligent, it will likely accelerate beyond our comprehension, and we tend to comprehend through fear and self interest, wisdom beyond humans is the opposite of this.

I doubt AI could or would do a better job of killing people and democracy than us humans.

The people who believe the vaxx bullshit don't read shit longer than a tweet anyway.