On the Catastrophic Risk of AI
schneier.com
schneier.com
> The press coverage has been extensive, and surprising to me. The New York Times headline is “A.I. Poses ‘Risk of Extinction,’ Industry Leaders Warn.” BBC: “Artificial intelligence could lead to extinction, experts warn.” Other headlines are similar
I wish we would put this level of urgency and worry into addressing human suffering in the world instead of thinking of self-serving self-aggrandizing fantastic sci-fi futures that are just distracting us from the current reality of society
A lot more people suffer and will suffer everyday from inequality, health issues and loneliness than anything that any AI will do to them in their lives
All I’ve read is science fiction from man-boys speculating on some sort of fanciful future that isn’t even close to current reality.
People like Sam Altman catastrophizing to politicians and panicking people about “Tom Swift and the electric brain” is frankly doing the industry a disservice and embarrassing.
AI ending humanity is on the same likelihood level as man-boy Musks dreams of colonizing Mars…. I know people love the fantasy, but it will never be real.
People should just stop with this made up stuff or the politicians will believe you and their stupidity is the real risk.
Climate change is a million times more real existential threat.
Our history is replete with examples of the slightly more technologically advanced group decimating their competition.
Humanity is directly responsible for the extinction of hundreds of species. Not because of any particular malice or offense, simply that our goals didn't align with the interest of any of those species.
Just looking at Humans alone, Unaligned intelligence is potentially catastrophic even when the advantage gulf is fairly low.
When the advantage gulf is large, it is potentially genocidal.
Turn it around for a second. What does it take for humanity to have a good future after we create intelligences smarter than us? Let's say the AGI listens to and obeys humans. Well, one thing we need to avoid is just an absolute chaos of terrorism and war. It's easier to do damage than to prevent it, and if everyone has an obedient, mastermind scientific genius in their back pocket, you could have it invent a pathogen and cure in tandem, keep the cure secret and release the pathogen, and do this over and over until your enemies are all dead.
So we need AGI that is aligned with humanity, in the sense that it won't do stuff that hurts people even if someone tells it to. How do we do that, and not potentially end up with AGIs that can't be corrected if they get some idea that turns out to be wrong? Ideally, we'd make AGIs that have a bias toward inaction, that are "lazy" in a sense, and try to do the minimum they can. But those AGIs won't be as useful as ones that work like eager beavers toward some goal, so there will be selection pressure towards those.
I think the way to look at it is like with evolutionary biology. The pressure that evolution puts on animals to develop certain genes/behaviors is similar to the pressure that humans will put on AI development, and it ends with AI being more capable and more likely to follow orders even if those orders mean bad things for other people. As they get more and more powerful, this means chaos. It's monkeys throwing poo, vs monkeys throwing rocks, vs monkeys with guns, vs monkeys with rocket launchers.
And look -- I'm not certain that things will end in disaster. But inventing something smarter than you and trying to stop it from taking over sooner or later seems very difficult, and I can't even get a good answer out of people who think it'll be fine, or good. The default case is not that it'll be fine, the default case is we have no idea how it'll go. It's a grey fog. Just because humanity has muddled along and managed not to destroy ourselves so far, does not mean that we're safe, by any means.
Is it embodiment, agency ? because it's fairly trivial (albeit expensive) to grant those with LLMs.
I don't think LLMs are anything like an AGI. I think that a lot of people are experiencing very strong pareidolia. They're falling for an illusion.
>I think we're not even close to having one.
Why is this the default assumption until proven otherwise? The issue is determining what decisions bring the most expected utility. The default assumption should be downstream from the utility analysis. We don't know whether we're close or far from AGI, but it sure seems like the timelines are a lot shorter than any of us originally thought. We need to prepare society for AGI (or prevent its creation) well before it is realized. Banking on an alarm that will alert us to its imminent creation is looking more and more like wishful thinking. So the course of action that maximizes utility is to assume we're close and prepare for it.
How true that is depends on how you define "better". But we've been making lots of tools over thousands of years that can perform tasks better than people. You need more than that to declare a thing "intelligent" in the sense the average person thinks of it.
> To place your bet that we will never reach AGI is bone-headed.
Never is a very long time, but I think it's a solid bet that it won't happen within the next few decades, at least.
> Why is this the default assumption until proven otherwise?
In the absence of solid evidence for the existence of a thing, assuming its nonexistence is the logical position.
> We need to prepare society for AGI (or prevent its creation) well before it is realized.
We agree on this. But that's not what's happening. What's happening is some people are scaring the hell out of other over an imminent that there is no evidence exists.
We have lots of time to consider these things carefully and to react with calm prudence.
> So the course of action that maximizes utility is to assume we're close and prepare for it.
Not in the way it's happening. The way its happening will, in my opinion, actually increase the harm that AI is threatening to cause, and will distract from the actual risks that AI as it exists right now poses.
This is what's so insidious about this debate, people are completely unable to reason with uncertainty. No, you do not know that we have "lots of time". Maybe we do, maybe we don't. But we do know that acting too late is far more dangerous than acting too early. Thus we should be biased towards implementing safeguards and guardrails now.
>and will distract from the actual risks that AI as it exists right now poses.
Perhaps this is right, and I have plenty of sympathy for this. But the answer is not to pretend there is no x-risk concern at all. The answer is to make sure we're taking seriously all manner of AI risk. Once we all agree we should be having these discussions, it will be much easier to discuss the more immediate risks as well.
I entirely agree, but that's not what's happening. What's happening is that some people are focusing entirely on the most extreme, and extremely unlikely risks and completely ignoring the much more likely risks.
Between the extreme hype levels and the fearmongering from influential people, the pool is well poisoned and having a reasonable discussion about risks is essentially impossible.
> Once we all agree we should be having these discussions, it will be much easier to discuss the more immediate risks as well.
I don't think anyone is thinking we don't need to have these discussions. But the discussions that we need to have aren't the ones we're having.
I disagree very strongly both with this statement, and the idea that there is no evidence we can create AGI. Absence of evidence is not the same thing as evidence of absence.
It would make sense to be skeptical if people were asserting the existence of something for which we have very strong prior beliefs that this thing does not exist. To say that there's an enormous green monster hiding on the back side of Jupiter, for example. Until we've looked at the back of Jupiter, we have no idea what's there, but why would it be enormous, and green? To say that there's life on Jupiter, that's a tougher one -- we could say that we are not sure, but based on our understanding of Jupiter's atmosphere, we think it's unlikely. If we did not know what Jupiter was made of, it would be perfectly reasonable to assume that it might host some sort of life. It would be wrong to assume that because we have no evidence of life there, that there is no life. If we had carefully inspected thousands of planets and never found life, it would be much more reasonable to assume there is no life on Jupiter. This is the whole idea of Bayesian reasoning.
But with AGI, it's not like we're searching the universe to see if it exists. There are a ton of smart people actively TRYING TO CREATE IT. The evidence that AGI will happen is what people have already accomplished. We've got goalposts on rails, we've been moving them so much. An AI that understands language, and can respond with perfect grammar, has a theory of mind comparable to a 9-year old, is already mind-blowing, sci-fi tech. And OpenAI is claiming that they have not reached the limit of scalability with bigger models yet. The models we see aren't even the smartest ones -- they're the ones that have been neutered by fine-tuning to be less offensive and less likely to give out dangerous info.
To claim that AGI will not happen, you need some very strong evidence that the people actively trying to create it, who believe they can create it, and who have had some massive successes up until this point, will fail. To assume that it won't be created because we haven't seen it yet, doesn't make any sense at all.
Obviously you don't which is why i'm asking you to set concrete goals on what quantifies AGI. No point in saying we don't have AGI when i don't know what AGI means to you.
To the extent that is true, it's because we're assigning it tasks that it is especially well-suited to do:
- writing filler copy
- generating boilerplate or commonly-used code
- rephrasing words
- summarizing
- performing a medical exam on my dad to see if he needs vascular surgery
- cleaning my bathroom
- writing a performance review for a software engineer
- investigating a react codebase that uses redux in some places and not others to determine a strategy for better state management
- driving my daughter to the gym after school
GPT-4 can investigate a relatively small codebase and determine a state management strategy. Especially the 32k version.
Waymo has had self driving cars deployed for years. Tesla is doing it effectively on a massive scale but with the humans as a backup.
Cleaning the bathroom requires a bit more dexterity than is available now but the AI is about there.
To look at one of your examples:
> GPT-4 can write performance reviews given the right context.
The context is the performance review. Understanding what an employee has done relative to their responsibilities and goals is the entire point. Taking that context and packaging it into a review format (the thing an LLM could _maybe_ do adequately) is trivial compared to the observations and analysis that build that context.
The claims that I responded to were the exact wording in your comment. If you wanted to make a different claim then you should have written that. It seems you want to modify your claim.
In fact GPT-4 could make the observations and analysis to build that. I was using "context" in a technical way here. All I was stating was that it needs the data to be input in some way.
- cleaning my bathroom
- writing a performance review for a software engineer
- investigating a react codebase that uses redux in some places and not others to determine a strategy for better state management
- driving my daughter to the gym after school
Personalized robot assistance - https://tidybot.cs.princeton.edu/
Software development - https://ai.googleblog.com/2023/05/large-sequence-models-for-...
I'm genuinely curious (i.e., this isn't a rhetorical question) because that seems to be a common assumption among folks who talk about intelligence, and I honestly don't understand it.
For example, I enjoy rock-climbing as a hobby. Doing it involves literally learning new patterns of movement. That's part of what makes it fun. I don't understand why that isn't considered intelligence, while learning the steps to, say, invert a matrix is.
Sure but does your ability to learn new patterns of movement have much to do with having the hands to climb ? Do you think an amputated person would be less intelligent than you because he could not climb the same rocks ?
Would a blind man be less intelligent than you because he couldn't drive ?
You can't go and ask GPT-4 to go clean your room right now.
But is it because it's not intelligent enough to understand what it means to tidy up your room and go about doing that ? Or is the default inability to see and touch the real issue ?
Many papers/demonstrations including what I linked point overwhelmingly to the latter.
The moment we built something showcasing strong general intelligence, we began to use it to make cognitive decisions and take actions in our stead.
Now an LLM browses the web for you. And as soon as June with Windows, it'll take actions on your computer for you as well.
If anything this recent LLM era has shown, it's that general AI won't need to escape any "box" because two sides will be missing. I don't think the idea that whatever Super Intelligence we build won't have influence on the real world will pan out at all.
Even the most isolated form of Super Intelligence will have Humans taking instructions from it. Instructions that you won't really be able to verify is ultimately for your own good.
And the fact of the matter is, long term memory is actively being worked on. You can already approximate reasonably with a Vector DB and embeddings, feeding the relevant results into the context.
As is increasing the "working memory" (context size) in a more efficient manner: https://arxiv.org/abs/2305.16300
This isn't even getting into using things like Tree of Thought techniques to improve reasoning abilities and limit the search space for each portion of the problem to a reasonable size.
Are we talking about individual levels of intelligence, or are we talking about groups with advanced technology? Do you understand why those are not the same thing?
Look at how we're giving LLMs more and more agency, embodiment and tool control. They're being "plugged" into everything. Palantir even has a military use LLM they're excitedly showcasing. They're several popular repos that give control of your terminal and browser to LLMs. The moment we built something showcasing strong general intelligence, we began to use it to make cognitive decisions and take actions in our stead.
Now an LLM browses the web for you. And as soon as June with Windows, it'll take actions on your computer for you as well.
If anything this recent LLM era has shown, it's that general AI won't need to escape any "box" because two sides will be missing.
No point saying "well a person can't kill thousands on his/her own so an ai won't be able to" when you have the ai influencing military tactics on a global scale.
General Super Intelligence will evidently not be as blacked boxed as any one human and that's not even taking into account whatever advantage super intelligence will bring.
My understanding of the discussion around risks is that it is the future potential of the technology that is worrying people, together with what currently seems like irresistible pressure to enhance its capabilities. An adjacent worry is that the technology might become rapidly self-enhancing beyond some point in the future.
Musk isn't someone I take seriously in this domain - but there are some very thoughtful and smart people trying to draw attention to these issues. I think that dismissing them all as "man boys" is uncharitable and, honestly, shallow.
Why do we assume that we can have a general intelligence, but we can make it do what we tell it? What keeps it from telling us to shove our objectives, it's going to do what it wants?
- Rob Miles
In the future this might be how we create AGI, without a deeper understanding of it.
This will be worse than any kernel bug because at least in OS security no one dies if a bug is found and you have multiple tries to bug fix your OS. AGI will be guaranteed to have bugs, like everything does.
So we should do nothing at all then? And actively make the risks worse if it serves narrow national, political interests?
Similar thing with AI. But actually probably more dangerous.
Maybe Schneier isn't as smart as everyone thinks he is. GPT-4 is already almost human level intelligence. (Notice that I did not say that it was alive or that it was a simulation of a person with all of the other human characteristics). All it takes is for them to keep making computers faster and more efficient like they have been since it started. Within five years or less it will be thinking 100 times faster than humans (and quite possibly double the IQ).
Then someone people gives a large group of them an overly broad instruction like "take control".
GPT-4 is truly an effective problem solver and similar systems will be available within a few years that are orders of magnitude faster. That is an extinction risk. You cannot compete with it as a human. For this type of system, the human activities or inputs are so slow that they seem to be frozen.
People may think that computing progress has ended or slowed down or something and it can't get that much faster, but for starters this is a very specific application which has room for optimization in hardware, software, and AI models. And there are entirely new compute-in-memory paradigms coming (a new S curves).
It is honestely unpredictable. And it concerns only GIGANTIC neural nets several orders of magnitude above a human brain (Telsa giga factory size, minimum).