It isn't obvious to me. And I've yet to read something that spills out the obvious reasoning.
I feel like everything I've read just spells out some contrived scenario, and then when folks push back explaining all the reasons that particular scenario wouldn't come to pass, the counter argument is just "but that's just one example!" without offering anything more convincing.
Do you have any better resources that you could share?
From the tech perspective: higher order objectives are insidious. While we may assume a narrow misalignment in received vs intended objective of a higher order nature, this misalignment can result in very divergent first-order behavior. Misalignment in behavior is by its nature destructive of value. The question is how much destruction of value can we expect? The machine may intentionally act in destructive ways as it goes about carrying out its slightly misaligned higher order objective-guided behavior. Of course we will have first-order rules that constrain its behavior. But again, slight misalignment in first-order rule descriptions are avenues for exploitation. If we cannot be sure we have zero exploitable rules, we must assume a superintelligence will find such loopholes and exploit them to maximum effect.
Human history since we started using technology has been a lesson on the outcome of an intelligent entity aimed at realizing an objective. Loopholes are just resources to be exploited. The destruction of the environment and other humans is just the inevitable outcome of slight misalignment of an intelligent optimizer.
If this argument is right, the only thing standing between us and destruction is the AGI having reached its objective before it eats the world. That is, there will always be some value lost in any significant execution of an AGI agent due to misalignment. Can we prove that the ratio of value created to value lost due to misalignment is always above some suitable threshold? Until we do, x-risk should be the default assumption.
1. AIs have made rapid progress in approaching and often surpassing human abilities in many areas.
2. The fact that AIs have some inherent scalability, speed, cost, reliability and compliance advantages over humans means that many undesirable things that could previously not be done at all or at least not done at scale are becoming both feasible and cost-effective. Examples would include 24/7 surveillance with social desirability scoring based on a precise ideological and psychological profile derived from a comprehensive record of interactions, fine-tuned mass manipulation and large scale plausible falsification of the historical record. Given the general rise of authoritarianism, this is pretty worrying.
3. On the other hand the rapid progress and enormous investment we've been seeing makes it very plausible that before too long we will, in fact, see AIs that outperform humans on most tasks.
4. AIs that are much smarter than any human pose even graver dangers.
5. Even if there is a general agreement that AIs pose grave or even existential risks, states, organizations and individuals will are all incentivized to still seek to improve their own AI capabilities, as doing so provides an enormous competitive advantage.
6. There is a danger of a rapid self-improvement feedback loop. Humans can reproduce, learn new and significantly improve existing skills, as well as pass skills on to others via teaching. But there are fundamental limits on speed and scale for all of these, whereas it's not obvious at all how an AI that has reached super-human level intelligence would be fundamentally prevented from rapidly improving itself further, or produce millions of "offspring" that can collaborate and skill-exchange extremely efficiently. Furthermore, since AIs can operate at completely different time scales than humans, this all could happen extremely rapidly, and such a system might very quickly become much more powerful than humanity and the rest of AIs combined.
I think you only have to subscribe a small subset of these (say 1.&2.) to conclude that "AI is an uniquely powerful and thus uniquely dangerous technology" obviously follows.
For the stronger claim of existential risk, have you read the lesswrong link posted elsewhere in this discussion?
https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a... ?
It seems likely that the majority of AI projects will be reasonably well aligned by default, so I think 1000 AIs monitoring what the others are doing is a lot safer than a single global consortium megaproject that humans can likely only inadequately control.
The only reasonable defense against rogue AI is prosocial AI.
Maybe I’m overly optimistic about the resilience of humans but these scenarios still don’t sound plausible to me in the real world.
Step 1. AI Step 2. #stuff Step 3. Bang
Maybe this is just what happens when you spend all your time on the internet...
I'm not that confident that if we put you in a box, tron-style, where you basically continued to enjoy your existing level of intelligence, but think 10'000x faster, have Petabytes of information at your fingertips and can clone yourself and losslessly and rapidly exchange knowledge with your clones and had a few days to think about it (~a few thousand years of thought at your normal speed) you couldn't figure out a way to effect a bunch of orders to DNA factories without anyone raising an alarm.
Are you?
Now what if we actually consider an actual AI after a few self-improvement steps. Any reasons to expect it wouldn't be 10'000x+ smarter than you as well, or roughly the difference in intelligence between you and an ant? Could you outsmart a bunch of ultra-ultra-slow-motion ants?
I mean... even orangutans can outperform humans at numerous tasks.
Computers have no intrinsic motivations, and they have real resource constraints.
I find the whole doomsday scenarios to be devoid of reality.
All that AI will give us is a productive edge. Humans will still do what humans have always done, AI is simply another tool at our disposal.
This is ambiguous. Do you mean
A. that there is some subset T1 of the set of all tasks T such that T1 is "most of" T, and that for each P in T1 there will be an AI that outperforms humans on P, or
B. There will be a single AI that outperforms humans on all tasks in a set T1, where T1 is a subset of all tasks T such that T1 is "most of" T?
I think A is unlikely but plausible but I don't see cause for worry. I don't see any reason why B should come to pass.
4. AIs that are much smarter than any human pose even graver dangers.
Sure. Why should we believe they will ever exist though?
Where we depart is point 4. Actually, both point 3 and 4 are things I agree with, but it's implied there's a logical link or progression between them and I don't think there is. The problem is the definitions of "outperform humans" and "smart".
Current AI can perform at superhuman levels in some respects, yes. Midjourney is extremely impressive when judged on speed and artistic skill. GPT-4 is extremely impressive whilst judged on its own terms, like breadth of knowledge. Things useful to end users, in other words. LLMs are deeply unimpressive judged on other aspects of human intelligence like long term memory, awareness of time and space, ability to learn continuously, willingness to commit to an opinion, ability to come up with interesting new ideas, hide thoughts and all that follows from that like being able to make long term plans, have agency and self-directed goals etc ... in all these areas it is weak. Yet, most people would incorporate most of them into their definition of smart.
Will all these problems be solved? Some will, surely, but for others it's not entirely clear how much demand there is. Boston Robotics was making amazing humanoid parkour bots for years yet the only one they seem able to actually sell is dog-like. Apparently the former aren't that useful. The unwillingness to commit to an opinion may be a fundamental trait of AI for as long as it's centralized, proprietary and the masses have to share a single model. The ability to come up with interesting new ideas and leaps of logic may or may not appear, it's too early to tell.
But between 3 and 4 you make a leap and assume that not only will all those areas be conquered very soon, but that the resulting AI will be unusually dangerous. The various social ills you describe don't worry me though. Bad governments will do bad things, same old, same old. I'm actually more worried about people using the existence of AI to deny true evidence rather than manufacture false evidence en-masse. The former is a lot of work and people are lazy. COVID showed that people's capacity for self-deception is unlimited, their willingness to deny the evidence of their own eyes is bottomless as long as they're told to do it by authority figures. You don't even need AI to be abused at all for someone to say, "ignore that evidence that we're clueless and corrupt, it was made by an AI!"
Then by point 6 we're on the usual trope of all public intellectuals, of assuming unending exponential growth in everything even when there's no evidence of that or reason to believe it. The self-improving AI idea is so far just a pipe dream. Whilst there are cases where AI gets used to improve AI via self-play, RLHF and so on, it's all very much still directed by humans and there's no sign that LLMs can self improve despite their otherwise impressive abilities. Indeed it's not even clear what self-improvement means in this case. It's a giant hole marked "??? profit!" at the heart of the argument. Neurosurgeons can't become superintelligences by repeatedly performing brain surgery on themselves. Why would AI be different?