If you agree with him on the likelihood of superintelligence in the next few decades and the difficulty of value alignment, what course of action would you suggest?
Once you believe some malevolent god will appear and doom us all, anything can be justified (and people have done so in the past - no malevolent god appeared then, btw.).
I mean, that's motivated reasoning right there, right?
"Agreeing that existential risk is there would lead to complex intractable problems we don't want to tackle, so let's agree there's no existential risk". This isn't a novel idea, it's why we started climate change mitigations 20 years too late.
When you see a gun on the table, what do you do? You assume its loaded until proven otherwise. For some reason, those who imagine AI will usher in some tech-utopia not only assume the gun is empty, but that pulling the trigger will bring forth endless prosperity. It's rather insane actually.
Further, our success as a species doesn’t come from lone geniuses, but from large organizations that are able to harness the capabilities of thousands/millions of individual intelligences. Assuming an AGI that’s better than an individual human is going to automatically be better than millions of humans - and so much better that it’s godlike - is disconnected from what we see in reality.
It actually seems to be a reflection of the LessWrong crowd, who (in my experience) greatly overemphasize the role of lone geniuses and end up struggling when it comes social aspects of our society.
But this is the question I will ask... Why is the human brain the pinnacle of all possible intelligence in your opinion? Why did evolution manage to produce the most efficient possible, the most 'intelligent' format via the random walk that can never be exceeded by anything else?
Individual humans are limited by biology, an AGI will not be similarly limited. Due to horizontal scaling, an AGI will perhaps be more like a million individuals all perfectly aligned towards the same goal. There's also the case that an AGI can leverage the complete sum of human knowledge, and can self-direct towards a single goal for an arbitrary amount of time. These are super powers from the perspective of an individual human.
Sure, mega corporations also have superpowers from the perspective of an individual human. But then again, megacorps are in danger of making the planet inhospitable to humans. The limiting factor is that no human-run entity will intentionally make the planet inhospitable to itself. This limits the range of damage that megacorps will inflict on the world. An AGI is not so constrained. So even discounting actual godlike powers, AGI is clearly an x-risk.
> Individual humans are limited by biology, an AGI will not be similarly limited.
On the other hand, individual humans are not limited by silicon and global supply chains, nor bottlenecked by robotics. The perceived superiority of computer hardware on organic brains has never been conclusively demonstrated: it is plausible that in the areas that brains have actually been optimized for, our technology hits a wall before it reaches parity. It is also plausible that solving robotics is a significantly harder problem than intelligence, leaving AI at a disadvantage for a while.
> Due to horizontal scaling, an AGI will perhaps be more like a million individuals all perfectly aligned towards the same goal.
How would they force perfect alignment, though? In order to be effective, each of these individuals will need to work on different problems and focus on different information, which means they will start diverging. Basically, in order for an AI to force global coordination of its objective among millions of clones, it first has to solve the alignment problem. It's a difficult problem. You cannot simply assume it will have less trouble with it than we do.
> There's also the case that an AGI can leverage the complete sum of human knowledge
But it cannot leverage the information that billions of years of evolution has encoded in our genome. It is an open question whether the sum of human knowledge is of any use without that implicit basis.
> and can self-direct towards a single goal for an arbitrary amount of time
Consistent goal-directed behavior is part of the alignment problem: it requires proving the stability of your goal system under all possible sequences of inputs and AGI will not necessarily be capable of it. There is also nothing intrinsic about the notion of AGI that suggests it would be better than humans at this kind of thing.
>How would they force perfect alignment, though? In order to be effective, each of these individuals will need to work on different problems and focus on different information, which means they will start diverging
I disagree that independence is required for effectiveness. Independence is useful, but it also comes with an inordinate coordination cost. Lack of independence implies low coordination costs, and the features of an artificial intelligence implies the ability to maximally utilize the abilities of the sub-components. Consider the 'thousand brains' hypothesis, that human intelligence is essentially the coordination of thousands of mini-brains. It stands to reason that the more powerful the mini-brains, along with the efficiency of coordination, implies a much more capable unified intelligence. Of course all that remains to be seen.
Perhaps, but it's not obvious. Lack of independence implies more back-and-forth communication with the central coordinator, whereas independent agents could do more work before communication is required. It's a tradeoff.
> the features of an artificial intelligence implies the ability to maximally utilize the abilities of the sub-components
Does it? Can you elaborate?
> It stands to reason that the more powerful the mini-brains, along with the efficiency of coordination, implies a much more capable unified intelligence.
It also implies an easier alignment problem. If an intelligence can coordinate "mini-brains" fully reliably (a big if, by the way), presumably I can do something similar with a Python script or narrow AI. Decoupling capability from independence is ideal with respect to alignment, so I'm a bit less worried, if this is how it's going to work.
I don't intend to say anything controversial here. The consideration is the tradeoff between independence and tight constraints of the subcomponents. Independent entities have their own interests, as well as added computational and energetic costs involved in managing a whole entity. These are costs that can't be directed towards the overarching goal. On the other hand, tightly constrained components do not have this extra overhead and so their capacity can be fully directed towards the goal as determined by the control system. In terms of utilization of compute and energy towards the principle goal, a unified AI will be more efficient.
>If an intelligence can coordinate "mini-brains" fully reliably (a big if, by the way), presumably I can do something similar with a Python script or narrow AI.
This is plausible, and I'm totally in favor of exploiting narrow AI to maximal effect. If the only AI we ever had to worry about was narrow AI, I wouldn't have any issue aside from the mundane issues we get with the potential misuse of any new technology. But we know people (e.g. open AI) are explicitly aiming towards AGI so we need to be planning for this eventuality.
We don't have any historical record if those peoples had discussions about possible scenarios like this. Where there imaginary guns in their future? What we do have records of is another people group showing up with a massive technological and biological disruption that nearly lead to their complete annihilation.
Btw., the conquistadors relied heavily on exploiting local politics and locals for conquest, it wasn't just some "magic tech" thing, but old fashioned coalition building with enemies of enemies.
But to answer the question, the danger of AI is uniquely relevant because it is the only danger that may end up being totally out of our control. Nukes, pandemics, climate change, etc. all represent x-risk to various degrees. But we will always be in control of whether they come to fruition. We can pretty much always short circuit any of these processes that is leading to our extinction.
A fully formed AGI, i.e. an autonomous superintelligent agent, represents the potential for a total loss of control. With AGI, there is a point past which we cannot go back.
The differentiating thing here is that blocking hypothetical AI risk is cheap, while mitigating real risks is expensive.
We can work on it in the long term and things like developing safe AI may have more impact on mitigating asteroid risk than working on scaling up existing nuclear, rocket, and observation tech to tackle it.
Is it that it's impossible to make something smarter than a human? Or that such a thing wouldn't have goals/plans that require resources us humans want/need? Or that such a thing wouldn't be particularly dangerously powerful?
A lot of existential risk involves bold hypotheses about machines being able to "solve" a lot of things humans cannot, but we don't know how much is actually just solvable by superior intelligence vs. what just doesn't work in that way. Human collective intelligence failed on a lot of things so far. Even the basic idea of exponential scaling intelligence is a hypothesis which might not hold true.
Also, some existential risk ideas involve hypotheses around evolution and on how species dominate - again, might not be right.
Most of the big problems we have are in situations where we need to preserve complex aspects of reality that make the systems too chaotic to properly predict, so I suspect AI won't be able to do much better regardless of how smart it is. The ability to carry out destructive brute force actions worries me more than intelligence.
What would you say the odds are on each of those three components? And are those odds independent?
I'd think they are quite independent.
Not sure I have any odds on those individually, other than I consider the overall risk really really low. The way I see it, there are a few things working against the risk from pure intelligence to start with (as it is with humans btw., the Nazis were not all intellectual giants, for example) and then it goes down from there.
This naturally gives a probably of Doom at least 1/3rd.
I'd say anything above 1% is worth legislating against even if it slows down progress in the field IFF such legislation would actually help.
We can do more than one thing at once, and we need to.
if we can't even do one thing at once given decades of trying (deal with climate change), we definitely can't do that one thing plus another thing (deal with AI)
The probabilities are really just too shaky for me to estimate. Not sure I would put a high probability of intelligent thing being dangerous by itself, for example.
Another reason containment won't work is, we now know we won't even try it. Look at what happened with LLMs. We've seen the same people muse at the possibility of it showing sparks of sentience, think about the danger of X-risk, and then rush to give it full Internet access and ability to autonomously execute code on networked machines, racing to figure out ways to loop it on itself or otherwise bootstrap an intelligent autonomous agent.
Seriously, forget about containment working. If someone makes an AGI and somehow manages to box it, someone else will unbox it for shits and giggles.
Also, right now there is nothing to contain. The idea of existential risk relies on a lot of stacked-up hypotheses that all could be false. I can "create" any hypothetical risk by using that technique.
ASI is not human intelligence.
As a lower bound on what "superintelligence" means, consider something that 1) thinks much faster, years or centuries every second, and 2) thinks in parallel, as though millions of people are thinking centuries every second. That's not even accounting for getting qualitatively better at thinking, such as learning to reliably make the necessary brilliant insight to solve a problem.
> The idea of existential risk relies on a lot of stacked-up hypotheses that all could be false.
It really doesn't. It relies on very few hypotheses, of which multiple different subsets would lead to death. It isn't "X and Y and Z and A and B must all be true", it's more like "any of X or Y or (Z and A) or (Z and B) must be true". Instrumental convergence (https://en.wikipedia.org/wiki/Instrumental_convergence) nearly suffices by itself, for instance, but there are multiple other paths that don't require instrument convergence to be true. "Human asks a sufficiently powerful AI for sufficiently deadly information" is another whole family of paths.
(Also, you keep saying "could" while speaking as if it's impossible for these things to not to be false.)
I'm fairly certain that what you describe is physically impossible. Organic brains may not be fully optimal, but they are not that many orders of magnitude off.
As for your corporation example, I do not think the effectiveness of a corporation is necessarily bottlenecked by the number or intelligence of its employees. Notwithstanding the problem of coordinating many agents, there are many situations where the steps to design a solution are sequential and a hundred people won't get you there any faster than two. The chaotic nature of reality also entails a fundamental difficulty in predicting complex systems: you can only think so far ahead before the expected deviation between your plan and reality becomes too large. You need a feedback loop where you test your designs against reality and adjust accordingly, and this also acts as a bottleneck on the effectiveness of intelligence.
I'm not saying "superintelligent" AI couldn't be an order of magnitude better, mind you. I just think the upside is far, far less than the 7+ orders of magnitude the parent is talking about.
For example:
> The parent was positing, at a minimum, a 30 million fold increase in clock speed. This entails a proportional increase in energy consumption, which would likely destroy any thermal envelope the size of a brain.
No reason to restrict the size of an artificial mind to that of a human brain. With no need to design for being part of a mobile, independently-operating body, the design space grows substantially. Even on existing hardware, you can boost the serial processing speed by a whole order of magnitude if you submerge it in liquid nitrogen. I imagine you could get much greater speeds still on hardware designed specifically for cryogenic cooling with cryo liquids at much lower temperatures. Maybe not 7 orders of magnitude difference, but 3 seem plausible with custom hardware.
Corporations are a good example of what's effectively AIs living among us today, but their performance is indeed bottlenecked by many things. A lot of them don't seem fundamental, but rather a result of the "corporate mind" growing organically on top of the runtime of bureaucracy - it's evolved software, as opposed to designed one. E.g.:
> there are many situations where the steps to design a solution are sequential and a hundred people won't get you there any faster than two
That's IMO because managers and executives, like everyone else, don't see employees and teams as a computational process, and don't try to optimize them as one. You can get a boost on sequential problems with extra parallelism, but it would look weird if done with human teams. You could absolutely do pipelining and branch prediction with teams and departments, but good luck explaining why this works to your shareholders...
The point being, while corporations themselves won't become superintelligences, due to bottlenecks you mention, those limits don't apply to AIs inhabiting proper computers. 7 orders of magnitude advantage over humans may be way too much, but I don't see anything prohibiting 2-3, maybe even 4 orders, and honestly, even 1 order of magnitude difference is unimaginably large.
We know simple examples like exact numeric calculation where desktop grade machines are already over a quadrillion times faster than unaided humans, and more power efficient.
We could plausibly see some >billion-fold difference in strategic reasoning at some point even in fuzzy domains.
It also unclear to what extent thinking alone can solve a lot of problems. Similar, it is unclear if humans could not contain superhuman intelligence. Pretty unintelligent humans can contain very smart humans. Is there an upper limit on intelligence differential for containment?
Those trade off against each other and don't all have to be as easy as possible. Information sufficiently dangerous to destroy the world certainly exists, the question is how close AI gets to the boundary of "possible to summarize from existing literature and/or generate" and "possible for human to apply", given in particular that the AI can model and evaluate "possible for human to apply".
> Similar, it is unclear if humans could not contain superhuman intelligence.
If you agree that it's not clearly and obviously possible, then we're already most of the way to "what is the risk that it isn't possible to contain, what is the amount of danger posed if it isn't possible, what amount of that risk is acceptable, and should we perhaps have any way at all to limit that risk if we decide the answer isn't 'all of it as fast as we possibly can'".
The difference between "90% likely" and "20% likely" and "1% likely" and "0.01% likely" is really not relevant at all when the other factor being multiplied in is "existential risk to humanity". That number needs a lot more zeroes.
It's perfectly reasonable for people to disagree whether the number is 90% or 1%; if you think people calling it extremely likely are wrong, fine. What's ridiculous is when people either try to claim (without evidence) that it's 0 or effectively 0, or when people claim it's 1% but act as if that's somehow acceptable risk, or act like anyone should be able to take that risk for all of humanity.
Second, even if your argument is "genocide of anyone who sounds too smart was the right approach and they just weren't trying hard enough", that only really fits into "neither alignment nor containment, just don't have the AGI at all".
Containment, for humans, would look like a prison from which there is no escape; but if this is supposed to represent an AI that you want to use to solve problems, this prison with no escape needs a high-bandwidth internet connection with the outside world… and somehow zero opportunity for anyone outside to become convinced they need to rescue the "people"[0] inside like last year: https://en.wikipedia.org/wiki/LaMDA#Sentience_claims
[0] or AI who are good at pretending to be people, distinction without a difference in this case
We might not even need to contain, all hypothetical.
You can be killed because your storage medium and execution medium are inseparable. Destroy the brain and you're gone, and you don't even get to copy yourself.
With AGI/ASI if we can boot it up from any copy on a disk if we have the right hardware then at the end of the day you've effectively created the undead as long as a drive exists with a copy of it and a computer exists than can run it.
With something like an AI you can never be sure as you must follow the entire chain of thermodynamic evidence of the past to when the AI was created that no copy of it was ever made. Not just by you, by any potential intruder in the system.
Extermination might work, though only on models large enough that people aren't likely to sneak out of the office with copies for their own use and/or just upload copies to public places (didn't that already happen with the first public release of Stable Diffusion, or am I misremembering?)
Recall the timeless counterpoint: