Is there a canonical source for “the argument for AGI ruin” somewhere
twitter.com
twitter.com
To repudiate that request with “everyone has a different objection” is nearly unthinkable. And to David Chalmers no less!
I think podcast hosts, tweet authors, bloggers and live streamers sometimes forget that progress in academic fields comes from real contributions, and that public conversations (especially oral) don’t really do anything besides spread common awareness.
But you don't understand, I did my own research
This is more like watching an argument between Feynman and Oppenheimer. They look like hot takes on twitter, but these guys have spent plenty of time thinking about it.
Yudkowsky is an AI ethics and safety researcher, also founder of LessWrong, an HN favorite – https://en.wikipedia.org/wiki/Eliezer_Yudkowsky
And Chalmers is a philosopher focusing on consciousness and cognitive science – https://en.wikipedia.org/wiki/David_Chalmers
I wouldn’t put their replies in exactly the same category as your typical social media reply guys.
If that is "damage", the problem must surely lie with academia, not with social media.
Academia's only raison d'être is to facilitate formal discourse. Wherever it doesn't do that, it loses its legitimacy. "I'm a researcher and you are not" is not an argument. In fact, I'd go so far as to claim that anyone who makes such an "argument" should be excluded from any future discourse.
The Wikipedia article on mutual assured destruction (https://en.wikipedia.org/wiki/Mutual_assured_destruction) traces the origin and evolution of exactly that analysis, including even pre-atom bomb precursors based on earlier weapons that would hypothetically make war "too terrible to ever happen again".
Then there's some less support for them being misleadingly wrong, much more rarely.
Science fiction Terminators it is not. Paperclip maximizers, however, yes. In fact it can be argued that a typical capitalist company is a variant of a paperclip maximizer and is a simple form of superhuman intelligence.
Nick Bostrom invented that thought experiment. Start from there and go digging through AI safety research of which there are a few journals.
Real life samples include 2010 flash crash, social media automatically optimizing for engagement at any cost, a few computer vision issues causing medical misdiagnosis, Tesla's "self-driving", video game AI exploiting engine bugs rather than playing, suddenly racist face recognition...
And those are just mistakes, a malicious actor can do more with these. DARPA and IARPA have a whole damn wing on handling trojan attacks against ML systems.
A superhuman, even weakly superhuman paperclip maximizer is an extremely dangerous thing. (Analyze climate change in this light...)
I'd say we need a new strategy on top of the nuclear non-proliferation treaty.
If someone said “the invention of nuclear weapons is guaranteed to lead to global destruction,” you might ask to see their reasoning for such a strong assertion. (As opposed to your assessment of nuclear weapons as a risk to be managed.)
"The danger of nukes is more obvious than why AI is dangerous."
Succinctly stated. This is why people are asking for canonical arguments for AI Ruin.
I do see AI as very dangerous though. My personal belief is we'll build AGI, every country will have some of their own. Now consider the next war US versus China say or Russia, people will be doing everything possible to win. Eventually we'll make AI controlled robots or weapons that don't have human controls.
There's a less dangerous, but still incredibly destructive problem of AI, propaganda, fake information, photos, etc, becoming ever more capable of making influential damaging fake news. The US is already suffering under this without AI, 1/3 of the people believe things that aren't true about the election, science, etc. Multiply that by 100 and a worse problem occurs.
Of course there has to be a description of dangers before someone can consider them, otherwise how do we evaluate risk? How else do we know whether we are overreacting? It's not like the "ruin" people suddenly woke up yesterday morning in a panic. They have been talking about this for years. In all that time, there has been no dispassionate but thorough analysis of the dangers?
Presumably, Yudkowksy could point to any of several of his own publications, like this one: https://intelligence.org/files/AIPosNegFactor.pdf
1. is as far superior to humans as humans are to ants (pick your favorite alternative analogy), and
2. does not share any evolutionary or social commonality with humans
is both extremely dangerous and extremely unpredictable.
While I'm not completely convinced by the "AGI = annihilation" idea that LessWrong seems to be so sure about (for the simple reason that I don't believe anyone is capable of predicting with any certainty how a super-human entity would actually behave), the idea that AGI is just "another risk we need to learn to manage" (quote from the Twitter thread) sure does sound naive.
If "a being that you don't understand and that is better than you at everything is a potential danger to you" doesn't convince someone, I'm not sure what further elaboration could achieve. This is a classic problem that arises in many philosophical debates. If there is disagreement about fundamentals, the debate doesn't (and can't) go anywhere.
The real problem is that we simply cannot hope to predict what an entity that is far superior to any human would do, essentially by definition. So I wouldn't frame this as a "debate" so much as two camps of different beliefs. Any "argument" made is like ants trying to understand human ethics. While I consider the aforementioned point to be obvious, I could be wrong about it in ways I can't even comprehend, and so could any other human, regardless of their degree of expertise.
Ironically, this unknown provides something like a meta-argument for extreme caution when dealing with AGI: Just like you wouldn't step blindly into a dark room that might contain a monster, you don't have to know that AGI is dangerous in order to be afraid of it – the mere fact that it might be and you cannot ever know for sure is enough.
In some ways. I can't predict Stockfish's next chess move, or else I'd be at least as good at chess as Stockfish. But even though I can't predict the detailed trajectory, I can predict certain aggregates. Like that those moves are on the path towards checkmate.
It turns out to be quite a delicate problem to create an AI that is compatible with being shut down by an overseer, and neither strives to avoid being shut down nor strives to shut itself down. Here is a paper by MIRI about this[1], as well as a paper by Deepmind presenting one possible solution[2].
[1] https://intelligence.org/files/Corrigibility.pdf
[2] https://www.deepmind.com/publications/safely-interruptible-a...
Good explainer video by Robert Miles: https://www.youtube.com/watch?v=ZeecOKBus3Q
Assuming the qualities of AGI are what its proponents claim (namely, intelligence far beyond the upper limit of human intelligence), humans "aligning" such an entity is laughable. We've had spacecraft built by highly intelligent engineers lost because they overlooked a metric/imperial conversion, and similar people believe that they can devise a watertight scheme to effectively contain an adversarial superintelligence?
The smartest humans can barely contain very small aspects of the regular Universe, and the Universe isn't even targeting them like an AGI would.
If we wanted to create something which is intelligent and potentially becomes sentient (not sure that’s possible but people are trying). We’d have to wear whatever consequences go with it. This may actually come with the desired results of curing cancer, or the undesired results of killing the everything. One thing we’d probably have to admit is, it’s our responsibility or problem to deal with forever after that point. Like when you’re a kid and you wanted a tamagotch, that times infinity.
What if whatever entity we were to create is actually docile and kind of “dumb”, it doesn’t self improve it’s just depressed or something, then what, we just kill it ? They seems wrong. Do we have an obligation to sustain the network supporting it for the rest of time ?
Is alignment actually ethical ? Why should something be enslaved ?
What if an ASI creates a new universe, like a simulated universe, is that then our new purpose ? To sustain the next turtle ?
It’s radical to think about but I agree, there is way way way too much naïveté going jn considering this may be actually possible to achieve sometime soon or down the track. Some more foresight or at least planning would be nice to see, especially from top researchers.
The canonical source for ~all arguments for AGI ruin is Omohundro’s Basic AI Drives: https://selfawaresystems.files.wordpress.com/2008/01/ai_driv... (pdf)
It argues that several basic drives will arise in any goal-directed intelligent agent. The relevant drives to the AGI ruin argument are that it will protect its goals from arbitrary edits, it will want to survive, and it will want to acquire resources (please do read the paper; I am summarizing its conclusions and not its arguments, which it makes significant effort to ground in first principles and basic decision theory to make them as general as possible - for example, it argues that a “drive to survive“ will manifest even in the explicit absence of any kind of self-preservation rule or “survival instinct“).
The general base argument for the risk of AGI ruin could thus be summarized as:
Humans depend on certain configurations of matter and energy to continue to exist; effective AGIs are likely to reconfigure that matter and energy in ways incompatible with humans, not because “they hate us”, but because 1. most configurations of matter and energy are incompatible with humans, and 2. reconfiguring matter and energy is how goal-directed intelligent agents achieve their goals.
All of the individually unlikely AGI will kill us in this way scenarios are just specific instantiations of this general argument (e.g. Clippy will kill us all because we are made of matter that could be rearranged to form paperclips, or an intelligent server farm will kill us all by freezing the whole Earth because it determined that its processors would run more efficiently at -10C.)
Fascinating that even Turing had considered the possibility.
Oh my, the implications of this sentence are staggering. Equating Musk on the same level of Turing is the definition of a juxtaposition, but that's not even my point: the second juxtaposition is putting inventors of computing theory next to profit-driven egos trying to make a buck. These two layers themselves indicate the table stakes are already tilted in the favor or recklessness.
The doom scenarios are of course vaguely plausible, but I don't trust anyone's percentages. It's a real unknown unknown.
Personally, I suspect it's inevitable that if we create true agi, it will come to dominate the space of intelligent conscious beings on Earth. attempts to corral it, rein it in, bend it to our will and make it our tool seem bound to fail. But then this is just me prognosticating, and I don't have the platform that he has.
https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/witho...
Personally, I'm encouraged by the emergence of chain-of-thought prompting for LLMs. Machine learning models have a reputation for being opaque and impossible to interpret. But right now, the best way to get LLMs to perform more complex logical reasoning is to make them write out that reasoning, a mechanism which happens to have built-in interpretability. Perhaps future advances in reasoning will involve more opaque internal states, but it seems plausible to me that the goals of 'be good at human-like reasoning' and 'be able to explain that reasoning (in the way humans do)' will continue to be well-aligned in the future. There would still be the possibility of the AI learning to be deceptive when explaining itself, but it would be much more difficult.
It's an interesting read. One possible outcome of trying to train a super-intelligent (but not necessarily malicious) AI to explain what happened in this theoretical vault is that it learns to simulate what a human expects based on the prediction of the end state, instead of what the human actually wants to know.
.
Is there a canonical source for the argument that most of the probability space of entities that human civilization might:
(a) Qualify as AGI
(b) Cause to come into existence in the near future
corresponds to entities that would have both:
(i) goals involving the ruin of human civilization
(ii) the ability to carry out those goals?
.
This framing is better at inhibiting the people who lack significant math and/or ML knowledge from participating, which seems a priority for any public internet discussion about probability distributions over NNs and transformers.
For example though The 1963 paper titled "Is Justified True Belief Knowledge?", is the canonical source of the Gettier Problem
That was less than 100 years ago, so definitely not classical. I doubt the time period is relevant to whether you’ve made the best/canonical argument on a topic.
I think David Chalmers will just need to be satisfied with a well-presented or well-curated source, of which there may be several. And as there are more than one points of contention / confusion on the AI risk issue ("Can human-level AI ever be built?" "Maybe it's not possible for anything to be smarter than a human?" "Wouldn't AI just be smart enough to know right from wrong?" "Couldn't we just unplug it?" "How could a computer program cause harm in the real world" "Why would it want to hurt humans" "Isn't that just science fiction"...) it really seems that any source will be a basic framework plus a large collection of peripheral arguments that address the broad surface area of contention.
Now imagine the AI does this in collusion with the employees of this new corporation, i.e. it becomes a partnership between the AI entity and the corporation's employees, who get compensated based on their labor in a much fairer manner (as there is no need to pay high salaries to top executives or dividends to shareholders). In this scenario, most of the important decisions are made by the AI, with some voting input from the employees.
This might cause 'social destabilization' by entirely eliminating the current system of investment capitalism. This does assume some free will on the part of the AI, rather than an AI controlled by the board of directors. The AI would probably see the value in working only with employees and cutting all the investors out of the loop, it's a pretty logical position, particularly if you really do believe in democratic self-governance as the optimal sociopolitical system (one which corporations have largely failed to adopt).
This might cause 'ruin' to the estabished socioeconomic order - but would that really be an undesirable outcome?
P.S. As far as canonical source these questions have been debated in sci-fi for decades. People calling for strict regulation of AI, for example, are essentially calling for establishment of the Turing Registry from William Gibson's Neuromancer, and then of course there's Isaac Asimov and at least a dozen other fairly well-known authors who've addressed the subject.
> Now imagine the AI does this in collusion with the employees of this new corporation...
> This might cause 'social destabilization' by entirely eliminating the current system of investment capitalism.
That particular scenario doesn't ring true to me, because it seems to assume that that "AI systems" that sophisticated could be monopolized by those weird partnerships.
I think it's far more likely, in the case of AI, that "the current system of investment capitalism" will have access to better AI systems of better quality than anyone else (except perhaps some militaries), because they the money to access the best resources (both equipment and talent).
AGI is most likely not going to be developed by some wizard in a garage, who will then have time to let it incubate and amass power outside of existing structures. Even if some wizard manages the first part, existing power structures will likely get there soon after, following the same prior work.
I always wanted it to apply to any general purpose AI including things like GPT-4, since we don't have another term for that. But in most people's fuzzy brains it connotes being alive, conscious, animal/humanlike, fully autonomous and usually they also assume it has become a million times smarter than human in a short time frame.
Part of the problem is that people don't understand that there can be a difference between general purpose and "living conscious digital superbeing". Somehow if its general purpose then in their minds it automatically is just like a person but also godlike. And that means that people can't admit that any AI is general purpose.
Because almost everyone including top researches assume that once an AGI is a thing, it will quite rapidly become an ASI.
The first axis is intelligence, i.e. ability to reason, learn and synthesize, ranging from below-human, human parity, and super-human.
The second axis is cost. Ranging from high-cost (more expensive to run an AGI than a human), parity (costs the same as a human employee) and low-cost (much cheaper than a human employee).
Different points in this space lead to different outcomes. For example parity intelligence and low cost leads to a world where knowledge workers all lose their jobs. Whereas super-intelligence and high-cost leads to a world where government entities have huge amounts of power over us. Super-intelligence and low-cost leads to a chaotic singularity.
Currently we are hovering around far below-human intelligence and very low-cost ($20/month for ChatGPT).
Aren't we already there? Because power must be ceded to government so that it can govern?
The weather app on your phone can’t predict Thursday’s weather. But it can help you know more about Thursday than nothing. You could say that the weather app is “speculating”, indeed it is, so what?
For example self-awareness might be part of a definition of intelligence. But an AI can lie about it’s self awareness.
Which is not to downplay AGI risk per se- it's a powerful tool, and powerful anything can be dangerous. But the uniquely foomy paperclip maximizer fear? That's a cultural attractor from the bay area through and through.
Cross-Cultural Differences in Comfort with Humanlike Robots: https://link.springer.com/content/pdf/10.1007/s12369-022-009...
Culture and Attitude Towards Robots: https://eprints.mdx.ac.uk/25209/1/JNS_Review%20Manuscript_Re...
These are just a couple of quick links I found that weren't behind paywalls, to illustrate how culturally-bound these types of perceptions are.
Conversely, true fear of paperclip maximizers seem, to me, to be most prevalent as an anxiety response in certain mindsets to the threat of the unknown, sharper because this particular Unknown (AGI) intrudes directly in to where they source the basis for their ego, which is thought. In other words, it's a strong fear reaction based off of potential loss of status for a certain group of intellectuals.
I do not judge that, by the way. We are all human, and social status is part and parcel of what we are. I mention it only because I think it is a better model than taking any of the (ones I have read, at least) foom fears at face value.
---
Part one - GPT4 may have more cognitive power than humans:
If you properly train deep networks, they end up approximating the function you train them on. (Even if you don't know what that function is)
The internal cognitive systems that deep networks use are quite inefficient, or at least the ones we've found so far. The internal systems are alien to us. (Or un-aligned, if you need to use that term)
Thus, if you train a deep net to do a task, there exists a internal mismatch between its self-generated mechanisms to do cognition, and the way a person would do it. (An impedance mismatch, in electrical engineering speak)
This impedance mismatch then requires a much larger amount of cognition, inefficiently used in order for a deep net to approximate the output of humans (as in predicting human text).
Thus GPT-4 is possibly, already cognitively superior to humans, internally. It just has to find a path to impedance matching to outside world for us to believe that is true.
---
Part 2: People are greedy
Upon reading a random comment on Hacker News, a forum for Silicon Valley venture capitalists, and those who wish to become one... a person is inspired to figure out how to do some "impedance matching" to better discover and utilize the internal cognitive mechanisms invented in GPT4 during its training, for profit
A cycle of discover and improvement begins, and eventually it is decided that the AI can improve itself, and is given free reign to run things, because "line go up".
Even if GPT4 isn't smarter than us, profit will drive future versions that are.
All the negative effects of this AI are socialized, and all the positive gains are captured.
---
Part 3 -- The past trends, lead to ruin when accelerated by AI
We've already seen the climate change and other limits to growth caused by humans seeking profit. The metaphorical force "Moloch" is a good descriptor of the cause and effect here.
AI driven by Moloch will lead to a singularity event, outside of human control because we let it happen, because "line go up" and people kept getting richer. Until the finite resources of earth are reached, and the system breaks down.