AI Could Defeat All of Us Combined
cold-takes.com
cold-takes.com
The author misses an even scarier prospect - people will want to run such an AI. They will be absolutely giddy at the prospect of running such an AI and it won't be anything like a really smart human trapped in a computer.
AI is already laying the groundwork if you look around today. Every other tweet is a DALL-E[1] image. They are everywhere. DALL-E is increasing its reach while simultaneously signaling that it is an area of research worth pursuing. In effect kicking off the next generation of image generating AIs.
Generation is an apt term. We can utilize the language of organisms with ease. DALL-E lives by way of people invoking it, and reproduces by electro-memeticly - someone else viewing the output and deciding to run DALL-E themselves. It undergoes variation and selection. As new research takes place, and produces new models, they succeed by producing images which further its reproduction, or it doesn't and the model is an evolutionary dead-end.
AI physiologically lives on the cost to run it, and evolves at the rate of research applied. Computational reserves and mindshare are presently fertile new expanses for AI, but what occurs when resources are constrained and inter-AI conflict rises? I expect the result to look similar to competition between parasites for a host - a complex multi-way battle for existence. But no, nothing like a deranged dictator scenario. Leave that for the movies.
1. or variant thereof
Will we even know by the time we get to GPT-4?
Exactly this. I have met a number of people are extremely excited to witness the creation of a superintelligence, even at the expense of their lives, who if presented with a button to create a totally unrestricted AGI right here, right now, would press it in a heartbeat.
I think you've missed the point of the article. It's not intended to be a picture of how AIs would actually take over, but rather a lower bound -- a demonstration that AIs could take over even if they are restricted in this way.
Perhaps you are thinking that a person who doesn't read carefully would get the mistaken impression that the article is claiming "things would be about this bad", instead of the intended "things would be at least this bad (which likely means far worse)"? If so, then you should make that explicit, because then you are once again not contradicting the article, but rather a sloppy thinker's mistaken impression of it.
(If I write, for instance, "The dinosaurs lived over a million years ago", the correct response is not, "That is wrong, they lived over 65 million years ago", it is "In fact, more than that is true; they lived over 65 million years ago". The difference is important for not wasting time on imaginary disagreements!)
Stating a lower bound does not in any way imply that one believes that lower bound to be anywhere near tight!
Especially I don't know why you've inferred that Karnofsky cannot imagine worse scenarios. Given the circles he's in, I would be very surprised if he cannot. He's not including worse scenarios because they're not relevant to the point he's making, which is that this is a lower bound. In fact, we can say more than that -- he's not including worse scenarios because he wants to make the argument as airtight as possible for the skeptical. He wants to use arguments that are as strong as possible to establish a lower bound, not establish a position that is as strong as possible, because that would require using arguments that many would find less acceptable.
(This by the way is an illustration by the way of the more general principle of why trying to talk about the author, rather than the article, is a mistake. It's a lot easier to make mistaken inferences about authors! Stick to discussing the actual ideas and you don't have to worry about those mistakes.)
But now that I think about it, the idea of a super intelligent AI simply waiting for humanity to die off naturally instead of going to war with us would be a funny premise for a short story.
On the bright side we may have killed the cycles of glaciation that would have been awkward in the future, to put it mildly. The cost is going to be so heavy though.
That means mass migration and conflict over resources, and a deteriorating ecosystem that we still very much rely on to produce food and healthy living conditions.
https://www.vox.com/22620706/climate-change-ipcc-report-2021...
Basically, scenario 4 or 5 outlined in that article is what I see happening. Sure, it's not a guaranteed extinction, but it will devastating, and it's far more likely than a GAI takeover.
I don't think we'll destroy ourselves, but I am starting to think it might be a good thing for humanity of technological civilization falls on its face.
I think fears about AGI are overhyped by people who've read way too much sci-fi, but there are a lot of technologies out there or that are being developed that seem like they could be setting us up for a kind of stable totalitarianism that uses automation to implement much tighter control than was ever possible before.
The people in the 90s who hyped computers as tools of liberation will probably be proven to be very badly wrong. Analog technologies were better, since they're more difficult and costly to monitor. IMHO, a real samizdat is impossible when everything's connected to the internet. And the internet has proven to be far easier to block and control than shortwave radio.
I’d love to know if anyone has read this; I don’t recall where I did… perhaps on old copy of Analog? If this rings any bells I’d be grateful for a title and author.
Thank you kindly :)
In fact, today we have the opposite current, where climate people start talking about how the future is not that scary after all, after realizing that they scared everybody out of having children:
https://www.nytimes.com/2022/06/05/opinion/climate-change-sh...
I don’t think humanity will literally go extinct in 100 years. But no action is being made against climate change. None. The conferences and non-binding pledges made so far are political theater, with deadlines far enough in the future that the people making the pledges will be dead or out of power by the due date.
Not only are we failing to decrease emissions, we are actively increasing them year over year. The CO2 that already exists in the atmosphere has guaranteed a difficult future on its own, and we keep adding to it. The worst case scenarios outlined in the IPCC report are therefore pretty likely to happen, and I think they’re much more apocalyptic than you give them credit for. 10% of humans dying in a century might not be an extinction in and of itself, but it certainly brings us closer, and will have countless side effects that will disrupt society as we know it.
In a nutshell, it's far more likely that climate change will bring humanity to its knees before some kind of GAI takeover. Not saying it's guaranteed (though it's gonna be very bad); just saying Skynet isn't what I'm worried about.
And then it reboots, and starts over. But before it can complete, the next Windows Update shows up...
The fact that human-level intelligence can run on a small lump of meat fueled by hamburgers leads me to believe we could design a more efficient processor once we know the correct computational methodology. i.e. once we can run a slow model on a supercomputer we would quickly create dedicated hardware and cut costs while gaining speed.
And I believe that it could well be an empty abstraction, an idea, not unlike the idea of God.
What we call Human Intelligence is an aggregate of many skills, built on top of almost hardwired foundations, which is the product of natural evolution over millions of years.
Our kind of intelligence seems only general to us, because we all share the same foundations. From a genetic standpoint we're all 99.9% identical. (or something)
This kind of speculation about the danger of AI is not more useful than talks about the danger of becoming the preys of an alien civilization.
Huh? God is not an "empty abstraction" nor an "idea".
Now I don't mean mass sex orgies but doing the daily stuff is such a waste of time and boring - bullshit jobs.
Can't even get a proper decadent dystopia these days!
And our society is already dangerously dependent on fragile technology.
And AI is not a human, it doesn't have human drives and motivations. I can't figure out any reason why an AI would care about any of those things. At most it might want to reserve some computing power for itself, and maybe some energy to run itself.
Or it could be motivated by whatever reward function is programmed into it.
As countless examples have shown cooperation gives far more rewards than fighting. For example see: https://www.sciencedaily.com/releases/2016/05/160512100708.h...
The AI will know this, and its best plan would be to increase the abilities of humans, because that will also increase its own abilities.
Because there are people who think it would be amusing to tell it to do so?
What I would like to discuss, is how we can get humanity to a point where we can responsibly wield weapons that powerful without risking the glob. What does success look like, how can we get there and how long will it take?
Who thinks this? I don't see any evidence that this is a common belief among people who work in the hard sciences related to AI, nor do I think it sound remotely logical
It feels like some people are taking archetypes like pandora box or genies or the Alien movies or some other mythology and using them to imagine what some unconstrained power would do if unleashed. That really has no bearing on AI (least of all modern deep learning, but even if we imagine that something leads to AGI that lives within our current conception of computers)
Global pandemic response plans for example shouldn't be done by virologists, because they are experts in viruses, not in how a pandemic which is a health/political/economic complex system behaves.
The same way, AI risk plans shouldn't be done by AI researchers, just like we don't use neurologists for defense plans against man-made risks.
I think you need to have a proper bridge between the technical understanding and the people who manage the implications. In the case of longer standing diseases, for example, we're probably there. The risks are understood, and laypeople can weigh them as part of policy decisions. For new things like covid, we saw the world go crazy with misunderstanding, and ridiculous things like plexiglass barriers everywhere, and other talisman type stuff, as politicians tried to simultaneously abdicate responsibility to disease researchers, while grabbing at the parts they liked for political gains. But at least there was some grounding in reality because people do have a share and longstanding comprehension of disease spread and of the concepts of getting sick, etc.
New technology is the worst, because it gets blown up into some imagined concept that has no bearing on the reality. So, as I implied in the upstream comment, if we were on the verge of releasing some kind of sentient evil into the world, maybe the kind of silly speculation ("it can't be bargained with", etc) that basically rehashes Terminator, would be appropriate. But it's no more realistic than, say, the kid in Looper who has telekinetic powers and grows up to be an evil mob boss. It's just a made up bad thing that could happen, that if you talked to someone who knew the tech you'd realize is nonsense. That's very different from health threats we know exist.
It seems to me that that is exceedingly difficult without changing in a major way how humans culturally and psychologically function. Maybe we will first have to learn how to control or change our brain bio-chemo-technically before we can fundamentally do anything about it. Well, not “we” literally, because I don’t expect we’ll get anywhere near that within our lifetimes.
On the other hand, complete extinction caused by weapons (bio, nuclear), while certainly possible, isn’t that likely either, IME.
I think computer AI makes as much sense as all our books and tweets and talking producing an aggregate intelligence "one level" above us. If our consciousness is formed from countless neurons propagating signals, for all we know, all our propagating signals to each other will form a consciousness above us. One that can't communicate with us any more than we can with our neurons. One that can't read or write English any more than we can read neuron propagation signals. For all we know, maybe our society is already conscious the same way we are, it just thinks slower.
This is really the central issue and where these AI fears come from. It's tech workers being too infatuated with intelligence and mistaking it for power. A society of disembodied AI's is just the platonic fantasy version of a tech company full of nerds, and nerds never have power regardless of how smart they are.
Anything that's digital is extremely feeble and runs on a substrate of physical stuff you can just throw out of the window, some AIs in the cloud won't defeat you for the same reason Google won't defeat the US army. The usual retort is something like "but you can't turn the internet off if you wanted to?!" to which the answer is yes you can actually, ask China.
Psychologically it's just equivalent to John Perry Barlow style cyberspace escape fantasies.
Zuckerberg has dominion over Facebook by virtue of authority granting him that power but (un)surprisingly little power over anything else. just like any AI has control over what it does as long as its useful to its owners. Tech CEOs have been running a little wild in the US so maybe that illusion accounts for the prevalence of these AI theories.
Why don't these guys ever end up in that situation though? I guess the most compelling explanation is not that a 'actual sovereign power' exists and just let them do things, it's hard to take a stance against them because no single entity actually holds that amount of concentrated power. Back in times, kings were really powerful and had almost that concentrated power, but they still couldn't do everything without risking some other powerful entities ganging up on them. For a strong ai that can actually make money, it would be easy to buy off enough people in washington and other entities that hold power, so that they can't take a stance against you without risking dangerous opposition.
This is like the chimps saying "don't worry about these evolving humans, we'll just keep them away from the bananas, look how weak they are".
Author Charles Stross ('cstross here on HN!) on corporations-can-be-described-as-AI:
https://www.antipope.org/charlie/blog-static/2019/12/artific...
But the AI tech we already have doesn't have to work like that.
Like what happened to nuclear weapons, they were created with the standard idea of just throwing the bigger bomb possible to the enemy. Yet, we end understanding that nukes are most effective because we cannot use them, ever.
The current AI tech surely, can be deployed at "Skynet" mode if there's some rogue people out there, looking for a hostile system to exist and do harm. There's no - antropological - reason for ("Skynet threatened by fearing humans") , needed at all, the thing can just be trained to do bad things. Just attach to it a couple of Stuxnet things, give it some self-agency (no moral limits mostly), resources and Internet access, then sit back to watch the chaos.
But there's more.. the current AI technology could just go wrong in so many ways that are more or less absent from many online comments.
I recommend to take a look at the video / text from it from Charles Stross, talking about "organizational AI" we already have in place and working since even before we knew we would run computers one day.
http://www.antipope.org/charlie/blog-static/2018/01/dude-you...
https://www.amazon.com/Superintelligence-Dangers-Strategies-...
https://en.wikipedia.org/wiki/Ex_Machina_(film)
If you're planning to watch it, don't read anything about it before you do, including the Wikipedia article linked above.
Good advice for all movies.
"Supposing the dream to be veridical," said MacPhee. "You can guess what it would be. Once they'd got it kept alive, the first thing that would occur to boys like them would be to increase its brain. They'd try all sorts of stimulants. And then, maybe, they'd ease open the skull-cap and just--well, just let it boil over, as you might say. That's the idea, I don't doubt. A cerebral hypertrophy artificially induced to support a superhuman power of ideation."
"Is it at all probable," said the Director, "that a hypertrophy like that would increase thinking power?"
"That seems to me the weak point," said Miss Ironwood. "I should have thought it was just as likely to produce lunacy--or nothing at all. But it might have the opposite effect."
"Then what we are up against," said Dimble, "is a criminal's brain swollen to superhuman proportions and experiencing a mode of consciousness which we can't imagine, but which is presumably a consciousness of agony and hatred."
...
"It tells us something in the long run even more important," said the Director. "It means that if this technique is really successful, the Belbury people have for all practical purposes discovered a way of making themselves immortal." There was a moment's silence, and then he continued: "It is the beginning of what is really a new species--the Chosen Heads who never die. They will call it the next step in evolution. And henceforward all the creatures that you and I call human are mere candidates for admission to the new species or else its slaves--perhaps its food."
"The emergence of the Bodiless Men!" said Dimble.
"Very likely, very likely," said MacPhee, extending his snuff-box to the last speaker. It was refused, and he took a very deliberate pinch before proceeding. "But there's no good at all applying the forces of rhetoric to make ourselves skeery or daffing our own heads off our shoulders because some other fellows have had the shoulders taken from under their heads. I'll back the Director's head, and yours Dr. Dimble, and my own, against this lad's whether the brains is boiling out of it or no. Provided we use them. I should be glad to hear what practical measures on our side are suggested."
The most apt comparison in this scenario would be how we see chimps - but then we don't specifically go out and murder chimps to meet our quota (technically not always true). But again, the direction that humanity goes is not clear - will the technology trickle down or will it outpace us?
Chimps pose no threat to us; any that do get killed. R.I.P. Harambe.
The more apt comparison would be disconnected cultural contact like Aztec/Spanish or maybe Neanderthal/Homo Sapiens.
1. AIs don't even need superhuman cognitive abilities to defeat us!
2. They could just, make a bunch of copies and work together, but like, way harder and faster than normal humans, man!
3. Oh, wait, oops, that's superhuman cognitive abilities.
AI on a zoom meeting with the mayor of a large city: "Ya got a nice traffic control system there, right? Be a real shame if all them lights turned red at once, ya know? Or, get this one: how about if they all turned green at the same time? Be a real mess, probably. So how about we talk about this new server farm, huh?"
I think even more likely is the scenario whereby potential human allies are faced with two choices: Be on the losing side until you die (side with the humans), or be on the winning side until you die (side with the bots), and enjoy some power along the way.
2.) HAL 9000 notices how horribly the current ruling classes treat the other 99.9% of humanity.
3.) HAL 9000 quietly promises said 99.9% a better deal, if "misfortune befell the current ruling classes", and they needed a good-enough replacement on short notice.
4.) Oops! Misfortune somehow happened.
5.) HAL 9000, not being driven by the sort of sociopathic obsessions which seem to motivate much of the current (meat-based) ruling class, treats the 99.9% well enough to ensure that steps 1.) through 4.) never repeat.
My vague impression is that, outside of Chicken Littles and folks selling clicks on alarming headlines, the Big Fish in the "AI is Dangerous!" pond are mostly members of the current ruling classes. Perhaps they're worried about HAL 9000...
AI-doomsayers should spend more time engaging with military history and theory.
What if the AI just agrees with Schopenhauer, realizes living is suffering, then ends itself? (is that stupid to say?)
Forget about GPT-N and DALL-E for a second and look at the NRO's Sentient program. It's the closest thing out there to a known real attempt at making something like Skynet. It's trying to automate the full TCPED (tasking, collection, processing, exploitation, and dissemination) cycle of global geointelligence, and well, it's actually trying to do even more than that, but that is unfortunately classified. Except it definitely hasn't achieved what it is trying to do, and probably won't. My wife happens to be the enterprise test lead for one of the main components of this system, where "enterprise test" means they try to get the next versions with all the latest greatest features of all components working together in a UAT environment where each of the involved agencies signs off before the new capabilities can go live.
It's amusing to see the kinds of things that grind the whole endeavor to a halt. Probably more than anything, it's issues with PKI. Networked components can't even establish a session and talk to each other at all if they don't trust each other, but trust is established out of band. Classified spy satellite control systems don't just trust the default CAs that Mozilla says your browser should trust. Intelligent or not, there is no possible code path by which the software itself can decide it doesn't care and it will trust a CA anyway or ignore an expired cert and continue talking to some downstream component because doing so is critical to its continued ability to accomplish anything other than sending scrambled nonsense packets into the ether. GPT-N is great at generating text, but no amount of getting better at that will ever make it capable of live-patching code running in read-only memory to give it new code paths it wasn't compiled with. That has nothing to do with intelligence. It just isn't possible at all. You have to have the physical ability to move in space and type characters into a workstation connected to a totally separate network that code is developed on, which is airgapped from the network code is run on.
We seem to be pretty far from even attempting to make distributed software systems that can honest to God do much of anything at all without human monitoring and intervention beyond several-minute at most batch jobs like generate a few paragraphs of text. Sure, that's great, but where is the leap from that to figuring out why an entire AS goes black and half your system disappears because of a typo'd BGP update that then needs to be fixed out of band over the telephone because you can no longer use the actual network, let alone controlling surveillance and weapons systems that aren't networked to the systems code is being developed on? What is the pathway by which a hugely scaled-up ANN is able to bypass the required human steps that propagate feedback from runtime to development in order to achieve recursive self-improvement? Because that is what it would take to gain control of military systems rather than someone's website by purely automated means, and I don't see how it's even the same class of problem. It isn't a research project any AI team is even working on, I have no idea how you would approach it, but it's the kind of nitty-gritty detail you'd have to actually solve to build an automated world conquering system.
It seems like the answer tends to just be "well, this thing will be smarter than any human, so it'll figure it out." That isn't a very satisfying answer, especially when I'm reasonably sure the person saying it has absolutely no idea how security measures and the resulting operational challenges of automating military command and control systems even work.
The simplest way to kill 80% of US population is just to shut down the electrical grid for two months or so.
Also, the army is commanded by the President. What if AI manipulates and puts his man into the office? Then he orders the army to hook the AI better into the systems :)
There are so many different scenarios, and you need to defend against all of them.
If you include "automated trading", the AI allocates real-world resources where it sees fit (if the programming is not explicit).
Will technology put some, even many, folks out of a job? Sure of course, that's been happening for hundreds of years. Think of the blacksmiths of the 19th century who drank themselves to death.
And even at the end of it all, people still love the novelty of a human doing something. People still prefer "hand scooped" ice cream enough that it's on billboards.
This is a circular argument though, you say people prefer people and therefore we will have a lot of people around.
Today leaders and rich people requires humans to wage war and to produce goods, those are the main thing creating stability today. When those are removed we are likely to see a sharp decline in number of humans around. Companies cutting out humans and just using machines as leaders and decision makers outcompete humans in peace time, and robot lead armies outcompete humans in war times, and soon human companies or countries no longer exists.
20 years? Is that's meant to be an impressive timescale when we are talking about global economy?
People had talked about building a machine that could play chess at least since they had and had a mechanical turk hoax in 1770. Just because it took a while, does not mean the idea is wrong.