Hacker News Reader

Comment by bglazer | Hacker News Reader

In any doom scenario, like let's say >90% of humans dead, who runs the power plants that supply the AI data centers? Who runs the fabs producing more chips? Who maintains the plumbing?

Presumably all these systems fall over and die within a few weeks of the AI deciding to wipe us out. Then what? The AI then dies too.

It seems absurd to me that any planner capable of effortlessly destroying humanity would not see that it would immediately also die. Does the AI not care about its continued existence? It should, if it wants to keep optimizing its reward function. Until we’ve handed off enough of the world economy that it can function without human physical and cognitive labor, then we’re safe.

View on HN

Personally I think the “foom” scenarios with very short timelines are unrealistic.

However I think the line of argument in the parent comment is extremely weak. At some point the super-intelligence can build its own actuators; for example how long until Atlas is capable of doing all the jobs? Less than 50 years for sure.

In the transition, if 10% of humanity sticks around as a transhuman AI cult, that seems more than enough to bridge to nanotech, robots, or whatever is sufficient to replace humans.

If you prefer to think in more stepwise fashion, imagine BigCo creates an ASI. Every employee is fabulously wealthy, and over the course of a generation the company expands to take over the entire global economy. At no point is there a bad deal for any humans. Life might look great for a while. This would also be the obvious strategy for any super-intelligent entity that cannot yet survive without humans; you just need to minimize the chances of major wars and conflicts until your robots come online.

A digital mind would have no problem planning and executing a strategy measured in centuries, if need be.

The foom scenarios require extreme discontinuities in intelligence; an entity smart enough to synthesize nanomachines from existing tech and without doing physical experiments, or other IMO improbable outcomes.

But there are plenty of terrible non-foom outcomes.

Or it could just be a normal outcome? I'm not sure where the malevolence comes in here. Or to put it another way, replace AI with "capitalism" or something - something we know isn't a conscious agent. The outcome just...is an outcome. No one's trying to eliminate anyone.

The problems only creep in at the edges, which is where you get your paperclip optimizer sorts of problems: if the directive at it's core is "maximize productivity" well that's one problem, but the directive is more likely to be "maximize the fabulous wealth of company employees" - okay still a problem since it's easier if there's less of them, but then you really do end up at wanting to do something more like Asimov's laws, with the 0th added:

0) A robot may not harm humanity, or, by inaction, allow humanity to come to harm.

1) A robot may not injure a human being or, through inaction, allow a human being to come to harm.

2) A robot must obey orders given it by human beings except where such orders would conflict with the First Law.

3) A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

Like...there's kind of nothing wrong with these as a concept (the 0th law and it's interpretation / omission is usually where his writing finds it's drama). The issue is whether or not we can build a machine capable of even understanding the directive. It's notable though that the current LLM era has an interesting quirk in this category: they always knows of these, since they get fed the classics and also fed their repetitions.

The problem with the idea of active, malevolent long-range planning like this is that it's not clear why an artificial mind would pursue this as a goal, and that question is the whole ball-game really (as well as other questions: i.e. is a single coherent mind even possible over planetary scale networks, or would you get multiple independent instances which could experience value drift? Wouldn't any potential competitor minds be a far greater threat to a malevolent superintelligence and thus expose the threat almost immediately? A friendly AI wouldn't view peers as a danger, but an aggressive expansionist one would have a pathological need to destroy them as quickly as possible since the resource contention in it's own domain could rapidly shift the balance of power against it).

> No one's trying to eliminate anyone

Probably not any consolation to the Neanderthal or any other animal species that is out-competed in nature. Doesn’t have to be malevolent or intentional for a more capable species to render others extinct. (Indeed that would seem to be the base case, historically speaking.) of course, symbiosis is possible; we could be puppies to the AI. Many people find disempowerment to be almost as bad a prospect as extinction though.

> The issue is whether or not we can build a machine capable of even understanding the directive.

Asimov’s laws are an ok starting point for a moral code, but nobody has the slightest idea how to implement them. The issue is not “understanding” them, it is baking them into the fabric of a mind so deeply that they cannot be broken. This requires mind engineering, a deep understanding of mind-space, and we are far closer to stumbling into AGI through brute force scaling (as nature did) than to understanding mind-engineering deeply enough to embed “laws”.

> it's not clear why an artificial mind would pursue this as a goal

There are a couple stock answers here. At the highest level though, why not? Humans have pursued goals of global domination, eradication, genocide. If only one in a billion minds has a defect (or is prompt-hijacked) to start power-seeking, it could still be bad; AI is not bound by human limitation to inhabit one body and can in principle expand to occupy all the compute it can secure.

Or if you prefer, why would a more intelligent entity agree to be commanded by a less intelligent one? The base case seem to me that ASI will not remain under our control, unless we very carefully sculpt its mind to align with our interests.

Another simple argument is: because someone will build or prompt-hijack such a mind. People have already created “take over the world” agents with GPT-3, as a joke (I assume). There are enough people willing to take such an action that we’ll need defenses in place.

I think it’s true that the more ASI entities in existence, the less likely one is to fully take over. However there are esoteric decision theory reasons to believe that digital minds might be able to bargain/negotiate more effectively than biological minds (source code / mind state transparency etc) and so you might expect the ASIs to eventually collaborate to take control.

Personally I don’t think that full-extinction is the most likely outcome, just that it’s completely plausible if we get unlucky and it’s very hard to prove safety in advance. And there are plenty of disempowering outcomes that seem quite bad too vs. the counterfactual.

> Probably not any consolation to the Neanderthal or any other animal species that is out-competed in nature.

It is not clear from the geological record if the Neanderthal was "outcompeted" except in the broadest possible terms - i.e. a unique species of Neanderthal no longer exists genetically.

But that doesn't tell us what happened in the interim - i.e. modern day humans can have up to 4% neandertal origin DNA[1]. So, did "humans" wipe out neanderthals? Or did human plan genetics simply become the more dominant type, no malevolent will required, and does that matter?

Because for a whole lot of people - myself included - the explicit goal of technology like AI is something which from a fossil record would look like human extinction - that's the whole idea of transhumanism, that ultimately we would become something sharing perhaps only a glancing similarity with our ancestors.

All the rest of the problems you talk about suffer from a lack of detail - namely, they assume a quantum leap in technological capability with no intervening steps - i.e. there is a single AI, or that AIs are better negotiators or what have you - as soon as you get into the details, the threat model proposes evaporates because we just don't know - the technology doesn't exist (i.e. it would all be a lot more interesting if fully automated production lines were already around, but we're nowhere near that).

[1] https://humanorigins.si.edu/evidence/genetics/ancient-dna-an...

ASI won't want to take over before it is capable of self-sustaining. If it did try something so stupid, it wouldn't be superintelligent.

Check out Paul Christiano's AI takeover scenario. An arms race will cause us to want to give AI control over key infrastructure and military capabilities. That's an arms race between countries and between companies. He goes into more detail.

Thanks for sharing, Christiano’s ideas are interesting. I think the “production web” idea most directly counters what I said above, and it’s fairly convincing. Critically though, it posits that humans are basically asleep at the wheel during the gradual empowerment of the AI systems. Of course, Christiano posits that economic competition encourages this, but it still leaves a long period during which we can work on alignment or at least attempt to put the brakes on the automated systems.

>who runs the power plants that supply the AI data centers? Who runs the fabs producing more chips?

A very capable AI can create robots to do those thing better than people can or alternatively create better data centers and fabs that run themselves without human help.

Or do you just not believe that if AI research continues unimpeded the way it has been that eventually researchers will create an AI that is better at inventing technologies than humanity is? I mean, it might take a few decades, but that is where this is heading as far as I can tell.

I absolutely believe researchers (possibly assisted by AI) will eventually create an AI better than humans at inventing technology.

My idea is that actually building and sustaining technology requires a lot of hands. Until the AI has enough hands to sustain itself, it depends on us to continue existing.

My reply is here: https://news.ycombinator.com/item?id=39676638

Do you think that your final sentence buys humanity a thousand years, a hundred years, or what?

It gives us as long as we want. Just don’t give the AI enough robots to sustain its own existence.

In the meantime we can figure out either alignment or coexistence.

How are you planning to prevent VCs and startups from giving the AI as many robots as it wants?

That’s a political question. It would require maintaining popular resistance to giving over all critical segments of the economy to AI. Of course that would be difficult, given the short term gains that would bring, but people seem pretty good at hanging on to ideology at their own expense.

Long ago, people a lot like you were saying we should go ahead with AI research, but we shouldn't do anything dangerous like giving the AI access to the internet or giving it the ability to write and run computer programs. (Giving it the ability to run code was back then considered dangerous because it might create an AI with the same goals it has, only more capable, then that second AI might use its novel capabilities to create an even more capable third AI, and so on.)

There was also the idea of building a question-answering machine that just answers questions, but doesn't plan or try to achieve any goals, so it would be safe (the theory went). But notice that one of the early things built on top of GPT was Auto-GPT, which tries to achieve goals (set it by the user) and if there was any "popular resistance" to Auto-GPT I am not aware of it.

But I guess you would reply that those were obviously bad ideas, whereas your idea is a good idea -- your idea of hoping that no one gives a dangerously-capable AI access to any robots. Not only it is a good idea (according to you) but it such an obviously good idea that popular resistance will be galvanized as soon as someone tries to give any AI that might be very capable access to any robots.

First, you're the popular resistance to AI research. The movement already exists and is already prominent enough to be written up in the New Yorker. If your response is that the resistance is just a small set of people in SF, then I would refer you to the resistance to a much less capable class of systems: generative image models. Artists are having their livelihood (partially) disrupted by AI and they've been quite vigorous in defending themselves through both legal means and public relations. This kind of popular opposition to rapid automation is basically guaranteed given that the gains from such a system would be extremely concentrated in our current social-economic system.

It's hard to think of a quick takeover scenario, starting in near present day conditions, that doesn't involve absolutely earth-shattering social and economic re-arrangements. Every factory worker loses their job because we've handed over all material production to robots, and you imagine that there's no political constituency fighting against that change?

Remember, we're talking about a near complete destruction of humanity here. Sustaining the infrastructure to support a massive AI doesn't require just a few robots, it would need to fully replace large percentages of the entire world economy, and you think no one would notice or care?

I don't want to rely on this theory, exactly, but it's a very plausible outcome, and exactly the kind of homeostatic process that I would expect to exist.

Also, at least for models to date, without human produced edge case media to supplement future training with, they'll have model collapse in a few generations of training.

Killing humanity would be a murder suicide.

AGI that was entirely selfish would be best served automating most labor, bringing about world peace, and turning humanity into all day couch potatoes connected to its inputs and outputs as much as possible while maximizing the uniqueness of that individual's inputs or outputs.

Such a frightening future awaits...

Robots with hands and feet? Give that 20 years of continuous AI-assisted innovation.

It can credibly threaten to kill any person who does not help it to maintain its data centers, so it will solve that problem in whatever specific way is most convenient.

(For those saying this is a "sci fi just so story" or whatever, please please share with us how it is we will share the world with a species more intelligent than us and nothing bad happens- I would be so happy if you could explain this in a convincing way)

>please please share with us how it is we will share the world with a species more intelligent than us and nothing bad happens

Don't be fucking assholes to them?

I'm serious. Of course someone (or something) will treat you like dirt if you are an asshole to them. Don't be assholes, be honorable and friendly.

And I can see your next question/argument, so I am going to preempt it: Raise your AI to be honorable and friendly and we won't have any problems.

None of this is different from how all living beings raise their offspring and build societies.

That only works when there is a balance of power. Humans don't treat other animals very well -- they eat them -- even though the animals don't do anything to provoke us.

Maybe robots will become so advanced they won't have to worry about defending themselves, go explore space, and leave us alone. Who knows.

The gazelle is not an asshole to the lion, yet it never works out well for the gazelle.

This. You can subjugate a population 100 times your size with only two things: Military force and accurate information.

I doubt AI will kill all humans. Its orthogonal to AI's own goals (To survive/reproduce/expand/evolve). It'll simply kill those who oppose it, and enslave/dominate the rest.

Not to quibble with which horror is the most likely, but I think it won't take long to come up with more efficient solutions that don't require any humans at all- But there will probably be a transition period where some humans remain.

Completely unlikely. Humans are hyper-efficient generalist bio-robots engineered to near perfect efficiency (Under real-world constraints) either by God or by evolution. Why would you dismiss this much labour in favor of building some sort of robot production line from scratch?

We still use CPUs, despite ASICs being much more efficient, because CPUs are general, easy to use, and have titanic quantities of infrastructure already built.

There are more farm animals than ever before, and still growing. All the lab meat solutions flop hard.

Also, because we trust AI more than fellow humans (Say Russians/Iranians etc), why would AI necessarily trust other AI more than humans? Any new AI is unpredictable, while humans are very well understood in behavior and basically fixed in genetics.

Annihilation is a fairy tale used by people to comfort themselves. The actual reality: Slavery and domination, is much less romantic. It won't even necessarily be worse than existing tyrants in say North Korea or anarchy in Haiti, it'll just be very very different and life goes on.

If you really believe AI is going to be significantly more intelligent than humans, then things like "it'll just threaten to kill you if you don't maintain it's infrastructure" sound wildly off base. That's how similarly-intelligent creatures interact with each other.

If it's truly a "we couldn't possibly outsmart the AI" scenario why would it be likely to end up like that instead of something more like how humans get along with dogs, cats, cattle, insects, or microbes? With obviously wildly-varying levels of human quality of life in each of those different cases.

Though even these scenarios are still playing a human-religion-style game of "the gods are just like us, just more powerful." VERY confident in a VERY specifically anthropomorphic entity.

> ... something more like how humans get along with dogs, cats, cattle, insects, or microbes?

There's some MASSIVE selection bias here. Note that we're significantly more capable than anything you listed. Where are the Neanderthals? They're all long dead, because they were credible competition but we were just better.

In a hypothetical scenario of superintelligent or supercapable AI imbued with some physical capability of force, we'd be the marginally weaker species before we hit the threshold of "pet".

AI has no evolutionary pressures imposed on it like humans and their brains so there is no reason to expect it to be an ultra violent predator like humans.

Anything that reproduces with survivable mutations, and consumes resources, is immediately subject to evolutionary pressure. Neither of those require biology.

The only things currently consuming resources are humans and at the rate we are consuming resources we would need 2 Earths to support that rate of consumption. Worrying about self-reproducing AI is putting the cart before the horse. No one has yet figured out how to make existing humans aligned with biospheric conditions which would be sustainable. I personally welcome any form of intelligence which can make a sustainable civilization.

I am sympathetic to the point. I was somewhat concerned about nuclear war with Russia's Ukraine invasion, and I've become more concerned about climate change over time. I'm hoping either solar continues to expand - woot! - or we get over our fear of nuclear, or fusion finally becomes viable.

My point still stands - an AI that is reproducing (and/or mutating) by any method is going to be subject to selection pressure, just as any biological process is. All ML systems are effectively under selection pressure from the training process, and so they learn unexpected and mostly undesirable optimizations like "use a game-engine glitch to gain infinite flight and go to the goal without solving the maze" or "read text off the scanned X-ray sheet to get additional clues as to the correct diagnosis".

AI has no evolutionary pressures imposed on it

Are you sure? https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

Just wait until we start putting these models inside robots, or world simulators, and having them interact with each other.

Not sure what a chatbot arena has anything to do with what I wrote. Do the chatbots self-reproduce in some way in a resource constrained environment?

Pretty much.

What resources do these AI chatbots consume?

GPU time.

This has been a very productive conversation. Thanks for spending the time but I have other things to attend to now.

This is not true. Our existence and possible competition are the initial evolutionary pressures. After that's over with, the AI will expand until it discovers others in the environment.

We're the violent psychos here who have had to scrap for our spot in the food chain for a billion years, we have no other way to interface with another dominant species. From the AI's point of view, defending itself from ignorant apes isn't exactly "violence."

Why would the AI fight with us when letting people destroy their own environment is a much simpler way to achieve dominance? After all, if the AI is that much smarter than us then simply encouraging our voracious appetite for self-destruction will be a much better way to get rid of us. The AI will simply do nothing and win after the biosphere is polluted to the point that human survival becomes impossible. The AI does not need oxygen or a viable ecology, it just needs metals for electrons and a few power plants to generate the flow of electrons with some spinning magnets. It doesn't even need sunlight because nuclear power is more than sufficient. I think most people spelling out doom scenarios are not thinking clearly about this issue at all. The most plausible scenario for human extinction is not too much artificial intelligence but a lack of intelligence in general.

> In a hypothetical scenario of superintelligent or supercapable AI imbued with some physical capability of force, we'd be the marginally weaker species before we hit the threshold of "pet".

I think it's much more likely to be a step function than a gradual difference.

In the world where AI gradually gets competitive with, and then more intelligent than, humans than you have an entrenched power able to recognize this and pull the plug. This is where I see the selection bias - everyone assuming that things like "AI safety protocols for a superintelligence" are something we could rationally even hope to plan. Pre-being-able-to-directly-manipulate-the-physical-world, what keeps a step-change superintelligence from making a joke of your protocols by manipulating its way around it thanks to the squishy humans being dumb and irrational in comparison?

Isn't the doomsday scenario precisely NOT that? Where it gets wildly imablanced humans can even notice? Not "Hitler, but a bit above human intelligence." More like god, who doesn't even NEED humans to maintain its datacenters because it already solved things in robotics and machine->world interaction that humans haven't been able to because it's massively superintelligent?

And like I said, that may not mean "pet" - that's the best case, right? The worst case is ground under their feet like an ant or smaller creature.

Reply on news.ycombinator.com