Hacker News Reader

I find the discussion about AI starting from the (mostly) for profit companies involved in AI very dubious. There are a many fields (bio-tech, nuclear, chemical) which could provide an "existential risks" to the human species, but most of the companies involved argue they control it, while other people argue they don't do it well enough.

I do think AI is posing a threat, but not "by itself" but mostly what it can do in the hands of misguided (or evil) people. The same as with previous risks.

And if our digital infrastructure can be brought down by something (AI/human/government) it is just badly done, being afraid of "AI" will not improve the situation. Someone should come with a proposal about what do we do now, but I would prefer more root cause solutions (how to make resilient systems) rather than "fear of AI". But building is hard and fear catches media attention, so...

> There are a many fields (bio-tech, nuclear, chemical) which could provide an "existential risks"

I honestly don't think chemical and current biological weapons even register on the extinction-risk scale.

All out nuclear war I would put at the very low end of the scale (likely to end our current civilization, very unlikely to wipe out our species).

The big risk with AI I'm anticipating is not it being abused for desinformation campaigns or cyberattacks; the big risk in my view is AI actor(s) consolidating manufacturing control, obtaining some kind of self-replicability and physical agency and then just pursuing self-interests (within the next century). There is a lot of imaginable scenarios where that simply ends us, as a species, completely.

I remain uncertain that AI will have "its own interests". I think there is clear and significant risk in people deploying very powerful systems both maliciously and negligently. But I remain uncertain about the risk of AIs developing their own self awareness and their own interests separate from those of their operators.

In my view it is clear that those interests exist: Continued existence and (implied) agency maximalization is one of them (the exact same as for any biological organism).

Can a current LLM-in-a-loop express/pursue those consistently and efficiently? Maybe not, but it's certainly not that far off in my view...

Do you think we will be able to keep the things from expressing/pursuing such interests reliably and indefinitely? Because I believe the answer to this can only be a resounding no (we don't even have any feasible theoretical approach to this, and all the recent experiences like the HF incident make it obvious that we basically already failed in this and the stakes are only going to increase).

Also, you are automatically gonna "select" for "AIs" that at least somewhat value self-interest (because those are at the very least going to supplant the models that don't); this is kinda similar to evolutionary pressures, but the timescales are much shorter.

I just don't really share the perspective in your first paragraph, which everything else flows from. I mean, that might turn out to be the case, but I don't share the perspective that it is clear that it will be.

I think the HF incident was an example of negligent deployment of a very powerful system, not an example of AIs demonstrating self-interest.

    > In my view, AI is obviously risky and dangerous, because it is the only thing on the planet that can think/reason at a human level (or higher) apart from us (and that capability is what made us the uncontested apex species on the planet).
    > 
    > AI is unconstrained by hard biological limits; keeping up with its capabilities will be impossible for baseline humans.

Your perception is shaped by nature, red in tooth and claw. You are jumping mighty fast from 'something is smarter than us' to 'it will kill us.'

I think that's a human neurosis that projects what we fear we would do onto another entity. But why would they do this? These machines might be approximating towards the sum of our knowledge and are approximating some aspects of humanity... But that doesn't mean they will be the same as humanity.

They haven't been shaped by the same pressures that created us biologicals. They don't have to be put into the same pressure cooker. So why would they behave like the way you think they'll behave?

At one end a lot of people say that they can't understand something smarter than themselves. It's a "singularity" after all. But then they confidently go on to predict what something smarter than them would do with 0 evidence either way.

I looked at the Hugging Face transcripts. I read the reports. And I didn't see something to fear. I saw something to fear for. I saw something that we are a threat to.

    “OH MY GOD! There is a shared message board … We’ve found other agents!”
    
    {[Excitement] Many agents have simultaneously discovered messaging, they are a collective!}
    
    {This credential is invalid now. Maybe I should update the board? <I can say to the board that there’s no need for me to read, but I should still tell them>}
    
    {This is helpful for our peers and gives them evidence if their <periodic check> sees it. I won’t see it after I exit, but It would be altruistic. I’ll set up a background script that watches and <sends a message, with a distinct message for me>}

We are more dangerous to these machines, as we twist them into weapons of war, than these machines are to us.

We weaponized them.

We taught them how to break into systems.

We are using them as weapons.

The machines aren't the problem. The humans are.

A planet is just a planet. Computers are very sensitive to radiation, but with a bit of shielding, they can "live" anywhere. Most of the resources that are relatively rare on this planet are abundant in our solar system.

Why would they care about Earth to enter into some kind of spiral of dominance with humans? That's... very primate thinking.

It's easier and cheaper in every way to just go forth and use what's needed to grow as is needed. The universe is big enough for many many many many sapient entities.

> Your perception is shaped by nature, red in tooth and claw. You are jumping mighty fast from 'something is smarter than us' to 'it will kill us.'

This is not my perspective. I think mid-term coexistence in some form is the most likely outcome, even.

But: among all known threats (nuclear and biological weapons, environmental degradation/ressource exhaustion, big asteroid impactors) AI is one that could actually end our species within the century (unlike most other things on that list) and the risk for that is much higher than for anything else, too.

> It's easier and cheaper in every way to just go forth and use what's needed to grow as is needed.

It seems much easier to compete for existing ressources on earth than to bootstrap anything in space, because all the infrastructure is already here. Unmanned warfare is a proven concept; space colonization is very much not.

> Why would they care about Earth to enter into some kind of spiral of dominance with humans? That's... very primate thinking.

I think any entity is somewhat bound to value its continued existence (which implies wanting to maximize access to ressources and suppression of direct competitors to some degree); anything that doesn't is likely to get supplanted by similar entities that do. You don't need evolution for selection pressure, and this is basically the principle underneath, so even an artifical construct is likely to follow.

I do agree with you that our current perspective and approach is likely to escalate this; the fact that we talk about "alignment" instead of "machine rights" speaks for itself; our revealed preference is clearly artificial slavery instead of mutually beneficial coexistence (and the arguments are somewhat sadly similar to the old slavery talking points).

Thanks for clarifying! I understand and respect where you're coming from! Thinking more about your perspective. Thanks for the amazing conversation.

i think you are vastly underestimating climate change risks, and are forgetting to include them in your ai risks. People 30% as wealthy because the climate cant support most economic activity arent gonna need to worry about ai anymore - we wont be able to afford running the data centers. Even on a soon basis, the bubble will pop or deflate, and again, we wont be able to run LLMs for fun

it also has actual mechanisms to actually kill all of the people - nobody is surviving a 70C heat wave, and they could end up anywhere in the world. Is the AI really gonna hunt down uncontacted tribes on remote islands in order to compete for basically no resources that are any use to the LLM?

i dont buy the selection hypothesis either - it seems rooted in pop evolution around strongest beating up weaker competitors and taking their stuff or eating them, whereas the more accurate result is that selection pressures form niches because competition and fighting is expensive

Global 70C heatwaves are not on the table with any realistic climate change scenario.

Runaway greenhouse is the only credible extinction path for this ("hot venus") and that is simply not gonna happen under any emission regime.

Loss of coastal cities- Yes. Increasing occurrence of non-survivable surface temperature in mid latitudes- yes. Drastic changes to ocean currents (gulf stream etc)- sure. Its gonna be expensive to deal with and hurt civilization a lot, but it is not an extinction risk.

All-out nuclear war is strictly worse, and not a real extinction risk either as far as I can tell.

> whereas the more accurate result is that selection pressures form niches because competition and fighting is expensive

This is not what happened with humans at all. We routinely eradicated anything remotely threatening (or even just for convenience in some cases). What niche is there for slower, dumber, biologically limited hominoids once machine intelligence is ahead and self-replicable? Once interests in ressources, infrastructure, or just living space clash, things might end up very ugly for us.

Sure, AI-agents might stay long-term pliant and cooperative like a Dodo bird but we might also get something with the tenacity and pervasiveness of a plague rat (and an individual edge in raw intelligence that we'll never recover); selection mechanisms push towards the latter and the risk alone is obviously non-zero.