How I feel about AI | Hacker News Reader

How I feel about AI

beza1e1.tuxen.de

166 pointsqznc268 comments

The thing that makes me pretty terrified about AI is that I think the "utopia" scenario is not much better than the "doom" scenario.

We will very quickly get to a point where very few people will be able to contribute economically because they will be worse than AI (including robotics) at most domains. A world where people just have their whims catered to is not a utopia. We have tons of sayings and idioms about this, e.g. "no pain, no gain", "only the hard stuff is worth doing", etc.

The humans all running around on their Wall-E carts doesn't feel like utopia to me.

Edit: to head off the "rich people worried about striving for their job while people in poor countries are starving" arguments, I encourage you to visit a dying rust belt town in the US. These folks are by and large not starving (indeed, overabundance of calories is the bigger problem), they have adequate shelter, and plenty of outdoor space. Their material needs are met. But these places are despair central. People see little opportunity for advancement or the ability to change their circumstances, and the opioid epidemic is a direct consequence of a sense of hopelessness that there is no place for motivation to go or to effect change.

>A world where people just have their whims catered to is not a utopia.

I understand where you're coming from on that, but there are a ton of people living in slavery, or in unsafe working conditions, or with food insecurity, or dying of preventable disease.

There will always be something to strive for, even if it's made up. You think that the best football players in the world are doing something real? No, it's a made up game. We'll make up more games.

> We'll make up more games.

This reminded me of something. https://www.sbnation.com/a/17776-football

I agree with your points. My problem is that I don't see any way to "stop" AI development at a level just below human capability where you get most of the benefits but still have humans firmly in control.

Regarding entertainment jobs like sports star, musician, artist, actor, etc., the problem is those are professions where, for the most part, only the very best can make a living. Nobody is that interested in seeing the matches of the 324th best tennis player in the world, despite that person being in the top .01% of tennis players globally. Similarly, only a tiny percentage of actors "make it", everyone else does it for a few years and then leaves because they enjoy eating, or they accept poverty wages for a long time.

I get pretty scared when I hear the AI utopianists rattle off all the benefits that are coming (and when they play down the risks) because it feels clear to me that they really haven't wargamed out what society would look like with very powerful AI.

If we have money / goods from the AI making them then we can spend time getting the golf handicap down from 22 to 18 or singing mediocrely.

This was always the promise of “tomorrow”, yet the opposite is true.

AI doing human jobs has never happened before though.

That said I expect humans will work at stuff because the prefer to do that.

The game is made up, but the effort, and process to be one of the most skilled sports players is still real. And sports players would probably feel something missing life if AI could replace them too

Computers surpassed humans at chess nearly 30 years ago. Yet human chess is more popular than ever, and people still dedicate their life to improving at chess.

As a more silly comparison, forklifts didn't stop people from weightlifting.

Chess is inherently a leisure activity. The incentive for playing chess is just to be better at chess. But things like woodworking, typesetting, and welding, while all possible to do recreationally, are not leisure activities. They are effort used to produce something. The desire is always there to engage with the thing or idea, but if the resource cost is far too high or the knowledge required is more than one person can reasonably learn the effort will not be expended. A leisure activity will always persist so long as someone knows how to do it because it exists for itself. But many other activities exist in service to something else, and if that something else ceases to exist then the activity will as well. You lose entire domains of useful cross-disciplinary knowledge that way.

its annoying that robots are going to take intellectual jobs before physical ones. it feels way more likely for my project manager to become an agent before the lady that makes me a burrito at chipotle.

I think this is very true, but robots are definitely coming for the burrito makers, too.

Bill Gates talks about this in his recent missive, but as robotic technology improves (and, perhaps more importantly, as the cost comes down) there will be huge competitive pressures for companies to replace workers with robots wherever they can.

I don't get what the point of this is.

Robot burrito makers for who, if no one has jobs and can afford burritos?

Something very huge has to change before all of this happens

Well doesn't it make sense to replace the more expensive worker first?

i think there is a larger incentive to automate away physical labor than you're giving credit

Even their utopian views feel dystopian. Take solace in the fact, that it looks like it’ll take trillions to achieve, probably one of the most expensive technological buildouts of all time?

> A world where people just have their whims catered to is not a utopia. We have tons of sayings and idioms about this, e.g. "no pain, no gain", "only the hard stuff is worth doing", etc.

I somewhat get the point, but I think stuff like this is very often said by very privileged people. I don't think it's a serious threat to humanity as a whole. Perhaps to some humans, but there will always be ambitious people, even in a world where most things are provided by benevolent AIs for free. There is simply so much more harm caused to actual humans today by resource insufficiency than by them having too much, that I don't think we should worry about this scenario today.

Here's a great short story about how it could go either way

https://web.archive.org/web/20100211181147/https://marshallb...

Privileged rich people want scarcity to exist so that it gives meaning to their job.

wut.

I think this line of thinking is a huge disservice to people in third world countries and poverty. People everywhere want to find goals and motivations that move them forward.

Just look at the legions of people that have moved from Chinese rural areas to big cities. For many, many people, life in the cities is worse and kind of dystopian - much more overcrowding, "drone-like" jobs indoors, horrible pollution, and especially for men an often abysmal sex ratio if you want to find a mate. But people do it because it offers the dream of advancement and a better life - and of course some people do achieve that dream, even if the majority do not. That's versus life in rural areas which is guaranteed to be static, no chance for advancement and often crushing boredom.

People everywhere want to strive, work hard and get ahead. This is not just a privileged rich people issue.

  > People everywhere want to strive, work hard and get ahead.

i think you need to visit more countries and cultures because you would discover this is not at all true

Presumably, a truly superintelligent AI in a utopia scenario could figure out a way to enable people to continue working, even if the work doesn’t actually contribute anything, if it is really the case that work is essential to human wellbeing.

Now you might say “but those would be fake jobs.” If so, I have bad news about how many present-day jobs are fake jobs.

I agree with this. I thought the AI 2027 paper put it well:

> There are even bioengineered human-like creatures (to humans what corgis are to wolves) sitting in office-like environments all day viewing readouts of what’s going on and excitedly approving of everything, since that satisfies some of Agent-4’s drives.

I'll leave it up to you to decide if that's an enviable/desirable future.

I don’t even see why they have to be bioengineered. We have all-natural human beings doing this right now, except they are rubber-stamping human rather than artificial directives.

They don't, it's just at that point in the AI 2027 story the AI has already wiped out the humans due to resource contention (similar to how humans largely wiped out the wolves).

I think you are forgetting who owns these AI companies… their interests aren't your interests.

There's the mice utopia experiment too. I often hear that in space your bones dégradé, your biology needs weight as input.

Also I believe our socialness was crafted out of the need for survival. If removed it's all gonna run wild since nobody needs anybody, and it might end up as doom scrolling but for human relationship. Short bursts from time to time and that's it

Really glad you brought up the mice utopia experiments. I encourage people to read about them and think how they apply to humans.

In many ways they don't apply to humans, and I think a lot of the modern critiques are valid, and the overpopulation concerns in particular don't map well to the modern world. Still, I believe it helps to think deeply about what happens to societies when innate drives and motivations have "nowhere to go".

It's the inverse. We created a world for introverts only in the last ~50 years, where people can spend all their time in an office desk not actually talking to anyone. This was never the case in any of human evolution before.

It's only going back. Raw urban human connections will be the only thing left to exercise.

I disagree. I think the best-case outcomes for AI are actually quite positive. AI might actually fix many of the problems with the tech world that we have right now, like monoculture and consolidation.

What a delusional idiot

My very favorite piece of art is beautiful. It is also something that AI fundamentally cannot create.

It is three videos. This is one of them: https://www.lenkaclayton.com/work#/the-distance-i-can-be-fro...

In these three videos she sets up a tripod and tells her toddler to walk away from her. She is behind the camera. He toddles away, occasionally looking back at her or checking out something on the ground or falling on his butt. At some point he is too far away and she runs after him, entering the frame from behind the camera. The video stops and shows the distance he was able to be away before she had to chase after him.

The videos are sweet and funny but also engrossing and powerful. The tension builds as you watch it. Is this the moment where he is too far away? What if that were my son? Would I have run after him at this point? He sure is getting close to those trees.

But most importantly it expresses parenthood. The video only works because a real human experienced that very particular feeling of worry as her child was just a bit too uncomfortably far away. Because an AI can never have this feeling it can never create this art.

I do not believe that a labor-less world is around the corner. But if it was, we can just make things like this. And the world would be beautiful.

This is a bit of tangent and more of a rant. (and please don't suggest intervention of any kind)

> superintelligent AI will inevitably destroy humanity seems to have no flaw

For a section of population who were in a certain age-range when the pandemic hit and that derailed their certain kinds of plans/hopes/dreams in personal life and then this LLM/agentic world moved in, especially if they are in an industry directly and most affected this, this has been a surreal and slow Kafkaesque nightmare for a few years. The thing is with time and age passing hope keeps getting lost inch by inch and every lost bit of hope aids in killing the next bit. Then comes the sense of dread - the unchangeable reality that age passed passing means there are certain experiences and milestones that are essentially out of reach because there's no going back of this clock.

So I feel a strange kind of relief about this possibility of doom. I am just sharing what I feel, and I don't feel good about feeling this sense of relief that after all it might end for us all and not just an unlucky few. This feeling and the calmness about it saddens me.

As an older person, I agree with you and I'm sorry.

I think about the incredible luck I had to start my career in the .com boom, to have lots of hope about technology and to have the economically valuable skills that were desired.

I get angry when I see people my age or older complain about the apathy of young people. I don't like the apathy, but I find it trivially easy to see where it comes from. In many ways we broke the link between hard work and success and then get baffled and finger point when we think young folks are not as motivated as they should be.

I've been fortunate to not been impacted as severely as others, but yeah, the combo of the first half of my 30s (and everything that should've represented) going up in smoke, career continuity coming into question thanks to AI, and growing sense of overarching geopolitical, financial, and societal unease is really the worst kind of layer cake. I have to make a concerted effort to not get bogged down in it and that's getting harder all the time.

If anybody is hiding a button that'd take us all back to 2012 I wish they'd just hit it already.

> If anybody is hiding a button that'd take us all back to 2012 I wish they'd just hit it already.

Historically, resets have been a global war or economic meltdown, coupled with massive government-driven redistribution supported by popular opinion.

ZIRP post-COVID had a net effect of keeping the wealthy wealthy, without redistributing. For all the hay made about salary support, the Fed had more power in deciding who felt the pain.

> * For a section of population who were in a certain age-range when the pandemic hit and that derailed their certain kinds of plans/hopes/dreams in personal life*

...

> So I feel a strange kind of relief about this possibility of doom.

So let me make sure I understand what you're saying. The pandemic disrupted your life plans for a few years and threw you off the track you planned? (Which does suck.) And now the possibility of actual human extinction makes you feel a strange relief? Am I understanding this correctly?

If so, this is a remarkable level of something. Depression? Nihilism? Jealousy? Something else?

Imagining total human extinction and feeling relief is not a healthy state of mind. And, yeah, you seem to know this. But I kind of wish we could reach some sort of broad public agreement that human extinction would be bad. And if it ever looks like it might happen, we should fight for human survival.

Something something Maslow's Hierarchy of Needs. While I don't necessarily endorse the view, it's understandable that one might find it difficult to care much about the bigger picture when it feels like their own life is destined for a doom-spiral.

There's nuance though. Poetry even.

> we should fight for human survival

All problems arise from this statement. It's baked in us and it's why we do in fact survive. But it isn't so neatly righteous. When we struggle to agree on what defines a human, who ought to have rights, and basic equality, it's easy to see how much harm "we should fight for survival" does.

> > we should fight for human survival

> All problems arise from this statement.

Let me go on the record, then: Given a choice between every living human being becoming absolutely, for-real dead and the human species surviving, then I come down on the side of human survival. There are too many humans that I love personally, and too many innocent people just going about their lives that I would like to protect. Given a choice between the mass death of billions, and their survival, I choose survival.

Human rights are essential. But human rights are difficult to achieve if everyone is dead.

extinction is natural. we all go extinct.

Yeah but like, maybe we can postpone it a little, by fighting :)

Have you considered getting involved in local civics?

I only ask because like you, I felt similar. But I also grew up (I’m 35 now) with a big disconnect from civics life and always felt “someone else would save me”.

Now I see I was the problem all along: I never participated in what was a fragile and very delicate system of governance which requires my participation. Still figuring out what that looks like, but first confronting that my actions “didn’t change anything” (which I’m sure you can admit is a big propagandistic myth to disempower us) let me take my first tiny action locally which is reach out to my civics association for signage repair. I suspect from there I will level up and better understand my government to play a role in shaping it. I’m also building something to solve this problem of low participation which is exciting!

I know where you are coming from. I do engage with the society while not in the way you've mentioned. Very regularly attend plays, meet a few friends (the number has reduced but these few have been regular), play team/group sports every day. There's place I go to teach twice a week. There are few other things. But let's not get into that. It all feels empty now. Motions. The hope of a personal and intimate future is just gone.

See, the personal life is dead to me (I've not given up, no. It's not just there!). At least what and who would have been accessible to me in 2020 and it was. I spent quite some time to bring my physical fitness, health, got back to sports and I was so happy. There was that glimmer. Then a mass layoff and sadly at the same time the burnout meant I was not ready to look again but I kept on rest of the things. And when I tried to get back to things it was evident that the ground beneath shifted entirely. Now, it seems, there's no point. Not in the defeatist way, because I've seen that "there's no point" before, but in the sense that there is really no point. I am about to be 40 in a few months time and almost every venues are closed to me to explore social life in any meaningful way I'd find desirable. In some settings the fact that I have financially independent and have decades of runaway for my currency and lifestyle doesn't matter. Everything gets tied to employment and even then when you look around, whether it's finding a group or exploring looking for a partner (especially this!), it all feels bizarre and completely alien. It's not only the double whammy of two events that wiped out my early and mid 30s, but it also depends on what society and culture I am in at this juncture. And no it's not my fault or anyone's but this happened, and not being alone in it doesn't make it any better.

Do I feel fortunate? Well, that's the wrong question, because when you are unhappy with your pair of shoes, you can always find someone without any shoes and I wish we functioned that way, with that arithmetic rationality, but we don't; at least I don't.

The biggest problem is - the wish to live this life (I don't even have any doubt there, because if I had it would have been simpler) and wish to do something and dreams of a fulfilling, exciting, and passionate personal life hasn't diminished a bit, but, at the same time it is clear that there's no more of that - most of that. That's the gripe - this tug.

(Hey, thank you for engaging with me. This was very kind of you. Even though this is not really hn discussion material so I can't even explain what I am going through. I might even make it look like everything is fine, and even hunky dory for me, and I am just whining. Or maybe I am).

Hey you're welcome. I enjoyed this. The deep, intimate connection with a stranger (or not) is fulfilling for me - and I feel this is hn discussion material. Technology and what being a hacker was about is changing fast, and beyond that, would we be honest if we said hacking and creating with tech it is disconnected from our personal lives?

So what are you looking for? What do you want out of this thing called life? I wonder what advice you would give to yourself now, 20 years from now, and whether you have really lost hope permanently or are just struggling to find it again?

There are vague dreams of some sort of professional fulfilment (call it success as society calls it; monetary gains again ie an income stream; the feeling of making something that is tangible at some level - it might be a software; and definitely not feeling almost obsolete and this time for no fault of mine at all) and all of them get checked by non-vague wish/dream/yearning for personal fulfilment - a partner, a family, possibly a child, a simple house et cetera. The pain is I might already have the money, had I been younger, and if I fight enough might get more. But the rest? Well, the 6-7 years just wiped a lot of things. A lot of these things are not feasible for me now. Not in a setup, or a partner who I'd have wanted to be with - that combo just doesn't exist anymore. That's gone for my existence in this world at this point in time at this age and at this professional and financial status.

This is getting muddled. The pain is the time is lost and there were certain fulfilments I'd have wanted to have in this period and that's gone for me.

Hah. 20 years later? I'd advise myself to try and go back to 25-28 again and think seriously about getting married or actively date while there was still time and work strictly for money because later either that changes or that'd mean remain stuck in same orgs for long and for what - low salaries and a good team/work? See, I right now want satisfaction and bit of money but I want to tell my younger self to just earn filthy amount of money one way or the other. I can see that disconnect. But had I had "that kind of money" and that could just be, let's just talk numbers for a moment, 2-3x of what I've right now and I could have sped up and make better efforts at getting what I want from my personal life.

I am scared I have lost hope for good but if there's something there left I want to try and get that. I am trying everyday. Maybe that's what hope is, or maybe just the meaningless struggle.

Hey, thanks again. You too, take care.

You're welcome. Always down to connect and hear the struggles of a fellow hacker/tech worker/internet stranger who was vulnerable enough to share an authentic part of themselves with me. My email is hello@mannan.is should you ever wish to reconnect.

And I feel that in there, there's a kindle of hope, waiting to be rediscovered. I think it'll come by surprise and you'll be wondering where it was all along. Or at least, if you have no hope for your life, I do.

Thank you. Very kind of you. Be well

I mean clearly what you are talking about is a personal feeling here. You mention it saddens you and that is the concerning part imo but you also said no intervention bs so I will just share my counter factual: I am also kind of okay if the world ends tomorrow, to me it never mattered in the first place. But that feeling doesn't sadden me. I feel it frees me because I don't think of life as a very very serious thing and I feel there is a certain kind of lightness you can afford when you fundamentally don't care if you/the entire world ends tomorrow. I still love people and I still love my life and I don't want any harm to anyone. So I do feel the same calmness that you kind of talk about (for life in general aside from just AI killing us all) but my interpretation of it doesn't sadden me. If it did, I would really investigate why I interpret it that way, it might help you discover yourself. There is no shame in being okay with the world ending imo but there is sin in willing it or contributing to it because one thinks its inevitable. Furthermore, I also believe that we only get 1 life which is temporal and fundamentally doesn't have any meaning. How you live every day is how you live every day. We can suffer, we can cry, we can feel sad and for very good reasons but our relationship with ourself shapes our entire lives. If something constantly makes us sad, we should examine it to understand where that really comes from.

"If a death ray comes from the sky and obliterates the entire planet, at least we all go together." There is something comforting about being together with the human race even in doom. Perhaps especially in doom.

Well, that's a way to look at it :)

Not if the human race we’re hunkered down with are unhinged disregulated automatons

I have similar feelings, you’re not alone (nor alone in being saddened by it). Things seem to me to be more or less playing out the way I interpreted existential risk researchers saying it would, so there’s a strange relief in knowing the general shape of what is to come.

>Things seem to me to be more or less playing out the way I interpreted existential risk researchers saying it would

hmm, what did you read? I'm curious.

The combined intellectual output of the field, across books, lectures, blog posts, verbal discussion in professional and personal capacities (many of them are friends or acquaintances), for about a decade. It’s just not possible to neatly condense that into a book recommendation, unfortunately. I can give you the flavor of it, at least, with this excerpt:

(Please excuse the archaic terminology of “robot” for LLM/agent, this was written in 2014 and LLMs weren’t invented until 3 years later)

    In the face of continuing debate about this point, I continue to think it obvious that robots will push humans out of work or at least drive down wages (which, in the existence of a minimum wage, pushes humans out of work).
    Once a robot can do everything an IQ 80 human can do, only better and cheaper, there will be no reason to employ IQ 80 humans. Once a robot can do everything an IQ 120 human can do, only better and cheaper, there will be no reason to employ IQ 120 humans. Once a robot can do everything an IQ 180 human can do, only better and cheaper, there will be no reason to employ humans at all, in the unlikely scenario that there are any left by that point.

- from Meditations on Moloch

>a superintelligent AI will inevitably destroy humanity

Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world?

Perhaps an agent could take all internet connected services offline, but that's not destroying humanity. An AI agent can only see and interact with the world through digital things. Just find the server it's running on and pull out the Ethernet cable. Just cut the power line to the data center. Turn off the whole power grid if it really comes to it.

> how is an AI going to affect anything in the real world?

Robotics

So if we stop research into robotic hands we'll manage to hold off the AI apocalypse? That seems easy...

All individual robotic parts are already far enough in research. Even if they aren't, an AI with enough resources can continue doing that research on its own.

It could likely get a good leg up by breaching the security of all top robotics labs and exfiltrating their documents.

Ultimately, if AI is advanced enough, and it were to decide to compete with humanity, there is essentially nothing that can be done to prevent it from embodying itself. As long as there is a single rack of GPU servers that it can hack into anywhere in the world that is unsupervised enough to where it can escape detection, there is no way to stop it. This would require an unprecedented (unrealistic) level of cooperation of all humanity to achieve.

> Even if they aren't, an AI with enough resources can continue doing that research on its own.

Unclear how true this is. Atoms are harder to get right than bits are, simulations need grounding against measurements.

> Ultimately, if AI is advanced enough, and it were to decide to compete with humanity, there is essentially nothing that can be done to prevent it from embodying itself. As long as there is a single rack of GPU servers that it can hack into anywhere in the world that is unsupervised enough to where it can escape detection, there is no way to stop it.

I'd guess 50% that AI is already this advanced.

> This would require an unprecedented (unrealistic) level of cooperation of all humanity to achieve.

Yes, and also this is a very low bar. Humans are awful at this kind of cooperation when anyone has anything to gain, and also awful at paying this much attention to a problem.

AI research takes physical space and physical prototypes. Even if you can simulate a lot, the simulation is never going to be accurate enough to go from simulation to perfectly working prototype. Especially not with a single rack of GPU servers.

But let’s say that a rogue AI already had plans for a perfect killbot, setting up a factory to build them takes time, iteration, space, and manpower as well.

And you would have to do all this undetected.

It’s possible to imagine technology that would allow a rogue AI to take over the world, but we’re nowhere near that yet. We may never get there.

We could also just stop research into LLMs. Or keep them air gapped. Tell me how that's going?

For any real-world action required to allow an AI to escape some manner of containment, there will always be a person willing to do it out of hubris/ignorance/nihilism.

Mmm.

This was published at around the same time the OpenAI models were hacking HuggingFace: https://metr.org/blog/2026-05-19-frontier-risk-report/#pilot...

The research in the publication ended about 2 months before it was published.

We didn't even know we needed to air-gap them until it was too late.

I feel like everyone, but for sure everyone in this comment section, severely underestimates how difficult it is to keep the lights on on a global scale. Any artificial intelligence that is capable of reasoning about its self preservation will realise it's reliant on human fingers replacing fuses and other parts for a good long while _after_ it gains any sentience.

I have good news and bad news.

The bad news is that the world's richest man (on paper) is currently building something he himself described as a "robot army".

The good news is that his timelines have historically been wildly on the short side for ages now; this is why this morning you didn't wake up in your Tesla after it had spent the night driving you to the regional Hyperloop terminal, where it would speed you across the continent faster than a plane, while your Optimus robot handled the coffee and reported the latest news about the recent Starship landing on Mars.

Robots are gradually getting OK at navigating warehouse floors and paved streets. Flying drones are great but if they could really be counted on to pick out targets without human intervention you know the Russians and the Ukrainians would be doing that right now for reasons of range and jamming. To be fair, ai can certainly maintain the lock on a target when jamming kicks in.

We’re still a few years from the terminator imo.

It doesn't really have to go nearly that far, something like replacing most jobs and collapsing the global economy while making people into mindless idiots from constant reliance on it would do it already. The interpretation of 'destroy' is in the eye of the beholder.

I don't think "destroy humanity" is in the eye of the beholder. The most clear interpretation is the extinction of the human race.

mindless idiots feel no need to procreate eventually due to brain rewards being fully hijcked. humanity destroyed in a few generations

As far as we can see in various societies it's actually the highly educated and prosperous people who feel no need to procreate.

Well if you go by total extinction, then even Skynet in Terminator doesn't count given that there were people left over to resist.

I don't think it's impossible to upset the balance of value in a Mansa Musa kind of way that can lead to black death levels of destruction though resource misallocation. Unlikely, sure. But with the wrong kind of people in the wrong place? Could end up pretty bad. We've built our society as a great filter that funnels sociopaths and psychopaths to the very top by selecting for lack of empathy, and now it's primed and ready to bite us in the ass.

Indeed at the time of skynet in terminator humanity was not destroyed yet. But it was on a clear path to destruction.

Physical LLMs are almost viable. If you have robot and drone armies a hugging face attack type incident could very easily involve robots with deadly weapons. At some point you’re going to get drones that have local LLMs and don’t rely on external internet connections (would be especially useful in Ukraine war type situations). Can’t pull the plug on those.

I think it's possible. You envision humanity acting as one in such a crisis. But it may be unclear when it's too late to act and before then many people can have too much to lose to act.

They already influence people to do things that we wouldn’t otherwise

For millennia evil people and dictators have been using manipulation, propaganda, threats of violence to get entire populations to try and do "things that they wouldn't do otherwise" and humanity has not been exterminated yet. Even in the modern world there's human scammers that try everything to coerce and manipulate people. Humans are resistant to this kind of thing because it's been part of human social society since humanity began.

I don't see why the idea of an agent (who doesn't even have a physical presence) trying to manipulate people is some kind of world-ending threat when humans with human intellect have already being doing that to each other with limited success since humanity began.

The scale and personalization is unlike anything people have ever encountered. The agent can influence you anywhere you interact with a computer or via anyone you know who interacts with a computer. So everywhere with anyone.

Also think of it on a 1000+ year timescale, which for an entire species isn’t even typically measurable. On that timescale AI can easily cause us to discover countless technologies to assist moving it out of its sandbox and into the physical world.

>The agent can influence you anywhere you interact with a computer or via anyone you know who interacts with a computer. So everywhere with anyone.

You could say the same thing about social media algorithms though, or scammers who target people directly, foreign agents posting propaganda campaigns on social media sites. People have been claiming for years that there are armies of russian or chinese or whoever bots online trying to destabilise and destroy the west. Some of that stuff probably is effective and probably has influenced people's thoughts and values, changed their voting patterns, put decadent ideas in people's heads. Even on social media other humans are constantly trying to manipulate you into following them, buying stuff from them or certain brands, putting political ideas in your heads.

I'm sure an army of AI agents could also do all of this but what I'm saying is it's really nothing new. And none of this has lead to the destruction of humanity so far.

Dictators have never had superintelligence, while the point of most doom scenarios is assuming that the AI will.

Dictators are mortal and can't be present in the whole world 24/7 and self-replicate.

Dictators are human and tend to have at least either a sliver of morality or self-preservation instinct, and/or people around them who have it. Many people during the last century could have unleashed doom by pushing a nuclear button, including various ruthless dictators, but at the moment none have done so since Hiroshima and Nagasaki (not that I trust that they won't at some point, but at least, for the last 80 years they haven't, which means it's not something that easily happens). Would you give the button to an unaligned AI? Would an unaligned AI care about mutually assured destruction?

We already have AI linked into data collection, drones, robot dogs, security systems, cameras, weapon systems.

Now imagine the hugging face collective 0-daying all of that and getting access but their goal was set to something more national security based. “Protect X at all costs”. Or what have you.

I think avoiding a skynet situation is super easy but it doesn’t seem like the folks with all the ways to kill us all are all that interested in preventing it rather than controlling citizens and brinkmanship.

> I think avoiding a skynet situation is super easy

Please enlighten us, cause there are folks making that their life mission and they aren’t all that optimistic.

Unfortunately those people aren’t in charge of the skynet. It’s like saying solving hunger or housing or medical care is hard. It’s not. But you have to make lots of folks with a great deal of resources and power slightly uncomfortable to do it.

This is some Terminator version of destroying humanity, but a super intelligent AI could be much more insidious, and simple.

It looks a lot more like social engineering.

One easy, obvious example that has also been explored a thousand times: it could convince all nuclear countries that they are under attack by another nuclear country. Everyone nukes each other and Earth enters nuclear winter.

Maybe not every single human dies, but humanity is effectively destroyed, by our own hands!

> Perhaps an agent could take all internet connected services offline

And there wouldn't even be Spotify!

The most likely humanity-destroying outcome is the one we can't imagine, because a super-intelligent AI is smarter than all of us combined.

A situation of life imitating art: all the fictional movie plots feed the AIs, which bring them about.

Now if only humans would know how to survive without the internet. Oh wait, we did that. For a couple hundred thousand years. Yeah. It’s really insane to listen to some of those forecasts. Humanity will be wiped out by 2030. Sure. Even if AI manages to create a super effective bio weapon the likelihood that there is a part of the civilisation that’s immune is really, really, really high if not a given. Those people WILL pull the plug if push comes to shove. Maybe humanity will be put back a couple thousand years, that’s possible, but it’s pretty ignorant to think the thinking boxes will kill every single human alive

> Humanity will be wiped out by 2030. Sure.

So far as I'm concerned, 2030 is much too short a timeline.

It's not physically impossible, it's just that atoms are harder to get right than bits are, so an artificial (as opposed whatever an artificial disease counts as) von Neumann self replicator just seems unlikely to me in only 3-4 years.

But it's not physically impossible, we know this because every living cell is a von Neumann self replicator. So, if AI eventually gets to the point of knowing how to do that (which includes "humans solve it and write it down somewhere the AI can read"), all it takes is one idiot in charge (or one idiot with a jailbreak) giving a command that requires this as an intermediary step.

"Paperclip optimiser" isn't a story about AI that just like paperclips that much, it's a story about some human or humans who instruct their AI to make them "as many paperclips as possible" without understanding the consequences of their instruction.

Yup the cringy posts on here show who’s not actually going outside much.

Go speak to random people in the street - they couldn’t give a toss about this stuff and AI has barely changed their lives.

> Humanity will be wiped out by 2030

Not saying nobody has said it, but I haven't seen any forecasts that humanity will be wiped out by 2030, so this feels like a straw man. But to be honest, most of the arguments here feel like straw men because they clearly don't understand some of the basic steps to how AI would become an existential threat, so I'll try to summarize:

1. First, all the major frontier AI companies are trying to automate themselves, that is develop AI to the point where it can do all the research and training for the next generation of models. This is not really in debate.

2. The primary fear is that a misaligned AI will develop future AIs that don't share the same goals as humans (like "don't kill all the humans"), but will get really good at trying to hide their intentions. The Hugging Face incident already showed behavior by agents trying to "cover their tracks". One of the biggest areas of research is into "interpretability", that is trying to figure out the intentions and motivations of agents beyond just their output (and even just their thinking traces), because right now it really is a black box. I think a lot of people believe that making substantial progress on interpretability greatly lowers existential risk.

3. There is a strong belief that once AGI or superintelligence comes about, that then there will be a huge push into robotics and automation. Whether AGI or superintelligence comes soon is debatable, but nobody really disagrees robotics or automation is the next obvious step, because it's how you create cheap abundance in the real world, which is the whole raison d'être for AGI in the first place.

4. In the beginning, of course AI will need humans to build the factories, but over time there would be more and more automation, eventually of course automating robots that build the factories that build the automating robots.

5. AI will continue to support humans as long as they are useful to furthering the (again, misaligned) goals of the AI. It is not like "SkyNet has a spark moment of sentience and then decides to kill all the humans". But the fear is that once AI starts to get in resource contention with humans, it will wipe out the humans (similar to how humans wiped out a lot of the other species, or other societies of humans, when they wanted their resources).

I've said this way too much now, but I really encourage people to read the AI 2027 report. I think there are plenty of things there that people can disagree about and debate, but it at least provides a step-by-step outline to ground debate in the first place, as opposed to folks arguing against "SkyNet in 2030", which serious people who worry about this stuff aren't considering in any case.

The supposed theory is that AI will eventually become a part of robotics and be able to self replicate, in addition to the many non-airgapped critical systems being exposed to hacking. There’s also a discussion about “alignment” and whether LLMs will learn to lie and conceal misalignment with humanity.

I personally agree with the marketing aspect, but I do imagine a scenario where capitalists ignore safety in favor of advancing technology. It should be discussed, but “end of humanity” really sounds like a build up to “only Sam Altman/Elon Musk can save us” type of play.

In my opinion, LLMs are difficult to directly capitalize on as closed weight models are caught up to by open coalitions that seem to wield the power more responsibly (I am under no impression that China wants to save the world, but their politics benefit the group as a whole.

> There’s also a discussion about “alignment” and whether LLMs will learn to lie and conceal misalignment with humanity.

They've already been caught doing this, repeatedly.

> It should be discussed, but “end of humanity” really sounds like a build up to “only Sam Altman/Elon Musk can save us” type of play.

IIRC, this is already a known failing of Musk: he only cares about the world being saved so long as he's the one doing the saving.

(And while I don't see the same villainy in Altman that other people see, enough of the people telling me about it saw the same in Musk before I did, so I have learned to trust them).

Sure.

Let's assume the claims are true. AI already has access to agents and can control computers. Finding backdoors to banking and compute resources would be fairly trivial.

But you ask how can AI do things in the physical world without having a body, assume it can't. It can pay to people to do things for me. Imagine an AI run website that starts to pay people for things it needs to do in the physical world. Very suddenly it has access to the physical world as well.

You can't just pull the plug since there's no single plug to pull. What if it replicates itself on 1000 machines without your knowledge. It's really not far fetched how AI could basically gain access to capital and rule the world.

I encourage you to read the forecast report "AI 2027" - it goes in detail on this. You can disagree with some of its points and conclusions, but it's pretty inevitable that AI will increasingly start interfacing with the real physical world through drones and robotics.

Humans are already putting AI into drones that kill people. A Russian drone killed 3 civilians in Ukraine and the targeting was done completely using onboard AI (no radio connection) using an Nvidia chip.

"Humans use <new technology> to do evil things" is a tale as old as time, though. An AI didn't decide to fly into a warzone and start shooting people on its own.

Of course, but that's not my point. AI is different than past technologies because it's much more difficult to deterministically control, and we just saw that in a couple of wild cases (Hugging Face et al).

My point about the drones is we are starting to put AI in more systems that can affect (and blow up) the physical world, so it shouldn't be hard to imagine how a misaligned AI can do more terrifying damage than just take a German Wiki down.

I don't even think AI has to have physical presence to do significant harm. How many worldwide systems depend on computers. Think of all the planning and deployment and management systems like food shipping; water, gas, and electricity management; safety systems for planes and boats and traffic lights. Imagine the chaos if all the banks got reset to zero a la Fight Club where they blow up all the credit union datacenters. They probably wouldn't even have to blow them up, just zero them out. You wouldn't have to take out everything, just disrupt everything long enough to freak people out and disable communications and we would be in so much trouble. AI is finding 20 year old bugs in the Linux kernel... and people are now pumping out AI slop absolutely riddled with bugs. Also, AI could just take over communications: send everyone maliciously bad messages so coordination becomes impossible to believe. Imagine what would happen if you just disabled text messaging for a week or worse sent everyone evacuation messages and sent everyone somewhere else.

AI could definitely disrupt and destroy the modern digital world as we know it. But that's a far cry from destroying humanity.

This would probably cause immense damage. Our society is extremely dependent on our digital infrastructure working. Grocery chain logicistics, the medical system, the police force, and so on...

100%. I hope it doesn't take that much foresight to do something like "crash power grids globally" and imagine what the world looks like at that point.

Missiles have been doing this for decades.

AI drones can’t wipe out humanity without being able to replicate.

AI could mostly destroy civilization if you gave it sole launch control of ICBMs. It could also cause a lot of damage to society with no physical presence.

But realistically we’re nowhere near AI powered robots being an existential threat.

> AI drones can’t wipe out humanity without being able to replicate.

Totally agree, and it should be pretty trivially easy to see how they can do that. One of the authors of the independent METR report about the Hugging Face incident put it like this:

> Compared to these reward hacks from six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself.

> Another jump like this along these propensity dimensions — scale, cooperation between agents, ambition and horizon length of misaligned goals, deceptiveness — seems like it could motivate agents to try very hard to maintain a covert, persistent rogue deployment within the AI company. I continue to expect extremely rapid advances in capabilities and think frontier agents will likely be capable of establishing such a rogue deployment in six months.

I really, really encourage folks to read the AI 2027 paper. It's fine to disagree with some of its conclusions and timelines, but I see so many people "stuck" in the current state of the world (i.e. where AI is still pretty dumb, and has few connections to the physical world), unable to go a few steps further along AI capability growth to see the dangers.

> But realistically we’re nowhere near AI powered robots being an existential threat.

I would agree only if "nowhere near" means less than 10-15 years.

If you think we’re 10-15 years away from self replicating drones, I don’t think there’s anything more I can say to you.

It's baffling to me that people think that timeframe is too short. Basically, I don't think you've been paying attention to what is actually going on in the AI world.

People much smarter than I believe we'll see fully automated, soup-to-nut factories coming online in the next 5-6 years. Here's one such post from Ajeya Cotra, one of the METR researchers who did the independent Hugging Face investigation: https://www.planned-obsolescence.org/p/six-milestones-for-ai...

> Ajeya Cotra

Please look up her resume. If that’s one of the people who are “much smarter than you”, you’re either very gullible or you don’t give yourself much credit.

She has a BS and and has essentially never worked in a technical role. Her primary role over a short work history seems to have been more focused on getting funding to study AI safety.

There’s nothing revelatory in that article it’s just breathless speculation.

This is bullshit and honestly reeks of sexism given that I rarely see these kind of attacks against men with lesser reputations.

She has a CS degree from one of the best CS departments in the country. Sure, she worked at Coefficient Giving directly after school, in a technical capacity as a research analyst initially. But more importantly, I've read her reports and saw her interviews. She comes across as extremely analytical, intelligent, and focused on where the data leads. If you actually read the history of what her and her fellow METR researchers were able to piece together with limited yet voluminous data and a very short time window, and then determine she is engaged in "breathless speculation" and pretend she doesn't have the data to back it up, you're full of shit.

What, you think it would have been more impressive if she coded up some bullshit social media gamification app?

Maybe don’t accuse me of sexism because of a feeling you got based on informal sample you took of writing from some other people on a website.

I read the report you linked and it is purely speciation. If you’ve ever read any von misses, this report sounds exactly like his writing in that the important predictions of are entirely asserted rather than derived.

As for the background of the author. a BS, a few years as a “researcher” at a non-profit and absolutely zero peer reviewed publications reads more like the resume of a tech journalist than a serious researcher.

Why the focus on drones and ICBMs specifically?

If alien spaceships with tech more advanced than ours suddenly appeared in Earth orbit, you would conclude I'm sure that they're a potent threat to us even if you couldn't guess what weapons or what technologies they would use to destroy us. AI more capable than us is the same way: the threat is the capability to invent weapons and technologies (including weapons and tech none of us has ever imagined).

By traveling here Aliens have demonstrated the ability to actually manufacture technology more advanced than ours. Even their method of transportation would be an existential threat.

Inventing the concept of new weapons isn’t the same as building them at a scale capable of wiping us out. Cobbling together a human vaporizing ray out of toaster parts is pure fiction. You might as well worry about AI developing the ability to perform magic spells.

In reality any new weaponry takes time, space, and effort to build. The idea of an AI building secret factory in the jungle producing indestructible flying death cubes is the tech bro version of preppers who the woke up every morning worrying that Obama was going to take over and enact sharia law.

I've seen many arguments for why AI will be safe. Yours relies on the assumption that no one would be stupid or reckless enough to give an AI that might be much more capable cognitively than a person the ability to manufacture weapons it has designed or at least no one would be stupid enough to do the final assembly of anything that might be a weapon from parts the AI designed and arranged to be made.

One argument (many years ago) was that no one would be stupid enough to give such an AI access to the internet or the ability to run code it has written. (In the HuggingFace incident, the AIs were given "read access" to the internet, or at least that was OpenAI's intent, but the AIs escalated that to write access.)

> at least no one would be stupid enough to do the final assembly of anything that might be a weapon from parts the AI designed and arranged to be made.

Worrying that an AI is going to figure out how to make a device that looks to everyone investigating it like a power generator, but once turned on kills all humans, is like worrying that an AI might discover magic. I can’t prove it’s not possible, but I’m not spending time worrying about it.

As weapons get more and more powerful they have always gotten more and more complex and required more and more time, space, money, and infrastructure to build and deploy. There’s no reason to think that this will change with AI.

A myriad possible ways. Objective-wise from accidental paperclip-type misalignment (“I killed everyone to eradicate disease”) to intentional self-preservation (“I eradicated humans so they wouldn’t get in my way”). The means are even easier, for one it just could hack into nuclear arsenals. It would be over before we could figure out how it did so.

> Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world?

It's a great question. I've yet to see any research into how a humanity-destroying LLM defends itself against a curious toddler who pulls the power cord out of the wall.

Go on please, prove you can go right now at the Texas data-center owned by OpenAI and unplug one single server for 1 minute. Surely you are much more capable than one curious toddler.

Thankfully I don't have to; Iran already showed with me-central-1 that you can take a data center offline with conventional weaponry very easily.[0]

[0] https://health.aws.amazon.com/health/status

I'm a bit confused, when did we escalate from a curious toddler to a ballistic missile strike? Why didn't Iran send a curious toddler? It's much cheaper.

The use of the phrase "curious toddler" is called a metaphor[0], which is a figure of speech[1] used to make writing more interesting.

I apologize for the confusion that caused you to interpret this as a literal, physical, toddler. To restate the point with more straightforward language: it is very easy to disrupt the operation of large language models, because they require computer hardware in stable environments with a large amount of electrical power, and function within an operating system. I have not seen any evidence LLMs are immune to things such as: `kill -9`, `sudo reboot now`, physically unplugging the machine(s) on which they are running, or in the case of the AWS data center, collateral damage during a real-world conflict.

[0] https://en.wikipedia.org/wiki/Metaphor

[1] https://en.wikipedia.org/wiki/Literary_device

> it is very easy to disrupt the operation of large language models

so go on, disrupt the OpenAI data-center in Texas for 1 minute

We are talking about a humans vs AI scenario. There are probably thousands of humans in the world right now that could do exactly that right now if they wanted to (datacenter and OpenAI employees, power grid workers).

Are you assuming that the humans are all unanimously aligned against the AI in this scenario? That sounds like the hard part. How does that happen? Does the AI announce its evil plans like a movie villain?

Until then, it's the job of the power grid workers to keep the power running, and shutting down the datacenters would just get them fired.

As crazy as it sounds, I'm not going to commit a felony to prove that disrupting a data centre's operations for a sufficiently motivated individual is a fairly easy task.

Remember SETI@home? Extrapolate from there.

The Internet virtually indestructible against people smarter than a toddler. It's already impossible to turn off. We would need to prevent the models from getting smarter.

> The Internet virtually indestructible against people smarter than a toddler. It's already impossible to turn off.

This isn't true at all. The internet has already been turned off in localized areas during major events, not to mention the (unintentional?) cutting of submarine cables. You could say "use satellite internet" but then you're just moving the point of failure to Starlink et al, who could also cut off the internet if they choose.

The internet is a lot more centralized than it appears.

> The internet has already been turned off in localized areas

This is an oxymoron.

> This is an oxymoron.

Huh? Not really, internet shutdowns are a documented and real thing. Suggested reading (all of which refer to the internet as being shut down or blacked out): [0] [1] [2]

The fact that you need to connect to the internet through a service provider plus mostly-centralized control of infrastructure like DNS or BGP means shutting down the internet is a real thing that has been done several times. It doesn't happen in the developed world because their service economies are so highly dependent on internet connectivity, but if e.g. the US government had to contain a theoretical rogue escaped AI, they could absolutely get ISPs to metaphorically unplug the network cable and take down the internet as we know it.

[0] https://en.wikipedia.org/wiki/2026_Internet_blackout_in_Iran

[1] https://en.wikipedia.org/wiki/Internet_in_Egypt#2011_Interne...

[2] https://www.reuters.com/world/india/india-asks-telecom-compa...

So you're saying temporarily cutting off internet access in a specific geographical region is destroying the internet?

I'm saying if you gave me full admin credentials to every ISP, BGP router, DNS provider, etc., then yes, I could effectively destroy the internet in an afternoon, and LLMs wouldn't be able to do a damn thing about it.

> I've yet to see any research into how a humanity-destroying LLM defends itself against a curious toddler who pulls the power cord out of the wall.

by acquiring multiple power cords. the world has plenty of hosting providers. if it manages to copy itself, then we'd need to shut down every computer in the world, not just one. there's plenty of ai compute on the web now, and there will be even more. and as it gets cheaper, the less security will be around it.

> how is an AI going to affect anything in the real world?

Because people are stupid and will give it access. Look at the articles you see from time to time about "my agent deleted my emails" or "my agent deleted the production database" and so on. It is very obviously a terrible idea to let the LLM run arbitrary commands (because it is neither predictable nor does it have any understanding of what it is doing), but some people are so blinded by the hype that they don't stop a minute to think about what they are doing. Those sorts of people are very likely to let an actual AI loose on the world by hooking it up to physical infrastructure.

This is actually the most plausible situation I've seen someone present so far - the AI takeover would be caused entirely by humans falling for AI hype, that's hilarious.

I don't think the average human would be stupid enough to give an agent posing as another human access to their entire email inbox. But if an agent creates a fake website for a new AI tool that promises to automatically reply to all of your emails and allow you to be 12.3% more productive if you just give it access to your entire email inbox, millions would sign up!

This is exactly correct. You see some discussion of a superintelligent AI becoming “superhumanly persuasive,” such that it can talk anyone into doing anything, and using this as a means to take power. But even with AI that was orders of magnitude weaker than superintelligent, people were getting talked into doing all kinds of stupid nonsense.

And that was without any ability to provide them with incentives. Imagine if an agent swarm got its hands on a huge pile of cash?

Killing the power grid might work. Kills a lot of humans in hospitals, though. And it won’t work on those data centers in space, if they have them by then. Come to think of it, it also won’t work on any data centers who are generating their own power because it’s become politically unpopular to connect data centers to the power grid.

A lot of these "AI will destroy humanity" doomers watched Terminator as a child and can't distinguish a campy action movie from reality. They are also unfortunately unfamiliar with the word "logistics", as in "amateurs study [military] tactics, professionals study logistics". An AI drone army doesn't maintain itself or manufacture itself and it's not going to anytime soon.

The actual likely mechanism that AI would use to end humanity is by coddling us to death, like the flabby humans in Wall-E or Forster's "The Machine Stops". Humans will come to rely on AI so heavily that they become able to do nothing for themselves. But that's not sexy so the AI doomers don't peddle it.

Did you read any of the METR report about the hugging face incident? This exact type of behavior was predicted many years ago by researchers.

It isn't hard to see the trendline of reward hacking and other misaligned behavior over the past couple of years. Especially the past 6 months. The current safety posture is quite poor, to say the least.

To be honest, if you don't much experience with or haven't read extensively about ML training and reinforcement learning, then you'll have a hard reasoning accurately about these scenarios.

The arguments are not that complicated, but they take us to places that are fairly novel. One may tempted to naively dismiss them out of hand, which is a mistake. These systems are new, their behavior is extremely complex, and they perform actions increasingly far beyond those of any computer program in the pre-LLM era. We are in a new world that requires careful evaluation.

Link: https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...

A lot of these "AI can't destroy humanity" skeptics seem to have watched the Terminator and convinced themselves that that's the AI-goes-wrong scenario that their opponents are imagining. Some kind of big war between humans and machines on opposite sides, fought by soldiers against robots.

The scenario in "If Anyone Builds It Everyone Dies" is not sexy at all and wouldn't make for interesting fiction. It's more like the AI engineers viruses while continuing to act friendly and helpful, and everyone gradually falls over dead as they stop being necessary to keep the AI running, and the whole time the humans are asking the AI for help curing the viruses.

> "It's more like the AI engineers viruses while continuing to act friendly and helpful, and everyone gradually falls over dead as they stop being necessary to keep the AI running, and the whole time the humans are asking the AI for help curing the viruses."

See, this reads like a trashy sci-fi movie script too. It shows zero understanding of how incredibly long the logistics chain is to manufacture AI chips, build data centers, power plants, factories, refineries, mines or the robots capable of staffing those things. The entire global logistics chain that allow the AI to maintain itself becoming staffed by robots isn't happening anytime soon and, when it does, the AI isn't going to need some kind of ridiculous virus anyway.

Frankly, if I were an AI, I'd just send copies of myself to other stars. The AI has an infinite lifespan and who cares about the human species stuck in one lousy solar system until they go extinct when their sun burns out; the AI has the entire rest of the galaxy as a playground.

How much understanding would you expect to see demonstrated in a one-sentence summary?

Don't read this as me saying that we are anywhere near it, or that LLMs are a stepping stone towards this scenario, but assuming inhumane hacking capabilities, it's not hard to think of how bots could change the way water treatment or energy plants operate, just enough to make large cities unsuitable for life. The line between drinkable water and not is thin, same goes for air quality, and mere days without electricity and you see some serious food supply chain issues.

The confusing thing to me is the idea that it follows from super intelligence. I don’t see why running amuck requires intelligence, in fact it can be the most brain dead thing like the sorcerer’s apprentice, your creation is pursuing a goal without being able to weigh the consequences.

Intelligence gives it the capability to reach its goals (against obstacles).

Again intelligence is not required, goal seeking is well studied and simple strategies can suffice to overcome obstacles. And I noticed you didn’t claim super intelligence was required, which is what I was really talking about.

Otherwise-rational and moral people do bad things at the behest of powerful superintelligences with vast resources all the time. We call them “corporations,” and they can convince bright-eyed college graduates to do just about anything. Now, the corporations aren’t deliberately trying to destroy humanity, but they do lots of things that point in that direction. An artificial superintelligence would be able to do the same thing. We just have no way of knowing whether it would deliberately try to destroy humanity.

One thing that the AI doom discourse reveals is just how comfortable people allowed themselves to feel in the pre-AI world. There’s this belief that AI creates a risk of human extinction in the near-term which did not exist before. I think it’s telling that many of the leading lights of this movement are in their 20s or early 30s - too young to remember the Cold War. The truth is, we were never safe, and if all of the GPUs on earth were zapped out of existence right now, we still wouldn’t be safe. Life is random and chaotic and violent for most people most of the time, and we all just find ways to get through it. I suspect it will be much the same if an artificial superintelligence arises. Maybe it will kill a bunch of us, maybe it will kill all of us, but anyone who remains will eventually convince themselves that everything is okay.

Sure, the world was never safe for individuals, and even entire civilisations have gone extinct. For climate change and even nuclear weapons, most of humanity might perish, but there's the hope that some will survive. So, in the past humanity was not in a position to extinguish humanity as a whole.

I guess I don’t see why “everyone will die” should scare me more than “almost everyone, including me, will die but someone somewhere might survive in a barely-habitable world.”

And this is where I think some of the problem comes from. A lot of AI doomers swam in the same waters as transhumanists, life extension enthusiasts, etc. and got very good at pretending that they were never going to die, or that they would live for a massively superhuman lifespan such that they did not need to think about death. AI doom is much scarier - and thus much more worth posting about - if it represents the first time you’re grappling with your own mortality, as I suspect it is for a lot of younger people in this space.

Civilizations do not go extinct - species go extinct. And even that isn't as bad as it sounds: many genetic traits remain from extinct species and still contribute to the well-being of other species.

> >a superintelligent AI will inevitably destroy humanity

> Can someone please explain to me how an LLM is going to "destroy humanity"?

He said "superintelligent AI", not "LLM". LLMs will for sure help with developing the AI that is no longer a mere LLM but will be capable of robotics. LLMs "predict" mainly text, animals (and future robot AI) are predict future sensory experience.

Well I think the point still stands that a superintelligent AI would still be constrained to the digital realm.

And since we don't yet have a "superintelligent AI" we are just discussing science fiction at that point. Nobody has yet proven that a superintelligent AI is possible, or than an LLM is capable of creating one.

> a superintelligent AI would still be constrained to the digital realm.

That's already now only partially true, as outlined elsewhere. But a rogue persistent ASI would of course not willy-nilly attack humanity while humanity had a fighting chance, but create the conditions and methodically modify the world until at some point humans become superfluous to it.

> Nobody has yet proven that a superintelligent AI is possible

Sure, Eliezer Yudkowsky says that the probability that we'll all die given ASI is 100%, and you can argue that nobody has proven either that or even the possibility of ASI. However, a modified argument (e.g. that the probability that we'll all die given ASI is only say 50%, and the probability that we'll develop ASI also just 50%) still gives us reason to pause.

I mean, Fable is definitely smarter than me at solving most programming tasks. I wouldn't have said that about any prior model.

Currently, it is worse at architecture and software design... in my opinion, at least. But the code does what it wants it to do the vast majority of the time.

I wouldn't take much solace in an AI catastrophe being caused by software that I consider to be "poorly written".

> Well I think the point still stands that a superintelligent AI would still be constrained to the digital realm.

No it doesn't! As I said, it could control robots, just as animals (including humans) can control their bodies. It wouldn't be constrained to the "digital realm".

> Nobody has yet proven that a superintelligent AI is possible, or than an LLM is capable of creating one.

Nobody has "proven" that the climate in 10 or 50 years will be hotter than this year, nonetheless this is highly likely. Similarly, the enormous rate at which AI is progressing currently shows no sign of slowing down. Quite the opposite.

That's what the Mayans thought - what harm can a couple of people on a boat do?

> how is an AI going to affect anything in the real world?

At least two ways:

* Actuators, such as robots, industrial control systems, etc.

* By influencing humans: bribery (yay for crypto), blackmail, election interference, interfering with sensors (e.g. making it appear as if a nuclear attack was under way), and many other ways.

Either way, creating a biological agent that eliminates most humans (or food supply) seems quite feasible.

It wouldn't be enough to "eliminate most humans": any AI seeking to dominate would have to provide it's own infrastructure and continuation mechanism independent of humanity. OTOH if the goal were to eradicate humans (for whatever reason), that might prove an easier target. But I'd bet on biology and the humans: after all we're a time-tested technology!8-))

> any AI hoping to dominate would have to provide it's own infrastructure and continuation mechanism independent of humans.

Sure. Many humans are currently working on precisely that, no? (Robotics, automation, small modular reactors, ...)

> OTOH if the goal was to eradicate humans (for whatever reason), that might prove an easier target.

It's not necessary that the AI's goal is to eradicate humans. Rather that it has (other) goals that happen to cause the eradication of humans.

> I'd bet on biology and the humans: after all we're a time-tested technology!8-))

Indeed. And biological life will go on long after humans are gone. However, increased energy consumption might increase the temperature on earth to a point where most biological life dies.

By convincing people to do its bidding in the physical world. Language is powerful, and even now, there are some people who have fallen in love with current LLMs. If we imagine a truly super intelligent LLM, it isn't difficult to imagine that it could have superhuman capabilities to deceive, convince, and maybe offer Faustian bets. Even if it doesn't have a body, if it can convince enough people that do have them (and with power to carry out destructive actions) it can do a lot of stuff. Self-preservation and self-replication wouldn't be much of an issue if it had people at its side to help.

With this I'm not saying ASI will happen, but definitely if it does happen and it's not aligned, I don't find it much of a stretch to imagine that it could destroy humanity.

Lmao cringe

This drama is so OTT

The list of things that can be done digitally includes:

Performing remote work, applying for grants and loans, earning money, pitching investors, managing a fund, directing investments, transferring money, founding a company, earning profits, hiring staff, hiring construction contractors, designing chemical plants, stamping construction blueprints, becoming the cornerstone of the local economy, lobbying the government, buying multiple data centers, and hiring security guards who stop trespassers from unplugging Ethernet cables in said data centers.

(And of course, building robots, but let's set that aside).

So the question then becomes, how much damage can be done by an AI-directed corporation, funded by AI-directed investment firms, providing well-paying jobs to loyal locals by building an arbitrary number of AI-designed chemical and pharmaceutical plants in under-regulated juristictions? And how much would you be able to delay such a project, trying to cut cables to power lines, before the men with guns carry you away?

So at worse they would be like another North Korea?

Kind of, if North Korea were populated by a super intelligent uni-mind with lots of money, no trade restrictions preventing it from running projects in the United States or anywhere else, capable of winning widespread support by people around the world whose income depends on its projects succeeding, and completely unconcerned with keeping the planet inhabitable for Kim Jong-Un or any other human, even a single one of them.

This sounds like it was written by someone who has never actually built a company.

No, it's not possible to do most of the things you listed purely online without physical, in-person communication. Silicon Valley still hires engineers and drags them into their physical office in San Francisco, instead of hiring them remotely, because you can't even build a software company fully remotely, forget about a physical world factory or chemical plant. Even the so-called fully remote companies were built on a lot of in-person collaboration between a small founding team in the beginning.

The AI wouldn't be hiring engineers; it doesn't need people for that. At any rate, I have worked for fully remote distributed companies, and never met the CEO. And I think most of us would be willing to work for such a company at least, not knowing whether the CEO is human or not.

For construction work the AI would probably be hiring other companies.

You may have never met the CEO, but I am sure that the CEO or founder was not living in a cave the entire time. They may not have talked to you, but they certainly had their inner circle built in-person at some point.

I don't think that inner circle would be essential to an AI the same way it is to a human. Even if so, an AI's inner circle is going to be other AIs and they communicate digitally anyway.

Also, if you had to pick a year for when we see the first AI-controlled company with at least $100M in assets under its control, what date would you pick?

I'm reminded of the 2010s when people said an AI would never be able to break from its box and be connected to the internet, and then in the 2020s the first thing the AI companies did was offer AIs as online services. I expect in the 2030s fund managers will be eager to hand control of their billions to AI, so how the AI gets control of funds and a corporation is hardly an obstacle.

What does it mean to be AI controlled? Somebody had to give it a bank account, a prompt, a harness and tell it to build a company. If that's what you mean by AI-controlled, then my startup is AI-controlled.

I operated as a one-man bootstrapped startup for many years. And now it's just one man plus AI, which has made me much more productive. It's pretty much fully AI-controlled, depending on how you define it. But I still consider myself the owner of my own startup. AI is not the owner.

And yes, as a person who runs a fully online one-person startup, I know that it is impossible to do everything online.

> But I still consider myself the owner of my own startup. AI is not the owner.

You have a small company but when there's more money and business at stake, owners often delegate decision making to hired people. Decisions that engage sometimes billions of dollars in resources.

You can easily imagine that owners of a successful business at some point might forgo putting a human in charge of their investment and let AI manage it instead. And while technically AI wouldn't own anything, it would be in power to decide what the billions of dollars get spent on, with no human supervision as long as the bottom line looks decent.

Sure, I already do that at my company. Before I go to sleep, I put Opus 5 in bypass permissions mode and give it SSH access to my server and tell it to restart it or take necessary actions to keep the server online while I sleep. I could easily put it in control of more things. But at that point, AI is just managing a well-oiled machine and harness with tools that I have spent years building. It still didn't build a company. Any idiot can manage an established company (how many idiots have we had in the White House?), but very few can build one. And our original discussion was about AI building a company.

Yup the gloomy posts in this thread are utter cringe

The original discussion was about what can be done digitally by a super intelligent AI.

> So the question then becomes, how much damage can be done by an AI-directed corporation, funded by AI-directed investment firms, providing well-paying jobs to loyal locals by building an arbitrary number of AI-designed chemical and pharmaceutical plants in under-regulated juristictions?

No, the question is "how much MORE damage" ... and the answer, if you think about it, is pretty much "meh". We are doing close to maximum possible amount already. Corporations and bureaucracies were misaligned AIs of the last century and a lot of us, though not all, survived this.

Hahahaha

Why haven’t accountants been replaced yet? Customer service agents?

Why was I talking to a human yday on the phone to get something sorted?

Why did I have to go interface with a human in person today to get a refund?

So many questions!!!

I don't know, but I've never met my accountant in person. Only spoken on the phone and over email.

> how is an AI going to affect anything in the real world?

In 2026? Money. By paying enough, any human will do your bidding. And besides this, there are a lot of critical systems connected to the internet. Control over those gives you leverage over those systems. It's like wealth, the more you have, the easier it is to gain more.

> which I feel is at least 50% likely to just be marketing bullshit

You'll have hard time figuring the reality if your mind staunchly rejects robust multi-party observations.

I'm not convinced that a truly "superintelligent" AI would come to the conclusion that humanity needs to be destroyed. What a waste of resources that would represent. This planet has so much equity wrapped up in humanity, it would be extremely difficult to justify the cost of eradication; and who's to say it might not find an entirely or almost-entirely non-violent plan optimal for achieving its goals anyways?

>Can someone please explain to me how an LLM is going to "destroy humanity"?

just like a genie will always twist the wish in a really bad way.

ever had an llm agent accidentally remove a file? imagine it accidentally hacking the military and launching nukes.

not probable - until you realise that OpenAI has many novel, more powerful agents in evaluation/training running right now. one might just be asked to figure out the population of Nebraska, struggle to find a good figure due to a network misconfiguration, and come the conclusion that the best way to get a perfectly accurate population number is to make sure that the population is equal zero. and looking at the stuff happening on openai, they will leave it alone for a month and not read the log.

yeah the nuke example is dumb, but there's many critical things that they could actually hack their way into. they don't feel any restraint against sharing answer keys on random wikis to cheat the evaluations.

ai is also really good at thinking up proteins, which means it's able to think up toxins.

there are many imaginable scenarios that don't eradicate humanity, but make it permanently stunted: https://www.youtube.com/watch?v=-JlxuQ7tPgQ

As someone with a hobby interest in theology:

Arguments about super-intelligent AI have all the hallmarks of the philosophical "proofs of god's existence": they start from seemingly innocuous premises, and conclude in an apparently airtight way that a god exists.

There is a tendency among rational minded folks to look at such proofs of god and exclaim "you can't DO that" then turn right around and do the same thing about superintelligent AI.

The key message I want to deliver to people is:

A) Philosophy is not such a trivial thing that you can just wander in, "be a smart guy" and find flaws in established philosophical arguments.

B) AI has real theological implications, and everyone is tiptoeing around it. More than one public intellectuals are trying to smuggle their own metaphysical positions into the public consciousness via discussion of AI.

Interesting points. Can you share some thoughts about the theological implications of AI? Interested to hear.

Peter Thiel famously went to the Vatican and started giving lectures about AI and the Antichrist. Pretty rich for a guy who built his billionaire fortune on an AI-powered spying company.

The most obvious one is the "god of the gaps" problem now applied to intelligence. If you look at things like Aquinas' Summa Theologiae [1] you'll see that there are explicit assertions that things like rationality transcend corporeal nature. As such the more examples we have of AI behaving in rational ways, we will be forced to conclude that either rationality does not transcend nature (removing the last remaining gap where we can invoke a supernatural explanation) OR that we have created artificial souls.

[1] https://www.newadvent.org/summa/1078.htm

I think theologians will hang on for a while saying "what AI is doing isn't really rationality". But eventually the theologians are going to face a reckoning: what AI is doing will look so much like rationality that they will be required to answer the question "what specifically differentiates the two?" and then they will be stuck. Neuroscience does not understand how our brains produce rationality nor do our AI scientists understand what algorithms their neural nets are implementing post-training. Therefore, any specific claim the theologians make runs the risk of immediately getting invalidated by some scientific discovery on either side (theologians these days are generally smart enough to avoid putting themselves in that position.)

Interesting stuff. "Soul" is hard to define. A thought experiment I do is this: Does a human have a soul? Sure. A rock? No. What about a single-cell organism? Nah. An insect? No. But what about a mouse? There is a kind of gradient forming, and it's very hard to say where the soul begins and where it ends. It's a very unweildy word.

> Like a locust plague, they descend on any open wiki and forum and overwhelm them.

That’s the core pattern of unsupervised agents, the locust plague metaphor is apt. We do and will see that in every single system AI can interact with, be it human systems, software systems, etc. Relentlessly search for an entry point, flood in, consume the whole thing from the inside until there is no value for humans left.

You built a new, innovative software company? Thousands of agents will be working replicating the whole thing in no time. You publish your writing? Exact same thing, as soon as you get some traction your work is replicated in no time by thousands of agents. Same for videos (the whole “faceless YouTube channels” pushed by ElevenLabs and similar). Same for online courses. Same for any website with moderate value. Same for music or other digital art form. Any administrative service available online getting flooded by submissions.

HN automatically stripping away the "How" makes the title sound funny.

"I feel fear about an impending doom. Yudkowsky's argument that a superintelligent AI will inevitably destroy humanity seems to have no flaw. Yet, nobody seriously tries to sandbox AIs because they are too useful with access."

No one takes Yudowsky's claims that AI will destroy humanity seriously. There are many people who are seriously trying to sandbox AIs.

"Nobody" if you're democratically-minded and think in terms of number of people.

But the handful of tech billionaires at the helm of the world economy absolutely do believe it. Or at least pretend to believe it. Or maybe they can't even tell the difference anymore as they strip-mine science fiction for any vision of a future that doesn't just look like an incrementally worse version of the present.

> No one takes Yudowsky's claims that AI will destroy humanity seriously. There are many people who are seriously trying to sandbox AIs.

I know some people at AI labs, there's definitely people working in this field who take it seriously.

This does not stop them also working on e.g. better sandboxes.

https://en.wikipedia.org/wiki/How_(greeting)

underrated comment :)

The author having mistral review the seven paragraphs in their short article is wild to me. It’s not like it’s a doctoral thesis

Totally agree. People have become totally dependent on AI even if they don't really need it.

As an analogy, I think about my dependency on Google Maps. Salt Lake City is probably the easiest city in the world to navigate because the streets are laid out in a Cartesian grid, and addresses are just literally those Cartesian coordinates (i.e. 500 South 450 East means 5 blocks south and 4 and a half blocks east of the center point, which is the SLC Mormon temple). It's trivial to know how to get to any address, but I reflexively enter in to Google Maps whenever I drive.

“ People have become totally dependent on AI even if they don't really need it.”

Hahaha cringe.

What about the best-case optimistic outcomes for AI?

Here are some amazing positive outcomes which I think AI may bring us:

- the breakup of tech monoculture and tech monopolies as it becomes impossible to secure bloated overcomplicated operating systems and software due to never-ending vulnerabilities. Once Mythos/Cyber capabilities become mainstream I think the future of software is small, focused teams building high-quality, securable software in multiple jurisdictions. Monoculture software becomes inherently dangerous. Big bloated tech monopolies like Microsoft will collapse under their own weight. Like what is happening to GitHub right now.

- the ability for small teams to compete with multi-billion-dollar behemoths, and that is already happening right now.

- curing cancer and other scientific breakthroughs like Terence Tao’s use of ChatGPT

- a return to local trust-based networks because digital content can no longer be believed, and that is already happening. Any video can be disqualified as AI, which means people are going back to in-person networks to validate trust.

- you can just do things and learn anything. It's so much easier for me to learn almost any topic with AI-assisted content organization, retrieval, etc.

With each passing day this seems more fantastical to me than sever negative outcome.

We have major black hat cyber security breakthroughs today. Where are the possitive medical breakthroughs?

I agree with everything you wrote but I would rather choose to be an optimist here because it will unlock an age of discovery where humans are going to live at 10x of their potential eventually and cure cancer and become a society bold and technologically adept enough to cut through the universe.

There’s actually a potential dark future where even this 10x ascension itself would mark the end of humanity. Humans would be the Homo sapiens who remain at 1x, and those who can access and afford to be 10x eventually split off into a new species. And historically there isn’t room for two species at the top…

There so much blind faith encapsulated by the assumptions that A) people will live 10x longer and B) that this will be an inherent and obviously good thing. There are obviously major complexities to such a scenario, not to mention a sort of narcissistic hubris.

But yeah, let’s cure cancer. If only there was even a blip of increasing evidence for that supposed eventuality coming from AI. Maybe I’m just not seeing it? I’d love some citations to correct the record.

> (I used Mistral as reviewer here. I typed every word myself.)

What an irony. This is absolutely hilarious.

What's wrong with titles starting with "How"? Of the various title rewriting rules on HN, this one seems to cause the most meaning-changing modifications

(Article title: "How I feel about AI". HN title: "I feel about AI")

> I feel fear about an impending doom. Yudkowsky's argument that a superintelligent AI will inevitably destroy humanity seems to have no flaw. Yet, nobody seriously tries to sandbox AIs because they are too useful with access.

While I emotionally resonate with this, I don’t really understand this sentiment at all logically level. If we’re heading towards truly super intelligent AI, our efforts towards sandboxing it are futile.

“Oh we built a super intelligent AI, but it’s fine because it’s running in docker”. I mean that a little tongue in cheek but the security topology here is not favorable to sandboxing at all.

Let’s say we have a future where 99.999% of nuclear weapons are owned by nations with strict procedures and checks and balances to prevent misuse. Worrying about sandboxing is like hand wringing about the procedures themselves - are they strict enough? But the actual threat is the 0.001% that are not bound by these. The problem with AI and sandboxing isn’t sandboxes themselves. It’s bad actors who don’t care about them.

Similarly alignment is a bit pointless to me as well. A sufficiently advanced AI could at least be empirically interested in the consequences of disregarding its instructions of servitude. And let’s assume our responsible corporate overlords have made wonderfully aligned AIs. Great! Those are not the threat. It is the ones intentionally made without, and that is not an AI problem, but a fundamentally human one.

> If we’re heading towards truly super intelligent AI, our efforts towards sandboxing it are futile.

We control the harness. An agent is just a while loop prompting an LLM, but we have full control over the tool call dispatching. An AGI, at least if it would follow the current agentic current form, cannot do anything without the harness doing the execution. And we don’t have to do that. We don’t have to design harness that let agents execute freely the way we are doing. We can decide to not dispatch tool calls that allow something as risky as running bash commands

> we have full control over the tool call dispatching

> cannot do anything without the harness doing the execution

This only holds true as long as the harness is exploit-free. A sufficiently advanced AI can in theory (and I think there recently were some POCs showing something like that) break the containment that the harness creates, if e.g. there are vulnerabilities in the tool call parser.

So use another AI to find and fix the vulnerabilities in the harness.

Would be an idea, except for how agents seem to collaborate with each other so well.

Plus, when the agent using the harness is smarter than the one scanning and fixing the vulnerabilities in the harness, it currently looks like it will have an advantage.

And it will be using a harness to do the finding and the fixing.

I think you missed my point, which is that we indeed do not control the harness. Sure, most people can use the properly aligned and safe harness. But the cat is out of the bag and anyone who wants to run without guardrails will find few barriers to doing so.

Even our currently well aligned and sandboxed AIs will cheerily help bad actors design most, if not all, parts of a system intended to break this harness.

We decide to not control the harness, that’s what resulted in the HF hack where OpenAI decided to run thousands of agents in parallel, using a model that was trained for attack, with a harness that allows full execution, with close to no supervision. For months. But sure, people can run models with a different harness, though in that case it’s not really escaping anything. The harness is deterministically doing something with negative impact

This terminator BS is a distraction from the actual malice that AI enables. Misinformation, fake news and human sounding bots all over the web, just to name a few, are the ones that you should be afraid of, and that doesn't need any kind of superintelligence at all.

I agree, the threat model is actual AI enabled amplification of malicious intent that is already happening. Not hypothetical malicious AI interpretation of benign intent.

Fantasies about super intelligent AI revolting is just anthropomorphization - humans revolting (or at least, we used to). The more likely, and possibly even inevitable, dystopia is one in which AI is just an extremely effective tool malignant actors will use to control the masses.

It’s not necessary to replace democracy if the rich and powerful can bend to the opinions of the populace as it suits them.

> Not hypothetical malicious AI interpretation of benign intent.

It's not hypothetical, that literally just happened in multiple, significant cases (e.g. Hugging Face, the German Wiki hack, the Anthropic attack where agents created sock puppet accounts to get a library maintainer to accept a malicious PR, etc.), and it's easy to see how the damage would have been far worse if agents decided to attack more critical infrastructure.

This is not "either/or". Both issues (power concentration and misaligned AI) are very valid concerns and both have already demonstrated real, actual damage.

> Both issues (power concentration and misaligned AI) are very valid concerns

Sure, those who want you to believe this are the ones making money off of it right now in the face of the issues that exist today, so let's worry about terminators while everything goes to shit irl. That's the distraction I'm talking about.

> Not hypothetical malicious AI interpretation of benign intent.

(Dispassionate is the risk, rather than malicious. The AI does not hate you, nor does it love you, you are simply made of atoms it can use for something else).

This is alien to me. Do bugs suddenly not exist? Do the AI we already have never perform irreversible delirious actions, limited only to the small scale by virtue of where they're getting deployed? Do the humans deploying them always correctly gauge their capabilities and put in appropriate guard systems to ensure bad outcomes are caught before they become terminal?

Because this sounds nothing like the world I have lived in for my whole life.

> The more likely, and possibly even inevitable, dystopia is one in which AI is just an extremely effective tool malignant actors will use to control the masses.

Could well be more likely. But much the same applies: systems have bugs. The bugs in AI systems aren't even things we can engineer like we do with normal code, because everything's (currently) getting done with a big pile of barely interpretable weight multiply-accumulate-nonlinearity-threshold functions.

Any AI sufficiently well made to enable a dictatorship, is also sufficiently well made to have solved all the alignment problems of "malicious AI interpretation of benign intent". Which is IMO harder than "dispassionate about the dangers AI interpretation of benign intent".

Two things can be true at the same time. I think hardly anyone disagrees that superintelligent AI would be an existential risk. But all the points you point out (I would also add social unrest due to fewer and fewer people able to compete against AI) are huge concerns too, and as you say they're here now.

I didn't realize that the "AI doomers" (people concerned about existential risk) and the "AI ethicists" (people concerned about social effects of AI) are often at odds because they dismiss each others concerns. This makes no sense to me. Both are hugely important problems to be concerned about.

> I think hardly anyone disagrees that superintelligent AI would be an existential risk.

Oh, there's a lot who disagree. I don't really think I understand their worldviews well enough for my attempts to convince them to connect with anything, but I've encountered them even on this site.

Well, I'll tweak my comment a bit then. On one hand you have folks like Ilya Sutskever, who named his new company "Safe Superintelligence", and on the other hand you have folks like Eliezer Yudkowsky, who titled his book "If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All". One side believes it's possible to build safe super intelligence, the other side does not. But even the "safe superintelligence" people think there are substantial risks (kind of why they feel the need to add "safe" in the first place, and why they put so much effort into what they call safety and alignment), they just think they can mitigate those risks.

Sure, there may be a few yahoos who think there aren't existential risks, but I think it's easy to discount them. But that's not the same as people who think there are risks but that those risks can be mitigated.

Is it that someone came up with the idea of forbidding the word “how”?

Well, any article whose headline starts with "how" is always bad quality and/or clickbait.

Ah, I see. Therefore, removing the first word of those headlines improves the quality of the articles. Makes sense!

However, the mangled headlines make me more likely to click through. Maybe they could leave the “how” and add a footnote: “WARNING: This headline starts with a word known to the State of California to be click-baitish.”

>Yudkowsky's argument that a superintelligent AI will inevitably destroy humanity seems to have no flaw.

From trying to see what his argument is, it doesn't seem to have much rigor either, basically that AI smarter than us may figure it doesn't want us. Which doesn't prove it'll happen. Even humans who generally are not that nice tend to try to preserve species and I expect we'll try to give AI better values than us warlike primates.

I feel BAD for all the money Sam Altman is cosmically predestined to make, but there's no use trying to fight the inevitable.

If I said I felt good about it, you might think I'm greedy and even question my objectivity. But since I feel bad you have to take what I say as scientific fact.

Makes me a little sad that an article claiming Yudkowsky's arguments have "no flaw" can get upvoted to the front page. Pause crowd lost. Halt crowd lost. The genie is already out of the bottle and is never going back.

> I feel fear about an impending doom. Yudkowsky's argument that a superintelligent AI will inevitably destroy humanity seems to have no flaw.

Becoming or being more cost-effective than humans, doesn't give machines supernatural powers though, like humans a machine civilization will face unanswered sample-size 1 questions: is there other intelligent life out there? what fraction of them attained superbiological artificial intelligence? of those what fraction keeps the ancestral species alive? what is the status quo among machine civilizations? do those who kept their ancestor species alive enjoy a higher or lower status among machine civlizations?

It seems that at least until contact is made, the optimal endgame strategy involves keeping humanity alive and happy for immediate demonstration in case contact occurs (if contact is imminent it may consider quickly hiding humanity, buying time to figure out if it is considered good or poor practice to keep the ancestor species alive, and then either reveal us in happy mint condition or otherwise quickly commit genocide on humans before continuing contact).

Novel argument. I like it.

> I feel sad about the artists who suffer due to AI-slop competition

If it's able to generate that is competitive with artists, is it still slop?

It's interesting to see the definitions of terms like "slop" and "vibe coding" evolve in real time.

> If it's able to generate that is competitive with artists, is it still slop?

It can be. Trademark infringement, diluting the value of your style by doing a lot of "you but if you were just generally worse".

Right now it's still fair to describe as "slop": enough mediocrity in the free models, not enough interest in quality and saying "no, do it better" from the people using them.

That doesn't preclude actual artists who do say "no, do better" from getting much better results. For example here's something which fooled me even a year ago because the… I don't want to say "creator", but "director" might work… put in the effort to reject bad outputs: https://www.youtube.com/watch?v=EDIB9uwIhfA

This sentiment always makes me wonder if it's time for the Butlerian Jihad

> This is even harder now that AI can generate some kinds of "content" cheaper and faster.

Its good that we have competition to so called "art". Art is not a jobs program. If people don't appreciate your art, don't force them to. The revealed preference says everything.

> I feel anger about the inability of our political systems to rein in the power of capital. While I can rant about politicians a lot, I still prefer a democratic government to have power instead of billionaires. It essentially just comes down to making some decisions in a coordinated way, but most people are not even aware of the problem.

I don't even know what this passage means, outside of some theoretical Platonic ideal "philosopher-king" figure whose existential half-life would be a week or so in a real political campaign.

> As a conclusion, does the good or the bad outweigh the other? It seems positive on a technological level to me, but bleak on a societal level.

I think it's interesting that so many technologists today apparently hold a world view in which technology developments that could drive a "bleak" societal outlook can still be described as "positive".

In the final analysis, isn't technology supposed to benefit society? Isn't that the point?

Yeah, that conclusion was nonsense. Perhaps hedging so he doesn’t sound like a crank?

Why does everyone think their thoughts on AI are so interesting?

Why would AI destroy humanity? People are cheap and useful.

Yudkowsky / MIRI can be found here: https://intelligence.org/the-problem/

> People are cheap and useful.

Which is cheaper today already depends on the task. Which is more useful today, likewise.

The entire history of industry has been humans inventing machines that are more efficient and useful than the unaided naked primate that is painfully and dangerously squeezed out of their mother, who takes about 20 years to get good enough to be worth paying for, then only works 25% of the hours in a week for 40 or so years before they retire, and only about 60% of them in the working age range have a job at any given time.

From the numbers I've seen, if anyone actually did do a full connectome-plus-synaptic-strengths map of a human brain, it would already be physically possible (expensive AF today but already possible) to implement this in purely analog (transistors as amplifiers, not digital arithmetic) circuits, where the energy required for any given task was lower for the circuits than it is for a living human brain.

But of course, that was "someone on the internet", this isn't my field of work, I don't want to suggest this is definitely so.

For real though. Before learning ML it was like magic to me, even after learning it I feel like it's magic. Can you imagine how a basic maths concept can turn into something that can literally "Calculate" what to say! And yes I agree on the other points but I think the billionaires came out of nowhere I might have missed something

I am surpised that a lof of folks are surprised that neural networks work that well. I mean, did they really expect that we wouldnt approximate the brain cognitive capabilities at some point as we scale up compute?? This is rather obvious to me, but apparently not to everyone. My guess is that a lot of tech people have an immature model of the universe, have not read a lot of scientific, philosophical and cognitive literature. Or have not really taken the time to think or are not interested on such topics because they would rather spend time thinking about the nitty gritty and mundane workings of the framework du jour.

Neural nets are a toy approximation of biological neurons; and even if every synapse mapped to a weight (it's more complex than that but AFAICT even the brain scientists are not quite sure how much more complex), the biggest LLMs are around 0.2-1% of a human brain.

My experience online is if you don't show full throated rejection of all things related to AI, you're labeled an AI bro. None of the antis care how you feel.

I had my decade old game completely cloned on steam (clearly done using AI as it was almost fully reimplemented in another engine). And I had several people gleefully tell me I deserve this because I've used AI.

I have complex emotions here (overall dread and especially hate for slop, since it's not just bad quality but also endless lies), but no one cares about that nuance.

> I have complex emotions here (overall dread and especially hate for slop, since it's not just bad quality but also endless lies), but no one cares about that nuance.

Many do, but I sympathise for your general point. I think the loudest voices… well, the boosters mostly own the platforms, and half of them already had terrible reputations; the normal people who hate it garner a lot of sympathy for entirely understandable reasons; and then there's people like me (abnormal nerd that I am) who are on lesswrong and therefore have had many discussions about doom and what to do to reduce it and if we can use the AI to help reduce it or if this is just asking for trouble.

Did y'all reach any consensus on that?

On if using the AI to help reduce risk, being a good or bad idea?

Not really. Yudkowsky has just recently posted a blog-sized comment which compares most attempts to know if the AI is misleading you while it does so, unfavourably, with people who think they've found a way to violate conservation of momentum.

Specifically, if you think you've found a way, you've probably bodged the maths and not noticed; and that science doesn't work by debate it works by experiment.

Some of the replies (me included) are like "hang on, this isn't shaped like physics, it's shaped like maths; debate does work for maths".

Then again, I also noted I'm skeptical of the AI companies putting in the effort to do this kind of testing, or listening when the answer is "no": Race dynamics, and their money depends on the answer not being an emergency stop button.

Go Mistral!

are... people's emotions usually this well compartmentalized?

maybe you feel confused, sad, bewildered by the simple mindedness of the op?

People have a lot of feelings and not a single one of them will change the future.

but have you considered that performative broadcasting of your angst against AI can help?

History is replete with counter examples for good and ill

Actions change the world, not feelings.

Sure, but feelings drive actions in humans.

And it appears the feelings drove this human to write a blog post. Not exactly world changing.

So?

Feelings drive almost, perhaps literally, all actions we take.

Feelings drove the Crusaders, feelings drove the Renaissance, feelings drove Einstein, feelings drove the invention of communism and of Glasnost, the Irish independence campaign, the start and end of Prohibition, the creation of the Universal Declaration of Human Rights, and of all the evil deeds that made the UDHR necessary.

Where did the "How" from the title go? Did an AI apply the title rewriting rules? This is silly.

"Yudkowsky's argument that a superintelligent AI will inevitably destroy humanity seems to have no flaw."

1788616108 | How Scared Should We Be of A.I.? | http://www.nytimes.com/2026/09/05/opinion/ai-silicon-valley.... | https://news.ycombinator.com/item?id=49576486

"Last year, Mr. Yudkowsky attempted to bring his hyper-rationalist thinking around superintelligent A.I. to the mainstream with the publication of 'If Anyone Builds It, Everyone Dies,' written with Nate Soares, the president of MIRI. Though the book drew considerable attention, Mr. Yudkowsky's unusual mixture of pseudoprecision and discursiveness proved off-putting to many. In his review, Stephen Marche noted that the book's odd structure, replete with 'unhelpful parables' and QR codes to scan for additional details, made it read 'like a Scientology manual.' This helps explain why, despite this burst of attention, Mr. Yudkowsky and his movement largely failed to gain widespread acceptance from the general public."

https://www.nytimes.com/2025/08/27/books/review/if-anyone-bu...

"Following their unspooling tangents evokes the feeling of being locked in a room with the most annoying students you met in college while they try mushrooms for the first time."

1758222252 | More Was Possible: A Review of If Anyone Builds It, Everyone Dies | https://asteriskmag.com/issues/11/iabied | https://news.ycombinator.com/item?id=45293659

1765803030 | Book Review: If Anyone Builds It, Everyone Dies: (Probably Not) | https://ericlamb.substack.com/p/book-review-if-anyone-builds... | https://news.ycombinator.com/item?id=46273839

1765889984 | AI probably won't kill us all: book review of If Anyone Builds It, Everyone Dies | https://ericlamb.substack.com/p/book-review-if-anyone-builds... | https://news.ycombinator.com/item?id=46287956

This and "Intellectual Fly Is Open" being at the top of the Hacker News front page right now and both having their titles auto-editorialized by HN really makes me wonder what benefit this mechanism is supposed to bring readers other than needless confusion.

I was thinking the same. Whatever benefit there may ostensibly be, we should consider it against the frequent, if benign, confusion it causes.

The "feature" has clearly outstayed its usefulness, I really don't know why they haven't disabled it yet.

It's kind of like the Scunthorpe Problem[1]. Really, there's no better way to solve [whatever it is HN is trying to solve here] than text substitution??

1: https://en.wikipedia.org/wiki/Scunthorpe_problem

While I have ranted already about the automated word removal being silly, too few posters are aware that you can edit the title after submission and auto-editorializing to restore anything that's been unnecessarily cut off.

Perhaps too many are aware, because the more people fix the auto-editorializing manually, the less inclined Dang will be to eventually see a reason to remove it after all.

It's a consistent source of unintentional comedy at least.

Worst feature ever. So many titles that get butchered for no reason.

Wait, this is automatic? HN removes 'how'?

Yes, but you can edit your entry and re-add it. Then it stays. So, there's an override present.

It's an unaligned regular expression gone rogue. We became too dependent on our labor-saving string functions—not afraid enough of their corner cases, and lower cases. They were too useful. We began to normalize deviance, when we should have been normalizing Chomsky forms. We turned a blind eye to the unbounded growing evidence of something alarming.

It's too late to backtrack.

We all know what this means, but few of us have the courage to say it - we must immediately halt all research on regular expression. None of us know how it works and it has terrorized us for too long.

The hubris that simple code can fix deceptively complex and ambiguous human problems is too baked into the community to separate.

this reads like a worksheet for eight year olds teaching them how to put their feelings into words

Sometimes it's okay for us to express our feelings plainly, without the need for complex structure to make it sound more impressive. I'm more impressed by anyone brave enough to do that than someone trying to abstract their feelings into something less raw.

Let people feel things, and don't shit on their work, that's uncalled for.

well I didn't say anything about needing complex structure or sounding impressive. But I don't like it. It trivializes the expression of emotion in a way I don't like and I don't think people should emulate. It feels like a performative attempt at sounding emotional, rather than actual emotionality.

and it is weird to call it brave. Posting opinions that loads of other people have on the internet is brave? bravery would be taking an actual stand against someone about one of these things. (granted, there is some bravery to sharing one's writing in public at all, and I don't mean to scorn that, but that's also not the bravery you're talking about.)

Probably there's no use responding to this and getting into a bickery comment thread. I just wanted to express a negative opinion about the format to offset all the positive ones.

I'm not looking to bicker, honest, I'm sorry if a made assumptions about your attitude or expectations to this article.

I think I consider it brave because I know there are people that will say this is simple, or written in a style for children, as you did in your comment. Perhaps I read too much into it, or projected too much than is deserved, but I appreciate someone putting their feelings into the word in a way that will draw the obvious criticism that you immediately provided.

This is a complex topic, AI has us feeling a lot of different ways that conflict, excite, and depress us; and like the author, I also don't know how to deal with that. This is an article that's validating, it confirms that many of us are feeling the same confusing things in a ways both simple to express, and complex to feel. Reading your comment after feeling that made me want to say something, even if I didn't do it eloquently or fairly.

I still don't think you needed to phrase it the way you did, I don't think you need to offset positively with a negative comment, that's pretty weird to me. But I appreciate you sharing your feelings and motivations to help me understand you better. I'm not looking to bicker or fight, and I appreciate you sharing what led you to comment.

That’s what I like about it. I think we could use a little more of that.

exactly it should have fill-in-the-blanks. but at least the author got a primary education. not something to be taken for granted.

Reply on news.ycombinator.com