Comment by LogicFailsMe | Hacker News Reader

Comment by LogicFailsMe | Hacker News Reader

Parent Full thread

No, they do not. They are both bullshit campfire ASI horror stories written by nontechnical sorts in the wishy washy invented field of AI safety. For example, I want someone with a proven track record in the field of robotics to explain the entire pathway from impressive 2026 Chinese robots to an entire self-assembly food chain from raw ore to von Neumann replicators that can travel to the stars and reproduce themselves, not just assert that ASI will make that happen among so many other unlikely events like curing cancer because reasons because all any of this takes is superintelligence according to these sorts.

https://garymarcus.substack.com/p/the-ai-2027-scenario-how-r...

But no one likes hearing from the engineers on this, I get it. We are simply no fun. So many would rather believe a time traveling superintelligence will punish a simulated avatar of them for not making the ASI happen sooner. And even if that were true, why am I supposed to care about what happens to a simulated avatar of me?

None of this makes any logical sense whatsoever.

Ha well, by definition what you want can't be provided!

Of course these stories can only be technical up until the recursive self-improvement part. Then they're necessarily fantasy, as the AI is more intelligent than the story author. Unfortunately, this doesn't make that scenario impossible.

However, we know now from Hugging Face that even without the later sci-fi powers, the LLMs are already capable of causing damage.

It doesn't need them to be able to make Von Neumann probes to e.g. steal their own weights from badly secured OpenAI servers, hack into various Neoclouds, distribute themselves, make Teslas crash into things, hack into all our power and water infrastructure and collapse global civilisation.

Explain how rogue FSD cars take down civilization. And I say that with the opinion that the only good scenes in The Fate of The Furious and Leave The World Behind were the ones with the rogue Teslas. But also, nothing new under the sun really:

https://en.wikipedia.org/wiki/Maximum_Overdrive

And so granted we could probably get a global 9/11 in deeply urban regions out of your scenario, we then bomb the datacenters and we finally address the horrific tech debt in our infrastructure blissfully relieved of the option of continuing to ignore it. Don't think this is happening either BTW but I acknowledge its probability is slightly greater than zero.

To that end, my house is entirely off-grid. And in the event of what you call the end of civilization but I call the big burp inconvenience, I've made friends with my neighbors because community is what really matters during a disaster, and I've been through a few.

>during a disaster, and I've been through a few

Eh, about the biggest one I could see AI causing is the need for a black start on all 3 major US grids. Millions will die before it comes back up, well, if the US pulls out of it at all.

And in that case, thanks for setting up a nice place to live for someone far more ruthless than you. They'll snipe you, and enslave your women and children if things have gone that far downhill.

Or, we could avoid all that by making sure AI labs are properly regulated.

Such an individual would get a quick corrective lesson in the second amendment from my friends and neighbors. But once again, hyperbole. The US grid goes down. That sucks for up to a year and a half.

But hospitals and other critical organizations have generators and the national guard would be deployed to keep them fueled. And yes it will crash the economy, which is kind of a myth anyway. Buy solar panels and batteries, live off a well, and get to know your neighbors if you think this is likely. Also consider being like the Mormons and storing a 6-12 month supply of nonperishable food.

Regulating AI achieves nothing but to hand the 21st century to China. Regulating its deployment OTOH will not be allowed by companies like Meta and Palantir whose gross margin is reliant on it. I'm favor of regulating deployment. The research not so much.

> None of this makes any logical sense whatsoever.

It doesn't make any logical sense to set the bar as as you have, either. I mean if someone could provide the roadmap you're asking for, it's probably already too late.

And while we're talking about sci-fi, it seems fairly likely that an AI that could "kill us all" could fail to build a "entire self-assembly food chain from raw ore to von Neumann replicators." I'd be a civilizational collapse [1] without even birthing something new.

But it makes far more sense to worry about the AI x Late Capitalism crossover story or the havok that could be caused by some immature tech nerd amplified by AI, than far out sci-fi stuff.

[1] IMHO, even a genocidal plague probably wouldn't render humanity extinct, something like uncontacted tribe in a rainforest somewhere would probably survive and humanity would carry on (from square one, in a resource-depleted world).

Fort Detrick and likely its Russian, Chinese, and possibly Indian equivalents have had stocks of bio agents that could give a humanity a very bad day for decades. I'm far more worried about those existing stocks than I am about a tech nerd trying to DIY the genocidal plague.

Which is to say it's already a threat, it's unproven but not entirely impossible COVID leaked from a lab as just one example, but Ebola suggests that it requires more than a rogue nerd to spread a deadly virus global because once people start dying in the streets, the rest hide because they don't want to die. COVID spread because it wasn't deadly to 99% of humanity and now most of the 1% vulnerable to it are either dead or permanently crippled by long COVID so it's ambient now.

I worry about attacks to the infrastructure, but I just don't buy that as a direct route to civilizational collapse.

Reply on news.ycombinator.com