Hacker News Reader

I'm quite concerned about climate as well as our other horrific environmental problems, but there can be more than one giant existential risk at a time.

I'm not concerned about today's AIs. But three years ago they couldn't add numbers accurately and now they're doing world-class mathematics. If it gets much smarter than us, then it will run the world economy, because anyone who doesn't use it to run things will be outcompeted by those who do. At that point we'll have lost control.

The way I see it we already have lost control. Hell, maybe we never had control in the first place. We've known about the effect of greenhouse gases for over a century and we still keep investing into fossil fuels. Because "economics".

If we had control we would have looked at the situation and gone balls deep into nuclear and other green alternatives many decades ago.

Some people certainly have been diving into nuclear & green for decades, and we now have far more wind, solar, and hydro power than we did several decades ago. Nuclear has been flat for only 2 decades and is about to start growing.

“We” certainly have had control the whole time, “we” have been consciously creating greenhouse gas emitting machines while knowing the consequences. It is the economics. The delay in environmental consequences combined with the immediate financial benefits of easy power & transportation made the choice too tempting for some.

The kind of control you’re talking about is self-control. We’ve always had the option to stop our bad behavior, and we chose not to. And we still have the option to exercise self-control with AI. The question is whether that option might disappear soon. Can AI get good enough that we will actually lose the choice to regulate ourselves?

> "we can just turn off the data center if we need to"

1. If it never happens or AGI is friendly then we won't need to. If a hostile AGI is created, one trained on all writing about war and strategy and deception, able to respond in microseconds, with has ample time to plot before it makes an obvious move, then thinking you will wait until you see it attack and then turn off a datacenter, is way too little, too late. Like saying "the nuclear bomb explosion is local to one place, if it hurts, we will just walk away from it".

2. That claim does not hold water once you start to dig into who, specifically, would turn off which specific billion-dollar datacenter, under which conditions. Google has been moving compute jobs around the planet to get nighttime electricity rates for bulk compute jobs for years, the idea that there is "one datacenter" is far too simple. The idea that employee 74,123 at Meta doing AI research is also able to get a Meta datacenter shutdown quickly on request, or would know enough about Meta's infrastructure to know which datacenter, seems unlikely. The idea that it even has one off switch, or that nobody will quickly power it back on, or that the compute jobs will not pick up automatically somewhere else, or that nobody else on the planet will be at a similar point in their research, also seem important.

3. This assumes there will be a simple critical point that makes it clear we need to "turn off datacenter now". The fact that you then go on to talk about climate change, imagine you had said "we can just turn the factories off if climate change becomes a problem" and maybe that will make it clearer. Who will turn which factories off, at what point in the climate catastrophe, and why haven't they? That's why it won't happen with datacenters either.

> "Like we're not going to have some runaway skynet thing, that's just completely off the table."

You're confident that tens of billions of dollars of investment and research, US v China national AI competition, some of the largest companies in the world researching AGI will be a flop and develop nothing? Despite the progress from GPT2 to present day, despite historic analogies like the Wright Brothers to the Moon Landing in one lifetime?

NVidia shipped an estimated 40 million GPUs in 2025. Say they average 1 TFLOPS each and a year has 31 million seconds, that's 1.2 million million quadrillion calculations in a year. That much compute power piling up around the planet year-on-year is not going to be important or relevant to the near-medium term future? Because it's just "off the table" that anyone could invent anything interesting with that much growing compute in the next ten or twenty years?

They get electricity through cables. You nick the cable, end of story for the datacenter. Dataceters in space, welll.... not a good idea. When AI systems gets hooked to humans via brain interfaces and human like robots, well we'd better start worrying.

That doesn’t address any of the points I raised about who would make the decision, or about compute jobs moving to another data centre, or about a dangerous AI deceiving that it was stupid until it had a plan to account for that, or an AGI responding much faster than humans, but it raises new points - how do you “nick” a 20 MegaWatt power feed without dying, and what do you do about the batteries and generator backup in a secured area?

Now are you imagining that the employee who can’t ask for a data centre to be shut down, can call the military and telling them a self-improving AGI is about to destroy the world and they need to attack an American company?

People in need are usually creative in their ways...

> What are you worried about? All the worries seem like bullshit to me. Like we're not going to have some runaway skynet thing, that's just completely off the table. These things run on data centers, we can just turn off the data center if we need to. I don't really see that becoming necessary either but we could.

We can't even turn off the coal plants when the owners specifically want to because it's not making money, in part because the President thinks Global Warming is a hoax: https://www.utilitydive.com/news/washington-earthjustice-sue...

Also, there's a whole bunch of companies pursuing space-based data centres, which I think are dumb for a lot of reasons, but not quite dumb enough to not be a threat in this kind of manner as a few thousand GPUs is entirely doable before anyone decides it's not value for money.

Also, we demonstrably have useless-to-just-bad monitoring in place, which is how all the things on https://www.felonybench.com happened. That the companies could have done better, and should have done better, does not change the reality that they in fact did not notice until after the events.

Also, there's at least two companies putting these things in control of wet labs. Good news: they're currently too bad at this to pose a threat. Bad news: "currently" is doing the heavy lifting.

Also, there's a lot of people who are blindly using agentic AI and not checking anything, despite all the advice and case studies about why this is a bad idea and to not do that.

Also, there's a lot of people e.g. on Hacker News whose response to all the attempts to create guard rails on the AI's behaviour is to call this "censorship" or "a lobotomy"; and also a lot of people whose response to calls for regulation of the companies is that this is "regulatory capture". (Worse news: The US president has rejected calls to slow down AI, saying "whoever wins AI, wins").

> And then there's fears that someone might use it to engineer a super virus or something like that, which isn't really a new worry. People could do that before too, but it turns out creating a global epidemic isn't really a thing very many people want to do, and it's harder than most people imagine.

The number of people able to do so has likely gone up by a factor of 10x or so in the last decade. AI's ability to use tools on behalf of their users is a further amplifier on this, unless safeguards are in place. Bad news, "safeguards" is a pretty vague term, and vague platitudes are not enough.

More worrying is the possibility that some idiot will set biology challenges similar to the cybersecurity challenges which led to HuggingFace, with similar lack of oversight, and the agents will do much the same thing but with DNA/RNA-on-demand services. I really hope their equipment is better secured than HuggingFace was.

Good news: the agents in the HuggingFace incident did not act with what would be called destructive "intent" if a human did it. Bad news: they still did the things.

> "AI" are just chatbots. They can't fabricate robots to take over the world, in fact they have very limited reach into the real world altogether.

No, they're not "just" chatbots. They're also tool users that can see.

Good news: yes, they can't fabricate robots. Bad news: Guess what Musk is trying to do. Good news: Musk's timelines are wildly overoptimistic.

> Sure maybe you could envision some kind of malicious hacking, sabotaging some industrial control system to create a disaster of some kind or whatever but that was a worry before AI too which is why most critical systems like this are air gapped and have safety mechanisms.

Bad news: We're really bad at properly air-gapping stuff. We only know about Stuxnet because it breached an air-gap. This kind of thing happens a lot, even for really important stuff.

> So I really just don't understand how people think AI will destroy the world.

Ultimately, all it needs for this is to do what it was told, instead of what was meant.

All humans acting perfectly would prevent all the bad outcomes I can think of, but (1) that never happens, and (2) "that I can think of" is doing a lot of heavy lifting there.

> Yeah I don't know why people are panicking about AI, if there's anything you should be panicking about it's global warming.

People can care about more than one thing.

Look, you're worried about climate change: AI appears to be quite effective at convincing people of stuff, just consider how bad it would be if all the people who get rich from greenhouse gas emissions outspend environmentalists on AI-driven PR/marketing.