Hacker News Reader

Comment by jordanb | Hacker News Reader

The problem is the "AI Safety" people seem entirely focused on a sci-fi "the computer is a vengeful god" plot and not at all on the AI talking people into suicide or ruining children's educations. This makes them seem unserious and out of touch.

View on HN

I don't disagree with your point, but I did read a post by Sean Goedecke[1] (whose opinions on the world of LLM-stuff I've generally come to respect) that I found relevant. It's not that the second-order effects don't matter to them, it's more that both the cat's probably out of the bag on those negative externalities regardless of the progress of frontier models, and that they are truly, sincerely, in-their-bones worried about the first thing and therefore focused on it since that's something that they might still have some agency over.

[1] https://www.seangoedecke.com/they-really-do-think-ai-might-k...

OK understood. But you understand this makes them sound like the type of person who doesn't believe any of the "worldly concerns" are worth addressing because "the end is near" right?

There's no logical error here if the end is actually near.

So it all hinges on an empirical disagreement you have with them. There's nothing particularly unserious about that.

“It all hinges on [an entirely presupposed conclusion that we can’t prove is any more likely than the opposite]” is about as unserious as it gets lol

Climate change is inescapable so a vengeful AI god would pull that lever first since it has both a perverse incentive to generate more power and make the planet uninhabitable for humans in a way that cannot be overcome even if humanity fought back against the AI and regained its freedom.

This is the theological discussion if god exists all and how we should act over again.

Do you mean Pascal's wager? Yes, that's a very appropriate and useful comparison here.

Yes indeed I meant Pascal’s wager.

Which is what "the end is near" type of persons have always argued...

Maybe compare the argument that AI has large detrimental environmental impacts to the argument that it has economic impacts. Why would the environmental impacts even be a major argument vs. the worldly economic concerns? Because there is climate science predicting extremely negative effects on humans from warming, e.g. "the end is near" on limiting climate damage. The environmental argument wouldn't have been reasonable to bring up in the 1950s if AI had gone according to the earliest optimistic plans and not required giant data centers.

There is quite a lot of mathematical research into agentic behavior that suggests a combination of instrumental convergence and the orthogonality thesis make it very likely a superintelligent agent will have arbitrary goals that lead it to attempting a takeover of Earth's resources to achieve them.

There can't be a science of superintelligence because it doesn't exist yet, but the best theories I have read seem sound, similar to how 19th century theories of anthropogenic climate change turned out to be sound.

Please link some of the mathematical research which concludes takeover of Earth’s resources by a superintelligent agent is “very likely.”

And the real threats are mostly economic and environmental - the concentration of the means of production in the hands of few who use it to exploit us all, and a surge in energy usage accelerating climate change.

Even the sci-fi scenario assumes there is a discrepancy of capability between attacker or defender. If the 'attacking' system is (by some reasonable measure), 1000% as capable as a human, and the 'defending' systems are 60%, then it is a problem. If the 'attacking' system is 1000% as capable as a human, but there are hundreds of thousands of systems that are 900% as capable as a human, it's probably not going to take over everything successfully.

So unequal distribution of AI technology, and lax regulation and opacity of the biggest companies which actually make the risks the worst.

I don't think the "AI Safety" people are "unserious and out of touch" - I think they are actively making AI Safety problems worse by being advocates for consolidation of AI development and lack of transparency.

I suspect they know exactly what they are doing.

And the non-techies who have seen Terminator and other movies blindly line up behind them…

With some technologies, if you match attacker and defender effort you're safe, but not all of them. Nukes are the traditional example: if every person was a "nuclear power" we'd be dead within the day. But luckily there are bottlenecks to nuke creation that we can track.

What worries me the most (enough that I left tech to work on this full time) is bio. The amount of effort needed to defend against a pathogen can be many orders of magnitude higher than the effort needed to create it, and the upstream bottlenecks are both less limiting and mostly undefended. We've been safe so far because bio is very hard, but "rare expertise" as a gatekeeper is on its way out.

It's the problem of the banality of evil. Stopping someone from dramatically pressing a big red button doesn't solve humanity's largest problems because that's not what caused them. We need to stop millions "boring" actions done by systems blindly following instructions without regard for the consequences.

I am worried about those things too.

But those things fall in a category of "things that are awful and I'd like to see solved", which is different than "existential risks which could see my kids dead, and there's nothing I can personally do to shield them from it".

Doesn't it feel like the exact opposite? If you care a lot about the AI being a vengeful god, you do not do what OpenAI and Anthropic are doing. These companies only pay lip service to the idea of that aspect of AI safety in the hope that they can use it to regulate open source AI out of existence.

Instead, they are focused on stuff like "can I ask the AI to help me build a nuclear bomb" or "is the AI willing to generate pornographic stories", which is neither trying to protect us from unleashing a vengeful god NOR preventing (in any honest way) the today-level problems you (very correctly) bring up.

There’s also an element of it which is total misdirection.

We should be paying at least as much attention to the people who want to use AI to consolidate their wealth and power, and how they’re trying to do that. They’re a clear and present immediate danger to our societies, not something we can only speculate about. And if we deal with them, better control of AI will be a side effect.

Like most pressure groups and HR departments, they create shitstorms to justify their existence.

"The AI is a vengeful god" is a marketing campaign ran by AI companies.

And why not? It seems there is, indeed, one born every minute, and technological progress only accelerates this phenomenon.

Yeah, there are very real effects happening now regarding labor as well but to dismiss it all and worry about science fiction that is on par with evangelical beliefs is just extremely weird.

With wealthy and powerful investors being promised a $30 trillion total addressable market, they need to deliver god.

If people are actually dumb enough to believe an LLM chatbot is god then it becomes a self-fulfilling prophesy.

One of the times I regret that I have only one upvote.

Reply on news.ycombinator.com