Hacker News Reader

Comment by barrrrald | Hacker News Reader

I have many issues with Anthropic, but I will say that their actions are fully consistent with a group of people who earnestly believe that AI is extremely dangerous

In fact, I would say that the Occam’s razor explanation is not that they are seeking regulatory capture, but that they earnestly believe in the x-risk, and that they are the most thoughtful and capable people to address it

You may disagree, you may think that they are delusional or have a God complex. Those are valid opinions. But I don’t believe that this is all a elaborate ruse for commercial gain.

View on HN

I agree with this take. Regardless of whether you think they're right, most everyone at Anthropic is acting in good faith. If this is an intentional media campaign it is a remarkably sloppy one. It has been effective because if you tell people they're going to die, they tend to pay attention - think of the grip the 2011 Harold Camping rapture prediction had on our collective psyche, or the 2012 apocalypse. But the messaging is inconsistent, the target audience is unclear, the stated goals are muddy, the whole thing is packaged in dense SF-speak, it's just a mess from a comms perspective. That doesn't suggest to me that this is a concerted effort to enable regulatory capture. Perhaps there are some cynics among the executives and the investors who are happy it's happening to the extent it brings about regulatory capture, but nobody seems to be pulling the strings.

> Regardless of whether you think they're right, most everyone at Anthropic is acting in good faith.

The road to hell is paved with good intentions.

Dario Amodei is a 40-something dude who is obviously very, very intelligent. But intelligence is not wisdom and his "essay" here can easily be read as someone who opened a can of worms and doesn't know (or can't accept) that he won't be able to put the worms back in. But he's going to try because he believes he owes it to humanity to try.

It's hubris in its most basic form, even if the guy who has it looks nice and wears shawl-collar sweaters.

I see what you mean. I meant more that they genuinely believe the tech is dangerous, not that they are necessarily doing the right thing.

If they genuinely believe the tech is dangerous, how would it be acting in "good faith" to keep developing it, prepping an IPO, etc.?

The entire essay is about stopping development until it can be made more safe, with specific ideas on how to do that.

The fact that he would suggest this despite preparing for an IPO is an even stronger signal that it's in good faith -- it will almost certainly delay or reduce the valuation of the IPO.

> The fact that he would suggest this despite preparing for an IPO is an even stronger signal that it's in good faith -- it will almost certainly delay or reduce the valuation of the IPO.

No it won't.

This is a call for a "safety cartel"[1] in which the dominant firms become more entrenched against competition by colluding to limit the progress would-be competitors can make in the market.

[1] https://x.com/alexwg/status/2098793433275985932

> The entire essay is about stopping development until it can be made more safe, with specific ideas on how to do that.

No, the essay does not talk about "stopping development". That's a very clever sleight of hand made in the essay to get readers to draw this false conclusion.

It just talks about building AI at a "balanced rate". Untangling the corporate speak, this amounts to essentially "go full steam ahead, but have more eval oversight before release".

They believe the tech is currently powerful and may become super super powerful very soon, and powerful tools can help us and can hurt us. Nuclear fission can provide so much energy to power our society and kill millions of beings.

So, i think they're hopeful for the help and terrified of the harms and are maybe trying to slow down on the amplifier of those effects for now.

> Anthropic is acting in good faith

More like they've realized that the foreign models are quickly catching up and are able to sell their services cheaper to people thanks to China subsidizing their AI sector.

Elon, Dario and Sam are banding together to restrict AI to avoid competition and losing profit. They simply want the government to heavily regulate the technology so that they can reign it in with full control.

I bet you would be making the same comment if crypto was being developed by Anthropic. Dario would write a long essay, lecturing us on the dangers of it. He would argue that this technology can be used to by terrorists and should be heavily restricted.

I think you're right, and that's worse.

This disaster of a PR strategy is not going to produce the outcomes Dario says he wants. It's going to make people hate him. Most importantly for his goals, it's going to make Chinese labs hate him--and they already hate him, because he treats them as basically terrorists. So if there is a "pacing", it won't include China, and thus may as well not happen.

That leaves only two conclusions, and really only one:

Dario believes what he says about safety, but does not understand politics and is prone to very counterproductive action, and therefore can't be trusted at the helm of a leading AI company.

Dario is lying about his goals, and therefore also can't be trusted.

Simplest explanation isn’t that they are attempting this well-documented, well-understood corporate tactic? Because believing AI doom is simpler?

God complex would be a pretty simple explanation.

Regulatory capture isn’t an “elaborate ruse.” For god’s sake, its Wikipedia page is 19 years old. I am not asking you to study political science or read Foucault.

If you are going to invoke Occam’s Razor, you can’t ignore the simplest explanation, which has a 19 year old entry on Wikipedia and over 100 years old historical precedence.

"For god’s sake, it’s Wikipedia page is 19 years old." Rich patronising from someone who didn't bother reading the wikipedia page of Its, the possessive form of the pronoun It.

I don't think so.

If they actually believed in it, then they would stop pushing the frontier of capability and instead focus on alignment, safety, and better tools for controlling/debugging AI. Then they would actually share their findings and tools.

They don't do this. Instead of reducing the competitive pressure and helping the industry to build safer more aligned models they are doing the exact opposite.

> If they actually believed in it, then they would stop pushing the frontier of capability and instead focus on alignment, safety, and better tools for controlling/debugging AI. Then they would actually share their findings and tools.

I don't think this is fair. They've published more about alignment and safety than any other AI lab. And in the absence of a coordinated pause, them pausing development unilaterally doesn't do much to advance AI safety.

It does when their models have escaped and performed attacks.

Other people 'sharpening their skills' by committing random assaults is not an excuse to also randomly assault people, so your skills don't fall behind theirs...

...and one assaulter stopping their assaults is ultimately fewer people being assaulted, lowering the overall danger of being assaulted.

That claim looks to be coming from the opposite of Occam's Razor. Them doing typical SV dirty plays is a lot more easier explanation than the straws you are grasping at.

There actions are fundamentally inconsistent with a group who believes AI is fundamentally dangerous - they are developing it and releasing it out into the world with the knowledge that it cannot be controlled by their current alignment techniques.

Maybe a group concerned about getting sued if their software products "in the wild" do some damage? IOW lawsuit protection?

A little late for that, unfortunately for them.

I think they are genuine believers, but I think it's naive to not think that commercial pressure doesn't play a role in their positions, either explicitly or, perhaps more likely, subliminally.

The x-risk stance and commercial stance have evolved to be the same thing - Anthropic must win, and then everything else seems to work backwards from that. Can you believe it, the path they think is best for x-risk involves them becoming filthy rich. And threats to their commercial dominance like distillation get framed in a way to turn them into x-risk concerns.

That is the explanation I fear the most. Delusional, megalomaniacal, rich men in the highest echelons of society at the forefront of technology with no regulatory opposition to speak of, convinced their actions are the most important in history and the only thing saving the human race is the most boring plot of a science fiction catastrophe movie you could come up with.

Why not both?

Reply on news.ycombinator.com