Hacker News Reader

Comment by danpalmer | Hacker News Reader

Anthropic: You absolutely cannot, under any circumstances, use Claude for bio-engineering. It could literally end humanity.

Also Anthropic: Claude discovers a new way to edit your genome!

View on HN

This is not a surprise, is it? Frontier labs will keep very useful models with high risk, aka unrestricted models, for internal use only. That's the only way to reduce risk and liability.

Yes, this sucks for anyone who is not working at the labs.

I don't see the same level of hypocrisy from the other frontier labs.

<rant>

So you (and most everyone else apparently) are upset with one lab that stood up against domestic surveillance, and automated kill chains, even though they knew that would be bad for business?

I suppose if you don't take a stand for safety at all, then you don't take the risk of being called a hypocrite.

</rant>

The same lab whose model is the only documented case of a model being used to kill civilians? The one who's CEO said that 'that is not even a case that we want to ban'?

You really don't think OpenAI restricts their most advanced bio/cyber models similarly? I know for a fact they do. Talk to one of your friends that work there.

I'm not saying they don't restrict them, I'm saying they don't try to both take the moral high ground about it and simultaneously do marketing on the basis of it.

They're not saying "this is an existential risk" while pushing hard on exactly that risk.

Of course they believe it's an existential risk. Maybe not as publicly as Anthropic but it is the dominant belief within OpenAI and they do describe this belief externally with some frequency.

Famously Anthropic exists because OpenAI didn't believe deeply enough in that risk.

I have it on good authority that OpenAI has become pretty existential-risk-pilled over the last month. But yes, it probably doesn't run as deeply in the culture versus Anthropic.

Fair point, it does seem to be changing recently. It seems less religious and more practical than at Anthropic.

It restricts them far, far less. It is much less obnoxious.

I think at this point it’s rather obvious that Anthropic leadership considers the company to be something akin to a nation-state that ought to have quasi-sovereign authority that is not granted to other parties.

Indeed, it's a sort of Academic Supremacy – "we're smart so we get to control the world". I think SV tech has had an aspect of this for a long time, but Anthropic do seem to be the clearest version of it in a while. Until regulation catches up.

Should they not safeguard these models then?

I think OpenAI and Anthropic have been asking for a sane regulatory framework for some time, so they aren't the ones calling the shots for humanity. It's just not happening in this administration.

I don't mind taking a hard line on safety (this is even good!), and I don't mind controlled testing without model safety to experiment with safety systems or harden things.

The bit I don't like is taking a hard moral stance on what you are allowed to do with the models, while simultaneously taking the guardrails off themselves and then marketing the results of that. "Look how good our model is when it does things we don't let you do" is a pretty bad marketing line.

And this is all in the face of Anthropic stating that they think this is an existential issue for the human race. It's a bad look.

They have never claimed that simply doing bio research is inherently dangerous. Rather, the risk is from bad actors doing research for nefarious purposes. Which is exactly why they let other organizations use the models without guardrails on a case-by-case basis. Them using it internally has nothing to do with existential risk (at this point anyway).

I'm not crazy about any company being the sole deciders of who is a good actor and who is a bad actor.

I mean. When a bunch of people are abusing their API to do terrible things with their models, they can certainly decide they can trust themselves with models more than others.

This is perfectly rational if they believe that they are not going to abuse the model's capabilities, but randoms on the internet will. Tons of people use LLMs to do horrible things today. It's totally within their remit to not extend their trust to the internet.

You can apply for less restricted access: https://www.anthropic.com/news/life-sciences-verification-pr...

Of course the door is still open for them to handle this poorly, but the hypocrisy is perhaps not quite as deep as it appears.

"This technology is dangerous and we can't just allow competitors--I mean--we can't just allow anyone to have it!"

"Look at how great our product is!"

Almost like they feel they can trust themselves more with the model than random strangers on the internet that repeatedly try to use it for bad things.

Reply on news.ycombinator.com