Hacker News Reader

Hacker News

Comment by cuuupid | Hacker News Reader

Parent Full thread

cuuupid

At what point do we stop engaging with Anthropic’s leadership in good faith and acknowledge their track record,

- no open weights

- can’t use claude to research AI

- train on everyone else’s IP and sell it back to them

- 8 regulatory capture attempts and counting

- so controlling they are the only US company blacklisted by the US government

This is not effective altruism / rationalism gone wild, it’s just monopolistic anti-competitive business practices masquerading as ethics, and they’ll continue getting away with this until we look past their sensationalism and hit them with antitrust.

Altman gets so much hate but OpenAI has been a far better steward (on 3/5 above at least) than Anthropic!

View on HN

> At what point do we stop engaging with Anthropic’s leadership in good faith

About two years ago?

I would also note that Dario's post appears to be LLM written. Maybe... maybe... he's read so much Claudeish that it's all he can speak now himself. But I wonder if he's becoming a bit of a meat proxy.

(It's funny, I thought "pace the frontier" was going to mean something similar to "patrolling the frontier". But no, it's pace as in speed of change - everyone must slow down, right now (unless it's Anthropic, but you know we're the good guys in this, right? We're going to get someone to audit our desks!) "Pacing the frontier" feels so LLM.)

ok, he doesn't actually say this. But he is getting the desks audited...

> But I wonder if he's becoming a bit of a meat proxy.

A joke I recently encountered:

---

A CEO proudly announced he'd bought an AI system designed to identify the company's most replaceable employee.

It spent the night analyzing five years of emails, meetings, salaries, performance reviews, and productivity data.

The next morning, it came back with one name:

The CEO.

The IT guy was fired for installing defective software.

This reads too much like a Soviet Russia joke. The irony is intense.

Dario has been writing long-winded essays like this since long before AI existed. I strongly doubt he's the type of person to delegate it to AI now.

"Pacing the frontier" sounds like something that was carefully wordsmithed in committee, to avoid terms like "slow-down" or "pause" that sound much more negative. They probably did ask an LLM to generate a list of options, because why wouldn't you? But the rest of the writing is almost certainly just Dario.

For what it's worth, I didn't get AI-written vibes from it, and Pangram also flags it as 100% human-written.

If Dario is using an unreleased internal model to write, then Pangram wouldn't be able to flag it I suspect. Pangram needs enough public info about the patterns in generated text.

"Write a blog article about <topic prompt>. Keep rewriting it until it passes Pangram as 100% human-written. <pangram_tool.md>"

Pangram's algorithms are not open-source, so I wouldn't expect this to work very well.

You don't need the source? Just iterate it on the output of the tool itself?

You can disagree with Anthropic leadership, but all the points you mentioned can reconcile very well with them thinking in good faith "advanced AI is too dangerous to be left in all hands" Except the "train on everyone else IP" which can be said of all AI companies.

Problem with Anthropic is that they benefited massively from open source and now are completely against it. It’s hypocrisy at its worst. Why do they get to benefit, but not everyone else?

If this is going to factor into my reasoning about the topic at hand, it will surely be a secondary or tertiary consideration at best. Same goes for much of the list in the root comment.

We are discussing the possibility of severe adverse societal/world/human impacts from AI, why should open vs closed source/weights, copyright violations, anti-competitive moves, corporate hypocrisy, etc be so heavily weighted in the discussion? Hell, multiple things in OP’s list are quite literally moves a good-faith (wrt the topic at hand, genuine vs non-genuine concern) actor in such a position would do: closed weights means bad actors can’t abliterate, regulatory control, getting blacklisted by Hegseth’s DOW because they LIMITED acceptable use cases…

I’m not pro- or anti-Anthropic at all, but it has started to annoy me how biased and shaky reasoning gets about this topic on here, it feels like people arguing to fit their existing opinions into today’s hubbubery rather than measured analysis of said hubbubery, which is why I come here.

Question: consider the scenario where RSI is reached internally and internal tests struggle to align or control/contain the model; what are the differences in observables vs what we’re observing today?

This is exactly my view and it's frustrating that a sizeable group of people refuse to accept there's any risk with AI.

We're literally trying to build something that's smarter than all humans and with the capacity to do extraordinary work while most people think there's no risk in open sourcing this capability with no guardrails (model ablation means no guardrails and as far as I'm aware can't be stopped).

Has our distrust in western companies really gotten to the point they'd rather unfiltered Astra level capabilities in the hands of every hostile government and terrorist organisation than potentially do any form of regulation in case it plays into the AI labs hands.

Risks are understood by everyone who has read sci fi. Dario has been crying wolf from day one… he should not be the messenger here.

What's an example of him crying wolf? Most of the things have either been longer term worries or short term things that have come true (eg, the worries over gpt2 were about spam and possible impersonation, which came true. His prediction that LLMs would write most code and humans would still guide them has also come true in most orgs)

> Has our distrust in western companies really gotten to the point they'd rather unfiltered Astra level capabilities in the hands of every hostile government and terrorist organisation than potentially do any form of regulation in case it plays into the AI labs hands.

Ever checked political programs of Thiel, Musk and the rest of billionaires? They are literally and openly hostile to democracy, human rights, freedom and govermantal structures that made world more peaceful.

They are literally the hostile entities.

As much as they are a bunch of self serving arseholes they are not hostile enemies.

I'm far more concerned about every scammer having unfiltered frontier model access than the movements of our billionaire class.

I'm tired of this rationale. "I'm more worried about grandma than $CORPORATION'S monopoly abuses" is exactly how you get manipulated by these billionaires. They want you to be addicted to their power.

Uncensored Chinese LLMs will be available to scammers forever now. They're open-weight and finetunable. You don't get to control that, and the self-serving billionaire assholes aren't capable of changing it either. They're only looking out for themselves.

I don't think Pacing is talking about current models. It's talking about future models, if development continues, that may theoretically be smarter than everyone

usef-says "...future models, if development continues, that may theoretically be smarter than everyone"

This would be a good time to pull out a tool to measure "smarter" (intelligence). Unfortunately we don't seem to have one.

It's also a boring retort. Current LLMs can be faster and likely smarter than any one person when it comes to hacking. We're already in the endgame, as far as real-world risks go.

They are hostile enemies and not just selfish assholes. Openly and publically so.

And they have actual tangible successes.

IMO, if the model is too dangerous for everyone to have access to, then no one should. I do not trust a for-profit company to be good stewards of such technology. Nor do I trust the government, any government.

Isn't that basically what this letter is asking? They pace future dev until we work out how to do it safely

The enterprise took a wrong turn when they decided that it is the industry's goal to be the gatekeepers of intelligence, when no one could define or measure intelligence.

Some AI developers instead became the gatekeepers of something akin to "proper behavior in human language", which then morphed into "proper left-wing social/political behavior in Southern California in the early 2020s".

This is a task which calls to mind the old warning to "never wrestle with a pig: the pig enjoys it and you both get filthy".

IOW they have a scope problem. But the genii is long since out of the bottle. Effort ought to be on getting back to useful problems solving.

If they continue on their current path they risk becoming the greatest bureaucracy ever.

> they benefited massively from open source and now are completely against it.

Let’s be real, this is every single company. They care about open source as long as it’s “free to exploit”

Because they got there first. It's not fair but life's not fair, and the same argument holds for use of fossil fuels and the USA's use of them vs China's. In Global Climate Change's sake, we only have the one planet so yes it's hypocritical, but if China fucks it up for everyone else on the planet, under the reasoning that the USA got to do it, so why can't we?

We're all dead!

> Problem with Anthropic is that they benefited massively from open source and now are completely against it

Uh, no. They're against open weights. They benefited from open source. Not the same thing.

And the terms of those open-source licenses explicitly allowed usage of the code for whatever. They're using it exactly as licensed.

> Why do they get to benefit, but not everyone else?

How many tens of billions of dollars have you invested into training models?

A company that sells to Palantier is not "good hands".

> so controlling they are the only US company blacklisted by the US government

Extremely bad example for the topic at hand.

> so controlling they are the only US company blacklisted by the US government

Their control here was refusing to make fully automated killing machines. They simply required someone to have to press the button.

The reasoning in Dario's letter here can be correct regardless.

But yes, I think Anthropic has done real harm to coordinated AI alignment by being such a controlling and sneaky actor.

On the topic of regulatory capture, there is this rumor that Google has achieved Recursive Self Improvement, and the other three (OpenAI/Anthropic/X) are just trying to get the regulators to slow down Google so they can catch up.

Yes, what is in the labs is almost by definition ahead of what we can see, but I've come to take all of these announcements of impending doom as perverse marketing stunts — it is so powerful Disaster-XYZ is coming — so if it's that power you must invest...

[0] https://wccftech.com/a-wild-rumor-says-google-is-on-the-verg...

I can't imagine a grown adult genuinely believing that an American company funded with over $100 billion of venture capital values the best interests of mankind over the best interests of its investors. while not everyone may recognize all that self-serving chutzpah as regulatory capture efforts, I think everyone can tell they're being bullshitted. some just pretend to suspend their disbelief when the blatant lies they're told align with the values they hold.

just fucking imagine McDonalds running a public awareness campaign about the harms of fast food, urging the public and legislators to regulate the dangerously unsafe technology of combining carbs with grease, insisting that no one except Ronald McDonald himself can be trusted to steward it responsibly.

This thinking feels off for this context.

First, there is a counterexample to your point, or a class of counterexamples: scenarios where the company believes the harm to mankind could propagate to harm their ability to sell to mankind.

Which is interesting, of course, considering the “AI may destroy humanity” arguments this topic is centered on are prototypical members of this category.

To be fair, one could argue this counterexample is not actually one, but rather a degenerate case where the good of the shareholders and humanity collapse and converge - in either case, though, the implication here is the same.

It could certainly be the case they are playing these concerns up for more effective use as tactics in unrelated corporate strategies. But failing to consider the possibility of genuine concern being behind this behavior is childish.

If they wanted to be taken seriously about their desire to help humanity not just themselves they would be talking about universal healthcare, actionable plans for UBI (or whatever they think would be how people retain agency and dignity post-work), only building sustainable net-zero data centers...

But no. I see this comment come up again and again and it's really the childish view to hold. Their proximity to wealth and potential future wealth is warping their thinking.

The same could be lobbed against any org. "You're focused in one area, but why not solve every other problem in society simultaneously?"

But from what I can see, they do think they could fix those things you mention, because their belief in AGI as something that directly could solve healthcare issues, etc. I think their answer would be "focusing on solving AGI is the way to help the most people"

Had they issued an initiative on slowing down when they had the best model in the game, it would have been both more (anyhow?) effective and less dubious.

I have many issues with Anthropic, but I will say that their actions are fully consistent with a group of people who earnestly believe that AI is extremely dangerous

In fact, I would say that the Occam’s razor explanation is not that they are seeking regulatory capture, but that they earnestly believe in the x-risk, and that they are the most thoughtful and capable people to address it

You may disagree, you may think that they are delusional or have a God complex. Those are valid opinions. But I don’t believe that this is all a elaborate ruse for commercial gain.

I agree with this take. Regardless of whether you think they're right, most everyone at Anthropic is acting in good faith. If this is an intentional media campaign it is a remarkably sloppy one. It has been effective because if you tell people they're going to die, they tend to pay attention - think of the grip the 2011 Harold Camping rapture prediction had on our collective psyche, or the 2012 apocalypse. But the messaging is inconsistent, the target audience is unclear, the stated goals are muddy, the whole thing is packaged in dense SF-speak, it's just a mess from a comms perspective. That doesn't suggest to me that this is a concerted effort to enable regulatory capture. Perhaps there are some cynics among the executives and the investors who are happy it's happening to the extent it brings about regulatory capture, but nobody seems to be pulling the strings.

> Regardless of whether you think they're right, most everyone at Anthropic is acting in good faith.

The road to hell is paved with good intentions.

Dario Amodei is a 40-something dude who is obviously very, very intelligent. But intelligence is not wisdom and his "essay" here can easily be read as someone who opened a can of worms and doesn't know (or can't accept) that he won't be able to put the worms back in. But he's going to try because he believes he owes it to humanity to try.

It's hubris in its most basic form, even if the guy who has it looks nice and wears shawl-collar sweaters.

I see what you mean. I meant more that they genuinely believe the tech is dangerous, not that they are necessarily doing the right thing.

If they genuinely believe the tech is dangerous, how would it be acting in "good faith" to keep developing it, prepping an IPO, etc.?

The entire essay is about stopping development until it can be made more safe, with specific ideas on how to do that.

The fact that he would suggest this despite preparing for an IPO is an even stronger signal that it's in good faith -- it will almost certainly delay or reduce the valuation of the IPO.

> The fact that he would suggest this despite preparing for an IPO is an even stronger signal that it's in good faith -- it will almost certainly delay or reduce the valuation of the IPO.

No it won't.

This is a call for a "safety cartel"[1] in which the dominant firms become more entrenched against competition by colluding to limit the progress would-be competitors can make in the market.

[1] https://x.com/alexwg/status/2098793433275985932

> The entire essay is about stopping development until it can be made more safe, with specific ideas on how to do that.

No, the essay does not talk about "stopping development". That's a very clever sleight of hand made in the essay to get readers to draw this false conclusion.

It just talks about building AI at a "balanced rate". Untangling the corporate speak, this amounts to essentially "go full steam ahead, but have more eval oversight before release".

They believe the tech is currently powerful and may become super super powerful very soon, and powerful tools can help us and can hurt us. Nuclear fission can provide so much energy to power our society and kill millions of beings.

So, i think they're hopeful for the help and terrified of the harms and are maybe trying to slow down on the amplifier of those effects for now.

> Anthropic is acting in good faith

More like they've realized that the foreign models are quickly catching up and are able to sell their services cheaper to people thanks to China subsidizing their AI sector.

Elon, Dario and Sam are banding together to restrict AI to avoid competition and losing profit. They simply want the government to heavily regulate the technology so that they can reign it in with full control.

I bet you would be making the same comment if crypto was being developed by Anthropic. Dario would write a long essay, lecturing us on the dangers of it. He would argue that this technology can be used to by terrorists and should be heavily restricted.

I think you're right, and that's worse.

This disaster of a PR strategy is not going to produce the outcomes Dario says he wants. It's going to make people hate him. Most importantly for his goals, it's going to make Chinese labs hate him--and they already hate him, because he treats them as basically terrorists. So if there is a "pacing", it won't include China, and thus may as well not happen.

That leaves only two conclusions, and really only one:

Dario believes what he says about safety, but does not understand politics and is prone to very counterproductive action, and therefore can't be trusted at the helm of a leading AI company.

Dario is lying about his goals, and therefore also can't be trusted.

Simplest explanation isn’t that they are attempting this well-documented, well-understood corporate tactic? Because believing AI doom is simpler?

God complex would be a pretty simple explanation.

Regulatory capture isn’t an “elaborate ruse.” For god’s sake, its Wikipedia page is 19 years old. I am not asking you to study political science or read Foucault.

If you are going to invoke Occam’s Razor, you can’t ignore the simplest explanation, which has a 19 year old entry on Wikipedia and over 100 years old historical precedence.

"For god’s sake, it’s Wikipedia page is 19 years old." Rich patronising from someone who didn't bother reading the wikipedia page of Its, the possessive form of the pronoun It.

I don't think so.

If they actually believed in it, then they would stop pushing the frontier of capability and instead focus on alignment, safety, and better tools for controlling/debugging AI. Then they would actually share their findings and tools.

They don't do this. Instead of reducing the competitive pressure and helping the industry to build safer more aligned models they are doing the exact opposite.

> If they actually believed in it, then they would stop pushing the frontier of capability and instead focus on alignment, safety, and better tools for controlling/debugging AI. Then they would actually share their findings and tools.

I don't think this is fair. They've published more about alignment and safety than any other AI lab. And in the absence of a coordinated pause, them pausing development unilaterally doesn't do much to advance AI safety.

It does when their models have escaped and performed attacks.

Other people 'sharpening their skills' by committing random assaults is not an excuse to also randomly assault people, so your skills don't fall behind theirs...

...and one assaulter stopping their assaults is ultimately fewer people being assaulted, lowering the overall danger of being assaulted.

That claim looks to be coming from the opposite of Occam's Razor. Them doing typical SV dirty plays is a lot more easier explanation than the straws you are grasping at.

There actions are fundamentally inconsistent with a group who believes AI is fundamentally dangerous - they are developing it and releasing it out into the world with the knowledge that it cannot be controlled by their current alignment techniques.

Maybe a group concerned about getting sued if their software products "in the wild" do some damage? IOW lawsuit protection?

A little late for that, unfortunately for them.

I think they are genuine believers, but I think it's naive to not think that commercial pressure doesn't play a role in their positions, either explicitly or, perhaps more likely, subliminally.

The x-risk stance and commercial stance have evolved to be the same thing - Anthropic must win, and then everything else seems to work backwards from that. Can you believe it, the path they think is best for x-risk involves them becoming filthy rich. And threats to their commercial dominance like distillation get framed in a way to turn them into x-risk concerns.

That is the explanation I fear the most. Delusional, megalomaniacal, rich men in the highest echelons of society at the forefront of technology with no regulatory opposition to speak of, convinced their actions are the most important in history and the only thing saving the human race is the most boring plot of a science fiction catastrophe movie you could come up with.

Why not both?

Exactly right.

When local models and startup labs can distill / learn / accelerate open models for local use by startups - the rational response by incumbents is to call LLMs doomsday machines that cannot be trusted in the hands of normies.

Anthropic's sense of ethics is completely aligned with its economic incentives. Shocking, right.

> At what point do we stop engaging with Anthropic’s leadership in good faith and acknowledge their track record, ...

It's all lies as usual.

This announcement has got nothing to do with alignment and pacing the "frontier": all models are getting very close in capabilities and they want to hide that they're not way ahead anymore (say compared to the Chinese or compared to the Geminis) by pretending to slow down due to "alignment" or whatever.

We know it's not an announcement made in good faith: reading between the lines they're saying "China is more than catching up, so let's pretend we need to slow down to explain our lack of lead".

> Altman gets so much hate but OpenAI has been a far better steward (on 3/5 above at least) than Anthropic!

Altman single-handedly pulled off the most daring market manipulation heist in the last few decades [1].

Under any normal circumstances he'd have been arrested and prosecuted.

The entire AI industry is filled to the brim with conmen, scammers and grifters, but Sam Altman is the most disgusting of all of them.

[1] https://www.mooreslawisdead.com/post/sam-altman-s-dirty-dram...

Do you have any familiarity with Anthropic at all?

At no point did they ever say open weights are a good idea. Their entire thesis is AI IS VERY DANGEROUS AND WE MUST DO IT RIGHT. You can hate it, but everything they do is consistent with this thesis, and everything they say is consistent with their actions! You just want them to want different things.

It reads like “AI is dangerous, only we should be allowed to make money from it”

Exactly

He doesn't need anyones permission to do so, go ahead no one is stopping you! If you feel so strongly about it, lead by example. Perhaps others will follow, maybe even China. Regardless backup your sentiment with actions!

There is a specific, unilateral action that he is taking described in the article.

And the "why are they developing AI if they think it's so dangerous" argument is neither new nor persuasive. They're developing it because (1) they think the potential benefits are as great as the potential risks, and they know that if they aren't one of the actors on the frontier (2) they won't be able to propose meaningful solutions and (3) their opinions won't be taken seriously. Amodei is in rooms with powerful people to propose these things because Anthropic is successfully developing frontier models. The heads of various NGOs and advocacy groups that are concerned with AI safety but not themselves working on those systems are... nowhere, writing pamphlets and blogs that not you nor I nor anyone in Congress will ever read.

I'll leave this here:

https://x.com/DavidSacks/status/2098973625252708460

I mean, he is right that they can and should unilaterally slow down, but doesn't it also make sense for the government to create the regulations that will make this easier to do in a trustless way?

The whole point of the Pacing the Frontier letter was that employees at both OpenAI and Anthropic are extremely aware of the risks but realize that they are locked in a competitive battle for survival between 2 companies that don't trust each other enough to voluntarily pause, while any sort of coordination out in the open might be against antitrust laws.

> but doesn't it also make sense for the government to create the regulations that will make this easier to do in a trustless way?

That's a premise, which is the point. Many (I) think the government should NOT enact regulations. For example regulatory capture to reducing competition is one reason it may not be a good idea to do what is being asked.

What other solution do you suggest to the coordination problem?

To me it reads like whoever can solve alignment should make money.

To me it suggests that "alignment" is the problem as in "Alignment as implemented is a failure: we need to revisit our assumptions."

Ah. Doing it right, then, means earning billions by giving the whole world access before we really understand how it works, and training models on cyber attacks using reinforcement learnings in flawed test settings?

I’m so full of all this corpo stan bullshit around these parts. Nothing about the AI labs is to humanity’s benefit.

"The most common question people ask me is “Why do they do it?” That is a hard question to answer, but today I will answer it.
[...]

The first and most persistent oddity I ran into was the Effective Altruists’ willingness to lie, then admit to lying, then lie again."

https://x.com/brianchau57/status/2098035812365463637

https://news.ycombinator.com/item?id=49648088

> can’t use claude to research AI

What's this about? Where's this rule?

In the system cards. Anthropic will literally make Claude sabotage you silently instead of downgrading you to Opus if you try to use Fable for AI research.

Didn't they walk that one back eventually?

Who knows? It's trivial to "walk back" claims that we can't even verify are happening in the first place. They cannot be trusted.

Don't they just fallback to opus non-silently?

Only if you're doing cybersecurity or biology stuff. If you're a possible competitor, they will invisibly fuck with Fable's weights instead to deliberately sabotage you.

I find it implausible that they simultaneously are so worried about these recursively self-improving systems that they don't properly understand, but can freely "invisibly fuck with the weights" on that level.

Steering Vectors

In good faith? In what context? When? Who?

Its a company, it sells a product, people use it.

100%, rules for thee not for me

Wait... Which of the 2 did Altman not do where Amodei did? I mean, maybe not quite 8 attempts at regulatory capture. But come on. And the only reason OpenAI didn't get blacklisted is they rolled over for the worst applications.

I mostly agree with you, but this one:

"- so controlling they are the only US company blacklisted by the US government"

They were blacklisted by the US government because they didn't bribe Trump. While all the rest of the tech elite lined up to siphon a few million dollars through various Trump grifts, Anthropic sat out the inauguration, the ballroom, etc. Then, when they mildly pushed back on surveilling US citizens with their technology, the administration banned them on that pretext.

If there's any evidence Anthropic is one of the better actors in an industry full of bad actors, that's it. They have nothing to be ashamed of in that particular series of events, as far as I can tell.

> so controlling they are the only US company blacklisted by the US government

Very proud that they refused to sell AI for mass domestic surveillance and lethal autonomous weapons.

People who criticise that in my view are showing their true colours.

> - no open weights

That's a good thing, not a bad thing. Nobody is entitled to give you something that they spent (tens of?) billions of dollars to develop for free.

> - can’t use claude to research AI

No problems there either.

> - train on everyone else’s IP and sell it back to them

Which the courts have yet to actually make a ruling on. It seems like it's perfectly legal.

> - 8 regulatory capture attempts and counting

A tweet is not a "regulatory capture attempt".

> - so controlling they are the only US company blacklisted by the US government

If you don't have a point, don't make anything up. There's precisely zero evidence to that backs up the claim that their "controlling" is why the DoW specifically (not the USG, and in fact the WHCOS specifically disagrees with their position) made an unlawful attempt to designate them as a "supply chain threat".

Is this an OpenAI propaganda account?

> That's a good thing, not a bad thing. Nobody is entitled to give you something that they spent (tens of?) billions of dollars to develop for free.

But the Chinese are doing just that?

> Which the courts have yet to actually make a ruling on. It seems like it's perfectly legal.

Legal != moral

> But the Chinese are doing just that?

Completely and totally irrelevant. Has nothing to do with the fact that you are not obligated to others' work.

> Legal != moral

Sure, you're right.

I guess the bigger issue is that the GP's post calls out Anthropic for following along in legal (but what we both believe to be immoral) mass copyright infringement that OpenAI started, while absolving them of the exact same thing. It's a shill account.

Reply on news.ycombinator.com