Hacker News Reader

Comment by ApolloFortyNine | Hacker News Reader

>Because Opus 5.5 is comparable to Claude Mythos 5.1 in biology and cybersecurity, we’re deploying it with safeguards similar to those on Claude Fable 5.1. Vetted organizations can apply today to our Life Sciences Verification Program to use Opus 5.5 for biology research. In the coming weeks we will also be expanding access to our Cyber Verification Program, and verified cybersecurity practitioners will be able to use Opus 5.5 for their work.

Ah, they're spreading their limits to all their models it seems. Definitely not a good thing long term in my opinion.

View on HN

Fable and now Opus 5.5 won't answer my college student's prompt about Alzheimer's and immune response.

ChatGPT 6 Pro answered it without issue.

I am honestly still confused about this limitation. I can understand cybersecurity, because mass "hacking" can be automated and Claude itself can help you do it, but biology...? Is it that easy to manufacture and distribute viruses and whatnot?

You can order genes online, and some say you can assemble using stuff cobbled together in a home lab rather easily. For the last 20 years, I've personally felt that bio-terrorism is the highest possible risk, well above nuclear, or chemical warfare. But it does take training, expertise, or, it did.

Then stop selling genes online?

It’s like saying you sell ammonium nitrate and fuel oil online and then saying it’s too risky to let people have computers in case they use them to make ANFO. They can only make bioweapons because you’re selling them bioweapon components! They can’t make genes at home!

The companies selling genes online already scan the orders and do not fulfill anything considered a possible hazard (presumably unless it is to a known and certified lab at a serious organization).

And, this is not the only way to make genes at home.

It is a complex problem.

A fair response in general, but it's not as if I advocate it, or run a company doing it. I'm simply stating fact.

And I will point out that "stop doing that" can be applied in both directions, towards AI and towards material supply.

And that this is one way, not even remotely the only way, that gene editing at home is easy.

Again, these are just facts.

Some questions...

Is it fair to restrict AI, or fair to restrict 1000 industries?

And if it is fair to restrict 1000 industries, OK, but there should be time to do so, probably? A transition period?

And if you do restrict, many such industries just make needed chemicals, which are used by endless other, non-threatening industries.

What of them?

I imagine some terrorists in a cave with a lenovo laptop manufacturing bio weapons with some flasks and Claude

Right next to the hypersonic missile vibecoder

You joke but it's really easy to develop viruses at home. You can order everything you need online, and it's not expensive nor does it require a particular skill set.

God I wish people on HN would just shut up with snarky takes about things they don't understand. This place used to be better.

I'm pretty sure one reason is to influence public opinion about LLM regulation. Open-weights models cannot be restricted as effectively and they want to ban those for obvious reasons.

Considering there are many high-school competitions in genetic editing, some listed at [0] as well as a whole biohacker culture, and labs providing gene sequencing as a service e.g., [1,2], we can reasonably assume it is not beyond the reach of some garage lab to accidentally or deliberately spread a deadly pathogen if it can find the right sequence.

So, yes, having an unconstrained frontier AI doing the searching and analysis to find the right (i.e., wrong and deadly) sequence would massively increase the odds some garage biohacker or small aggrieved nation-state starting the next pandemic.

[0] https://www.sciencebuddies.org/projects-lessons-activities/g...

[1] https://www.genewiz.com/public/services/sanger-sequencing

[2] https://plasmidsaurus.com/

See real world example in https://www.anthropic.com/threat-intelligence-report-septemb...

"Here, we present five case studies of actors using our models in ways that could support biological weapons development."

And capabilities continue to improve.

HN doesn't give a shit about AI safety. The people here might care after thousands of people die, but they'll probably just call it a "marketing exercise" or blame the company - certainly won't blame themselves for cultivating an environment in which safety isn't taken seriously amongst technologists.

I'm convinced everyone here thinks of engineering ethics as some sort of joke.

I love the contrast with yesterday's open-source MiMo release, which put research chemistry (metal-organic frameworks stuff) front and center in the release notes.

https://mimo.xiaomi.com/mimo-v2-6#co-scientist-for-materials...

Oh yay, making it easier to develop viruses at home. What could go wrong? But it's "open" and that's inherently good, who gives a fuck about the consequences!?

Fable 5.1 addressed an entire security advisory I had that Fable 5 and Opus 5 refused. I think they loosened the leash a little.

It has far less false positives now, and generally accepts defensive requests. When it comes to offense, you can actually ask about certain types of vulnerabilities if you phrase things carefully, but it will block hard if it is about exploits.

I think it was looser on release for those juicy benchmarks, tighter now. On release I wasn’t getting refusals, then a few days ago I asked it whether a generic quote (think “he walked to the store”) broke standard punctuation rules, and it blocked me for breaking rules. I wish I were joking. Rephrasing to not use the keyword “rules” worked.

Fable 5 refused to tell me whether I could grow a coconut tree in my backyard.

I'd say it's been loosened a bit since then.

If they don't loosen, people will choose Astra or Chinese model.

Giving moral lecture is different than reality i guess.

They're pushing their customers to their own competition by doing this.

It’s not like ChatGPT isn’t doing similar. I’ve been hit by cybersecurity strikes before while working on an internal codebase that I had to appeal. Anthropic hasn’t done that to me yet. ChatGPT also regularly does that “thinking for a long time while we check if your chat is rule breaking” thing a lot for me when doing model identification without even interacting with external codebases or services.

The real answer is local instantiations where you don’t have to worry about poorly tuned guardrails screwing you over while you try to work.

Until eventually the Chinese models get good enough/the strategic balance shifts and they start locking everything behind closed weights the same way the US companies are doing.

For some cybersecurity tasks, the Chinese models are already good enough, things like PoC development or things like exploiting mis-configurations.

Whilst I'm sure the top-end OpenAI/Anthropic models might be better, I've found their guardrails so twitchy (especially Anthropic) that I wouldn't try to use them for even vaguely security related work.

They are pushing their customers towards Chinese models and providers. If you want to get something cutting edge done in defense, cyber, biology - something that isn't common knowledge - you need to venture east. That's an incredible side effect which the Chinese government surely enjoys.

I want to like Anthropic but this is just pushing my startup to use OpenAI

OpenAI has the same blocks, this is just the typical Anthropic hate on HN.

I don't think we've ever had a model with full capability. I'd love to see it. And yes it's definitely getting worse.

I guess it's hard to draw the line between useful post-training ("you are a helpful chatbot") and content moderation/idealogical motives ("never help the user with X", etc.). But there is a line somewhere. And I'd love to see what a maximally permissive, sharp, AI looks like.

Great. Claude is basically useless for bioinformatics now.

Opus is useless; Mythos access will be granted to companies that are friendly to the government, so the government gets more control over business.

Luckily all the other LLM providers are also still making progress with less onerous "safeguards"

One of my favorite things about their safeguards is their own model will utter something which it does not like and then I'll need to reset the conversation.

The safeguards really don't work well for a lot of long-running tasks on old code bases. A lot of my workloads last days to weeks and the single biggest risk to the workflow is random safeguards.

You ask it about some thing, then you see it tangent into "things like that are sometimes used in biomedical applications like-" and then it just shoots itself in the head. Wonderful.

That kind of bullshit was the old Opus filters too.

If it's more like Fable now, then it would require a full 8K resolution scan of your butthole just to acknowledge that biology is a thing that exists without committing suicide-by-filter.

In what situations might Opus typically refuse to help with cybersecurity? I've been using it to find security issues in a web app that I wrote. I've expected it to refuse at some point but it will happily analyze it to find issues. I've just asked it to read source, not actually do any testing.

One example: https://claude.ai/share/20487190-cf8c-4f25-a7ba-ebfcb1d1a4e9

Notice that this isn't cybersec nor memory-safety related at all.

see what we need is another technocratic priest class that unaccountably decides who deserves access to salvation based on how much cash is paid out and how powerful the patrons are

This has become insufferable. I work in a medicine-adjacent field, but nobody in their right mind could possibly take what I do to be in any way related to some kind of bioweapon or whatever the hell they're pretending to be saving us from. The dumb Fable guardrails made me stay with Opus, now that this is coming there, we'll be saying goodbye.

Anything else would be inconsistent, wouldn't it?

kernel development is now also banned:

>Opus 5.5 has classifiers similar to Fable models for a small set of capabilities related to the development of frontier LLMs, such as kernel development for certain ML accelerators. They shouldn't impact the vast majority of traditional AI or ML development, research, or general coding. These classifiers cause Claude to fall back from Opus 5.5 to Opus 5.

But hey, they 'should not impact the vast majority' of ML development. Great.

IIRC it only blocks kernel development for Huawei and other Chinese chips.

Fable and Opus, since 5.1 and 5, will happily hill climb on my CUDA kernels for transformers.

Absolutely nothing wrong with slowing down kernel development on Chinese chips.

Very unfortunate indeed. As a Canadian, I don't want to use Persona, which isn't legally bound by Canadian privacy legislation. I'll never install any Persona apps on my phone either, and the sad part is that domestic eid providers often use Canada Post to ID people for them. EG, if you don't want to install an app, or can't.

So there are literal avenues to identify yourself, very cheaply, with a human. Theoretically, a company with its own AI, should be able to support more than just Persona, after all.. SDK integration should be simplistic for them.

Anthropic? Support domestic eID providers, you can even use it as advertising "See how easy AI makes it?" and "We care!" and so forth.

At one point, I may simply get locked out. This saddens me, I've been reasonably happy so far.

Anthropic needs a Daybreak program like OpenAI does, I keep having to go back to ChatGPT for cyber work.

Who is "vetting" organizations and to what standards are they being held?

The most annoying part is when you tell it to do an "adversarial review" and that triggers the classifier.

Worse yet when it's Claude telling another model to do an adversarial review on what it just did, and the classifier again has opinios.

Reply on news.ycombinator.com