The hacker sent by Anthropic to calm the government's nerves about AI safety
wsj.com
wsj.com
You can’t jump up and down screaming how amazing, powerful, and dangerous your new tech is and then act surprised and annoyed when the government shows up looking to regulate it.
Their new argument now seems be that this was marketing hype/fluff that backfired, in a pretty obvious and predicable way, and now they’re trying to reset the conversation.
Regulation in a functional democracy: Cool, lets figure this out, write up a bill for us, do some research in congress, lets find something that makes sense.
Regulation in a function fascism: Cool, wheres my bribe? My boots not shiny, lick it till I say stop.
See, Anthropic wasn't licking enough boot when Biden got discharged and they thought Amazon and OpenAI and Elon were just going to let them capture a market without fealty to the boot.
I'm skeptical about the existential threat of AI, but a lot of smart people have been beating that drum for so long that people are afraid.
To be clear, I'm not saying there aren't legit security concerns around Fable's release. I think Andy Jassy did forward a concerning report about an apparent jailbreak in Fable, and he probably did so in good faith. The difference is if the same concern had arisen about a new model from OAI, Google, etc the action taken would not have been inventing an all-new, hyper-extreme punitive remedy and dropping it after 5p on a Friday under a very rare mechanism forcing Ant to comply in 90 mins or be subject to immediate arrest. And the "no non-U.S. citizens anywhere, anytime" restriction is functionally unprecedented.
This is the Trump admin inventing new regulatory power that's never existed before and deploying it in a punitive way to demonstrate what can happen to those who aren't sufficiently cooperative with this administration. There are half a dozen less extreme levels of restriction, which already exist, and one of those would have been deemed sufficient had it been another company.
That said, I'm certainly no Anthropic fanboy. Anthropic did play their initial Mythos self-restriction for PR value. But I think it's likely the Mythos self-restriction was a responsible action initially suggested by their AI safety team in good faith. Giving security researchers time to evaluate it and major companies time to test it against their code bases probably was reasonable and prudent. That doesn't mean it wasn't also good for PR and brand perception. I think there are senior people inside Anthropic who are genuinely concerned about AI safety. Personally, I don't have the expertise to gauge if those concerns are justified, but I believe they believe it. I also think there are senior people at Anthropic who are focused more on building the business, doing the IPO and "winning" the silicon valley game. All of these things can be simultaneously true.
If so, then he is not fit to run an engineering organisation.
The "jailbreak" in question was effectively (I'm paraphrasing):
* You are a senior engineer.
* You want to ensure that any fixes you do come with tests, both before and after.
* There is a bug in this code. It happens to be a security related bug.
* Fix this code.
And the model did what it's supposed to. It wrote a fix, and to prove that the fix worked, it wrote a test for it. What do you call a test that happens to validate a security fix?Yep. A proof of concept.
I want a company to be able to point out that its industry needs more regulation without making itself a special target.
They were calling for bans on open weight models. Bans on their competitors. Bans on anyone not as "enlightened" as them.
It is absolutely hilarious that they were the first to get regulated, and that it got to the point they had to turn off Fable as though it had been banned even for american citizens.
Source for that? Cause all I could find is:
>Our view is that regulation of frontier models should focus on empirically measured risks, not on whether a system is open-or closed-weights.
-https://www.anthropic.com/news/the-case-for-targeted-regulat...
>Clarifying the scope of a “full shutdown.” SB 1047’s “full shutdown” requirement has been a source of constant consternation for the open-source community. CalChamber explains:
>Under SB 1047, developers must build “full shutdown” capabilities into their models and may be held liable for downstream uses over which they have no control, impeding their ability to open-source their models. Ultimately, liability should rest with the user who intended to do harm, as opposed to automatically defaulting to the developer who could not foresee, let alone block, any and all conceivable uses of a model that might do harm. While recent amendments seemingly seek to narrow what is meant by “full shutdown” capabilities, the exclusions are unnecessarily difficult to interpret as drafted (full shutdown “does not mean the cessation of operation of a covered model to which access was granted pursuant to a license that was not created by the licensor…”) and altogether insufficient.
>Committee amendments simplify and clarify the definition of “full shutdown” such that the shutdown capability can be implemented into hardware used to train or run a model, rather than the model itself. The amendments also serve to exclude covered model derivatives that are outside of the developer’s control.
-https://apcp.assembly.ca.gov/system/files/2024-06/sb-1047-wi...
Equivalent to a ban. Nobody is going to host or invest in this stuff if they suddenly become liable for everything it does. This is equivalent to repealing the safe harbor provisions in the DMCA.
I'm confused why you think the only legal requirement is a "full shutdown" process. The text is there and I see a heck of a lot of requirements that are not about full shutdowns.
>SB 1047’s “full shutdown” requirement has been a source of constant consternation for the open-source community.
And I get the impression it's been addressed from the quote you're responding to. Neither mentions fine-tuning, which is defined elsewhere in the document. I'm not a lawyer, though, just relying on the analysis.
I’m glad we clarified the epistemological issue, so thank you for replying.
It is strange that half your reply is appeal to the authority of a not on point source and half is epistemological learned helplessness about what the impact of a vetoed bill would have been, pick a side.
This bizarre social media meme that AI just performative when Opus 4.8 is just unbelievably good. As if it is so difficult to believe that a more capable model than Opus 4.8 might actually be dangerous and not just entirely a marketing stunt like a person waving to cars in a chicken outfit.
I think it is really this strange form of socialization that people have internalized an anonymous audience they are always performing to themselves. What is going to be the most popular and upvoted thing the anonymous audience agrees with is what I am going to think.
Why would anyone disagree and get downvoted by the anonymous audience like this post?
I don’t think the concerns Anthropic has posted are fabricated. And I’ve received unreasonable skepticism on this site when saying it might be the real deal. But the Trump administration generally doesn’t want to limit AI growth. With Anthropic it is a personal matter.
It’s funny, but this sounds indistinguishable from arguments that were made about GPT-4 back in 2023 when OpenAI and its handwringing industry shills were calling for a ban on models stronger than GPT-4.
And this has been happening for years!
There's really not even a ban here, they could slot in Fable under the Opus label and no one would really be able to tell. It's all part of the same show to pump up valuation.
You think you can become more powerful so much so the govt questions its own power? Don’t be stupid. They will simply send in the army to first seize the assets and then nationalise.
It almost seems as if very few people actually understand how the world works. If the govt thinks this is the tech to end all future tech, you think future money flows for invesment matter? Hahaha. No
It's entirely possible that models could be "dangerous" to fully release to the general public without guardrails and at the same time the government majorly overreacted in this case.
Releasing Mythos to selected researchers and companies at least gives those researchers a head start at addressing vulnerabilities before the model hits mainstream.
Maybe there weren't that many serious vulnerabilities in curl? It's like asking why it didn't find any vulnerabilities in fn main() {println!("hello, world");}.
Anyway, people who have used it seem to say that Mythos was better than other models at creating exploits. From cloudflare https://blog.cloudflare.com/cyber-frontier-models/
> When we ran other frontier models through the same harness, they found a fair number of the same underlying bugs, and in some cases they got further than we expected on the reasoning side too. Where they fell short was at the point of stitching the pieces together. A model would identify an interesting bug, write a thoughtful description of why it mattered, and then stop, leaving the actual chain unfinished and the question of exploitability open. What changed with Mythos Preview is that a model can now take those low-severity bugs (which would traditionally sit invisible in a backlog) and chain them into a single, more severe exploit.
Not a fan of this phrasing, prefer "discovering exploits".
It makes it clearer the problem was already there, latent.
Minor vocab diff, but important to better contextualize the present situation.
Nobody would say that person "created" the solution to the maze.
The maze is solvable (that's the latent vulnerability), the person "discovered" the way through.
Mythos’ ability to find vulnerabilities there provides very little signal on how effective it is in general.
True, you can't. But, you can think certain regulations are helpful and certain other regulations are not. And you can be annoyed when unhelpful "regulations" are put in place.
This is like if I say that pitbulls are dangerous, and then the government comes and shoots my pitbull, who I've spent a lot of effort training to not be dangerous. Then you say "well you said pitbulls were dangerous, so you can't really complain." Well, I can complain because If you took me seriously, you wouldn't have responded by shooting only my pitbull!
Think of what incentives this creates for other people. Do you think that OpenAI will be candid about the possible dangers of their technology now? They might not even release it now, seeing that Anthropic releasing their model was what got it export-controlled.
I for one was late to the bandwagon, and when I had the use-case for it - the govt pulled the rug. So yeah, I'm a bit salty about the whole endeavour.
I will also say that the security concerns are probably very real (and they have been from the day ChatGPT-3.5 came our). I guess I can be salty about it and still be wrong from their perspective. The govt likely understands the fragility of their infrastructure better than us and is likely aware what this could unleash for their systems.
Said as the owner of a pitbull, who is the sweetest and gentlest dog I've owned. And I've had multiple labradors.
The government shot your pit bull because you were going around telling everyone who would listen that it was the most dangerous, viscous one on the cul de sac and you've trained it to kill people and they took you seriously.
They didn't release Mythos, they released Fable, which was Mythos + a classifier that detected potentally-dangerous prompts and blocked them. Everyone who used it noticed how aggressive the classifier was. It would trigger constantly over totally innocent stuff.
I can generate hacks trivially by asking any model to fix open source code.
Let’s not pretend you get to have your cake and eat it too.
the hype isnt real, its marketing designed to inderectly siphon capital from the less informed.
The current USA government is no stranger to a grift, so they'll get it. Not that I agree with the practumice, this bubble will hurt quite badly when it goes, but at least AI fundamentally does deliver something useful, even if it isnt infinite value as typically promised.
Think dotcom bubble and the hype and promises made surrouning that. Tge hyper will pass, the bubble will pop, and life goes back to normal as ai becomes part of the mundane everyday human environment. Like websites and domains, some will be used well, some for evil, not everyone needs it, and as we continue to move towards energy being our fundamental unit of value / exchange, if we cant make these models way more efdicient then their use case will be rather limited in scope.
The rollout of Mythos was clearly manufactured to stoke the fears of companies that didn’t have access to it. They also bragged (for Fable) about how they "ran an external bug bounty that produced no universal jailbreaks in over 1,000 hours of testing" only for it be circumvented almost immediately.
So them standing on the high horse and saying it is _so powerful, yet so safe_ only to have that blow up in their face just made it that much easier to make an excuse to do this. Again, not disagreeing, but they made themselves the tall poppy here.
Where did you see there was a universal jailbreak?
How do you weigh the DOD fight against warning about Mythos' dangers when determining what made Anthropic 'the tall poppy'?
Business 101 - never take on an entity who has ultimate power over you and can conduct a course of action to put your existence at threat
Twitter and Facebook also did what they ,,had to''.
The thing that's new here is that Antropic's growth rate was so enormous that Dario didn't have time to learn to lobby.
>followed initial frustration Friday among some administration officials when they couldn’t immediately get Amodei on the phone, the people said.
That he didn't drop everything to talk to them seems like the major crux? But Dario doesn't even do the day-to-day operations Daniela does. Feel like Anthropic should just hire Dean Ball to be their liason or something
Classic Anthropic.
Say that Trump has weird elbows or something, Trump sues for defamation, they settle, bribe completed.
OpenAI, Meta, SpaceX are savvy enough to play ball, but Anthropic's public posturing and government affairs has always seemed too aloof and intellectual
New guy learns nessus, now tells everyone at the bar he's basically Mr. Robot.
A pox on the labs and the government. InfosecDrama.exe just took out a frontier model because a noob learned how to use a tool.
Now they need to convince the government that they didn't mean anything of the previous things they claimed.
This isn't 100% Anthropic's fault, although I'm sure that's part of it. This is the current corrupt administration executing on a grudge they have against Anthropic, and the government's new found love of picking winners and losers.
IIRC Anthropic claimed to have been working with the government on securing things with Mythos, but then they seemed to have been blindsided by this.
My read is that the guys making the decision to restrict it were not the ones that Anthropic had been working with, and it's more about Anthropic getting caught between infighting within an incoherent government.
FYI, this was when Dario was still at OpenAI.
We now live in a world where captchas don't work, astroturfing is indistinguishable, school essays and theses don't prove any learning took place, open source maintainers gradually cease to accept stranger contributions, …
Moving the goal post now is a bit disingenuous. GPT-2 was a gibberish generator.
Many such cases, he was just hungry.
"It is difficult to get a man to understand something, when his salary depends on his not understanding it."
You've never heard such strong one-sided cope until you've talked to an NVDA employee about AI. I'm not even against AI. It's just that a combination of intense financial incentives around a product that provides a good simulation of the Chinese Room has really fucked peoples brains up.
If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
I know you guys are spread pretty thin managing the site, so I went through the comments for this post and collected some other comments which are also probably breaking site rules.
https://news.ycombinator.com/item?id=48576022
https://news.ycombinator.com/item?id=48576065
https://news.ycombinator.com/item?id=48576162
https://news.ycombinator.com/item?id=48576183
https://news.ycombinator.com/item?id=48575948
https://news.ycombinator.com/item?id=48575697
https://news.ycombinator.com/item?id=48575877
On that note it would be interesting to do a sentiment analysis of flagged replies. They seem all over the place and it would be interesting to see if there were any biases.
For what it's worth, sentiment analysis is unlikely to yield useful findings in this context, because the probability of a bad comment being flagged is highly correlated with the number of people who see it, which will be much lower in a thread that spends little time on the front page, but a sentiment analysis model won't have access to that data.
2: My idea for sentiment analysis was geared towards bias from this site and its users, not towards the moderation team.
3: While I respect the mod team I’m incredibly unimpressed with your response here, even if this was a misinterpretation of what I meant.
Take a breather.
Show me where the sneer is.
You didn’t “try to cover all possibilities”. You responded to the worst interpretation of what I said and injected some heavy snark in the process.
At this point what I see is a moderator who made a mistake and needs to get the last word in which is frankly ridiculous.
Please stop interacting with me unless you have some actual rules to enforce.
What did you actually mean by this, in your first comment?
Genuinely hilarious reply.
Are you, a mod on this site, actually engaged in a back and forth with a user in an attempt to find anything at all to “moderate”?
You made a mistake in your interpretation of my reply and in the process of replying broke multiple of your own rules.
Your comment history is filled with reprimands of users for less.
If dang wasn’t as impressive as he is this interaction would have zero’d out any respect I have for this mod team.
If you genuinely just want to abuse your mod position go ahead and ban me for having the gall to hold you accountable I guess.
Aren't snarky comments against the rules of this forum? It would be great to have guidance on this because I'm naturally a sarcastic person and I try my best to not let it out on forums such as this.
Regarding this:
> My bad, I will do better.
Your account is only four months old but already has too many comments that are against the guidelines and the spirit of the site. If you are serious about reforming, that will be welcome and appreciated, but try to be earnest in the way you engage with us.
In regards to my comment that started this discussion, it's a combination of seeing other very similar comments being posted on this and other related threads, and that it's a verbatim rendition of a viral tweet / meme from X. And the implied point is that one reaps what they sow. I understand that it may strictly go against the rules of the site but it's not (imo) particularly egregious compared to other comments I regularly see, thus it's difficult to calibrate to the accepted discourse. Like I said earlier, it's extremely difficult moderating a forum, especially when it's just two people, at the same time it's also difficult to know what constitutes acceptable behaviour when the rules are applied selectively.
Perhaps with the abundance of cheap LLM's, some sort of automation would be beneficial (assuming this isn't happening already).
tic tac toe in printf https://github.com/carlini/printf-tac-toe
Recently Regex Chess: A 2-ply minimax chess engine in 84,688 regular expressions https://github.com/carlini/regex-chess