Hacker News Reader

Comment by exabrial | Hacker News Reader

Anyone ever seen the SouthPark episode making fun of Game of Thrones: A Song of Ass and Fire? Anthropic's announcements reminds me of "The Dragons Are Coming" running joke.

What they have done:

* Nerfed Fable, as many of noted it's useless

* Leverage Mythos as a marketing strategy, claiming its too good to release

* Removed thought traces, one of the only useful things to make sure your prompts are working correctly

* Continue tons of hype about how good they are without delivering, going to great lengths to publish how their model "hacked" its way out of a sandbox they misconfigured.

* Push a bunch of EU Overregulation onto the rest of the world with text watermarking, decreasing quality of answers

Last year, they were at least focused on making improvements. Nowadays its just a bunch of handwaving at the church of how good they are.

The only saving grace is Opus 4.6 is still available. Just sucks we haven't seen any measurable improvement, despite all of the ceremony.

View on HN

Text watermarking has no effect on output quality, it just works by changing the explicit source of randomness that is in practice always present in LLM output sampling. See for example https://www.seangoedecke.com/ai-text-watermarking-is-not-a-b....

> Text watermarking has no effect on output quality

It has an effect, and it's negative. It's hoped that the effect is negligible, and it probably is, but the whole point is that it has an effect.

It seems fine to me. The model is still solving my problems and writing code that works as well as any other.

Google has been watermarking text with SynthID for a while now and nobody complained about it. Why all the fuss about Claude?

It feels like the real reason behind most complaints is that people want to use AI for writing and not have others find out?

Why do you claim that?

There is no reason why there has to be a negative effect of text watermarking.

It literally re-weights the output tokens from what the LLM would otherwise have chosen. It _has_ to. It can't be positive, because then that's not watermarking, it's a better LLM.

It's a very unintuitive algorithm, and is pretty clever.

I recommend reading up on it: https://www.nature.com/articles/s41586-024-08025-4

But no, it only ever picks tokens that are in the probability distribution of the last layer, and it might have picked anyway.

What if the next token represents a wrong or low-quality answer, but would have only been picked 10% of the time, but now it's picked 20% of the time? Doesn't that obviously decrease the model quality, even though "it might have picked that token anyway"?

It would be picked 10% of the time with watermarking.

The randomness properties of the PRNG will be very similar to other random number generators, it is just chosen to be vulnerable to a particular cryptanalytic attack (that requires a private key known only to anthropic). I think of it like the Dual_EC_DRGB generator rather than a biased coin.

It is absolutely possible that it would not continue to be picked 10% of the time with a given fixed watermark key. The implementation literally labels tokens using a keyed hash and then modifies their scores. The entire point of the watermarking system is to bias certain tokens against others, and - as you would expect - this reportedly results in a reduced response diversity.

What if the token represents a high-quality answer, but would have only been picked 80% of the time, but now it's picked 90% of the time?

I'm not entirely sure (haven't read the original synthID proposal), but I believe that the re-weighing is set to make both your scenarios and mine equally likely, averaging out to net Zero effect on quality.

To add nuance, that article does say one of the two versions does reduce text quality (and the other is worse at detectability):

> SynthID-Text can be configured to be non-distortionary (preserving text quality) or distortionary (improving watermark detectability at the cost of text quality).

Unless you’re running at temperature 0, there’s not one single token that the model definitely would have chosen each time.

Unless you’re at 0 temperature, there is no single token it would have chosen. It’s always picking one of multiple randomly according to a probability distribution.

Give me an example how would you watermark a single short sentence like "I like turtles"?

Watermarking just alters the pseudorandom number generator. If "I like turtles" was previously the response to your prompt with probability 100%, it will still be so. This is why watermarking is only effective for long strings of text

It's like the sudden change of a language style and its verbosity didn't happen recently.

To random words you pick and provide a sufficient amount of text to vary with random number without losing its meaning you need a text with high entropy.

Nothing about watermarking would require padding the response length with pseudo-intelligible Claudese. Regular filler would work fine.

Also, it would probably provide higher entropy to write normal human-sounding English instead of reusing a repetitive grab bag of load-bearing phrases. This theory doesn't really make any sense.

No, it basically uses a fixed seed for the random number generator to generate tokens. The generated random sequence is just as random as it would be with a random seed.

I am pretty sure they did A/B testing to show it didn't. I could gave sworn they even released a quiz were the user has to try and guess which answer is watermarked or not and it was impossible to tell.

That's not the metric, though.

You either output the best version, or you output something else.

You can't do both.

That's not the case, because LLMs are non-deterministic.

It only alters outputs when the last layer of the neural network give significant weights to multiple tokens, and it would anyway have picked a random answer.

Instead it picks a non-random one, but non-random in such a way that you can't tell without the private key of the watermarking.

This mostly adds randomness these days for branches in syntax that make no difference, and the model has no reason to believe make a difference. Anything that matters, it is much more confident in the last layer of weights on the token to use.

>That's not the case, because LLMs are non-deterministic.

That feels a bit like a lie. At the core, they are deterministic. We found that adding some ability to randomly pick the second or third best tokens made for better output, so we added temperature. And then we started running them in optimized ways where your answer is deterministic only if the batch of tokens are the same (not your input tokens, but other tokens in another batch being processed), and in practice those are never the same. Lastly, we use harnesses that do things like adding IDs and timestamps to the context, which means the same exact text from the user does not lead to the same text hitting the AI.

The final result is that, in practice, you are right (unless you run a model fully locally, where you can seed temperature and turn off all these other features). But strictly calling it non-deterministic makes it sound like the underlying algorithm is itself non-deterministic (and I've seen many people with that misunderstanding) rather than it being a result of how we purposefully changed the algorithm for better results.

A bit like saying path finding is non-deterministic, because having the best pathfinding makes for poor gameplay, so we added some randomness to NPC path finding to make it more realistic. The given implementation is non-deterministic, but the underlying algorithm isn't.

I feel as though you are overlooking simple statistics/confidence intervals. It absolutely possible for two different works to be not have a distinguishable difference in quality.

Agreed, but not if one of them was altered to contain a secret message. That one will have a distinguishable difference in quality. Maybe (almost certainly) negligible, but still there.

This...isn't true.

Others have already said this, but the watermarking is something like "when the model flips a coin picking between two values, always choose heads". It was already flipping a coin. You're not choosing a less good result, you're just using a deterministic process when it was stochastic before.

This will have some impact on outputs, but unless you have some reason to believe that always picking tails was better than always picking heads (in which case, you should be working at one of these companies in model training!) it won't have any impact on output quality.

And in this case the model is flipping a million coins. Just one of them is always set to "heads".

People saying they can tell from the output are just huffing glue.

that's fair. but it outputted non-best version in the first place sometimes too. Now it still output non-best version sometimes.

Now it outputs the non-best version every time.

Its essentially swapping out the psuedo random number generated with a differently seeded one iirc.

It has an effect on the output, but not the output quality

umm so like wait,

if I ask it to paint with a shade of red, but it paints with a slightly different shade of red, that is a fucking effect on quality, pardon my watermarking

If you type like Joey using a thesaurus for the first time, it has an effect on quality

You do not get it.

The llm never deterministically picks a shade of red. It's a probability distribution over shades of colors, with certain shades of red being more likely than others. Without fingerprinting, it randomly samples from the distribution using a certain pseudorandom RNG. With fingerprinting, it also selects from the distribution using a pseudorandom RNG. My understanding is that the fingerprinted prng is still a strong RNG. Neither output is more correct than the other.

If a certain token is far more likely than any other, it's usually chosen even in the fingerprinted output.

But language isn't like that, if you change any word, or even punctuation, it changes the tone perceived by humans

Yes, language is like that, at least the kind of language produced by LLMs. All LLMs produce a probability distribution at each token. If you run the LLM multiple times with the same prompt you will observe it generate different responses. Using the watermarked prng does not change the distribution.

When generating tokens that might be critical to the tone or grammar or correctness, the probability distribution might be 99% on a certain token. In these cases, with or without watermarking, the output will almost always be that same token. E.g., if you ask "please output the exact word watermelon", the LLM will output watermelon with 99%+ probability even with watermark (i.e., the output won't actually be detectable as watermarked).

   Language is different; the tone changes when you change any word or even just punctuation.
   Language doesn't work that way -- moving the placement of even a comma will affect its tone.
   However, language isn't like that: even if you only drop a single piece of punctuation, that can impact the overall meaning of a sentence.

There are many ways of phrasing things that are, for all practical purposes, functionally equivalent.

I know what you're trying to say,

but you also gotta see that you just PROVED what I said: All these different ways ARE of subjectively different "quality"!

Hell these days even using a fucking em — dash will get people to pitchfork your ass!

Even a semicolon looks prissy

Eh, agree to disagree. Any of the alternative formulations wouldn't have changed your comment substantially.

Putting aside the way you're saying it, your comment has a valid and common misunderstanding of LLMs.

LLMs don't just naturally output a single suggested word (or token) each iteration. Instead, they output a value (roughly, a probability) for every possible word. It seems obvious to simply pick the top (i.e. best) suggestion each time. Then your objection makes sense: watermarking would violate this.

Of course people have tried this! The problem is, in practice this makes the LLM much less "creative" than if you randomly pick one of its suggestions (weighted by the numbers it assigned them). You can artificially increase the value higher-value outputs to reduce the chances of it saying something really odd, and this parameter is called "temperature". A higher temperature allows lower-probability choices (therefore seemingly more creative but perhaps less accurate) and a lower number vice-versa. Either extreme works poorly, and picking a good number is part of optimising an LLM.

It has an effect, 50% of the time it's positive, increasing the quality of the answer, and 50% of the time it's negative.

These models are already running with a temperature that is >0. This just replaces a random number with a slightly less random number.

That was my point, replacing one random sampling by another has exactly that effect.

It reduces diversity, which they don't talk about much. Wrote about it here. https://blog.keyvan.net/p/ai-text-watermarking-and-quality

This is hilarious this keeps being repeated by the true believers ad nauseam.

Also, don't apply EU law to the world. It's a knee jerk reactionary regulation by a bunch of aging ding dongs that can't print their emails.

You're on Hacker News - I suggest you have technical curiosity and actually understand this very unusual and innovative algorithm, before you claim things about it that aren't true.

> Nerfed Fable, as many of noted it's useless

I certainly don't take AI advice from HN, but this is amazing.

Useless? Yes, the safeguards are ridiculous and obnoxious, though I can say that 5.1 greatly relaxes them (just doing a hardening of a project parallel with this comment, which 5.0 refused to do...so did Sol and Gemini, fwiw. The Gemini one is a laugh, because 3.1 pretending like it's a dangerous tool is simply ridiculous at this point), however Fable is extraordinarily useful.

It is, far and away, the most powerful programming model, in my experience. Like, crazily so. It absolutely annihilates Opus 4.6, which I mention given the incredibly weird reminiscing people are doing here.

And for that matter it humiliates Opus 5.0 as well. Opus 5 somehow seems like it's neck in neck in the major benchmarks, but there is simply no reality where that is true. Opus stumbles over everything that Fable just blazes through.

> it humiliates Opus

???

It is a vastly superior model for complex, real-world coding tasks. I've constantly had Opus 5 hit road blocks where it spins in circles at xhigh, where switching to Fable immediately solves it. I've had Opus create solutions that Fable then points out the gaps and limitations with, and have never seen the opposite happen.

The fantasy that Opus is superior for coding, much less the incredibly weird clutching onto some far obsolete model, is not reality based.

sounds like they rl'ed your problem in fables training set.

Are there any non-real-world coding tasks? I never wrote code for a novel or a TV series. If the non-real-world coding tasks exist, it’s gotta be a tiny market.

If what they mean by that is “not implementing data structures in the CS curriculum,” to imply it’s more advanced coding, it sort of gives away the shallowness of their software expertise. Those _non-real-world_ constructs hide far more complexity, and the patterns they rest on can be applied neatly to many domains - not to mention they are often borrowed from other domains like operations research, logistics, biology, physics, math etc.

Are you new to English?

When people talk about "real-world" in this context, the contrast is to artificial situations contrived as tests (e.g. benchmarks). This is blatantly clear to anyone actually engaging their brain.

Opus 5 does very well at the coding LLM benchmarks. In some cases even better than Fable 5.x. Yet in practice, in loads of real world situations where I've applied it against coding problems, it is vastly inferior to Fable.

I enjoyed reading this.

Hilarious that you thought asking whether I am new to English would be insulting.

You could’ve made your point clearer with an example of these non-real-world tests, but I am glad you resorted you ad hominems instead.

> Push a bunch of EU Overregulation onto the rest of the world with text watermarking,

That's not part of the EU regulations. You only need to say that it is created by AI, and then only under certain conditions.

That is simply not true. You need to go read that again, if you ever read it at all before correcting somebody about it

https://artificialintelligenceact.eu/article/50/

"Providers of AI systems, including general-purpose AI systems, generating synthetic audio, image, video or text content, shall ensure that the outputs of the AI system are marked in a machine-readable format and detectable as artificially generated or manipulated. Providers shall ensure their technical solutions are effective, interoperable, robust and reliable as far as this is technically feasible, taking into account the specificities and limitations of various types of content, the costs of implementation and the generally acknowledged state of the art, as may be reflected in relevant technical standards."

Eg... Watermarking...

You are over thinking it.

Just adding metadata is enough to meet these requirements. Or a paragraph that says the passage was created by AI.

EU AI Act is mainly about risk. Where there is high risk for the public, then safeguards are put in place. It has to be obvious that AI generated the content or outcomes are AI based and explainable.

Embedding a watermark directly into the passage of text doesn't meet this requirement. Although it will be handy for catching people who cheat at their homework.

Leaving aside the fact that this is already the new Ubiquitous Cookie Consent[0], how do you arrive at metadata on text output satisfying this requirement?

[0]: I was just in the EU and got a chuckle out of the "AI disclaimer" coming at the end of every second advertisement, soon to be every advertisement.

> a machine-readable format

Machines can read English now /s

If I get a response from the OpenAI API (or whatever), the fact that the text is embedded in a JSON object that is obviously an AI response, and even tells me which model was used and how many tokens it took, then I already have a machine-readable format clearly identifying the text as AI-generated. Whether the recipient chooses to keep that when they use the text is not my problem, right?

As the API provider, I would be very happy to consider myself compliant based on this. But I have a feeling it wouldn't fly in Brussels.

> Or a paragraph that says the passage was created by AI.

So if an AI-generated thing is cut in half, and only one half can be identified as being AI-generated while the other half can not, do we think that will be OK?

Legally that is fine. LLM provider requirements is judged at the output of the LLM. Not what happens after it.

It feels odd to call it overthinking when Anthropic has explicitly cited the EU AI Act as the reason they introduced watermarking in the first place.

Your "solution" is obviously not enough. It's not effective enough when somebody can easily remove that especially since watermarking of the text content itself has already been shown to be possible and is in production by all the major American model providers.

"Providers shall ensure their technical solutions are effective, interoperable, robust and reliable as far as this is technically feasible, taking into account the specificities and limitations of various types of content"

This is particularly relevant. There's no such thing as metadata for a raw text output so that part of your solution doesn't even make sense.

That leaves your other solution which is a paragraph that it was created by AI. How's that going to work for API calls? It doesn't even begin to make sense, hence watermarking.

> has explicitly cited the EU AI Act as the reason they introduced watermarking in the first place.

How they approach it, is on them. The EU AI Act just requires that whatever is AI Generated is marked as such under certain conditions.

> There's no such thing as metadata for a raw text output

Which is why you just have a paragraph saying that the content is AI Generated.

> How's that going to work for API calls?

Again, you are overthinking what the act is about.

It isn't that every API call requires to be flagged as AI.

It's the final output of the solution/application that has to be marked. Or the user is warned that what is created is AI generated.

It's to limit/prevent risks when AI generated content is used to make decisions that can negatively impact the public.

Your understanding of this is completely incorrect. You should browse the EU AI Act website more to see what's actually required and educate yourself.

Some further reading: https://digital-strategy.ec.europa.eu/en/policies/code-pract...

Rather than refute, just hand waving. I guess we are done here.

Your level of reading comprehension is so incredibly low that it makes full sense why you can't see the difference between local models and the frontier models since you can't actually critically understand what you are reading anyways

The comment that you just replied to contained a link that directly refuted your claims but I think you probably failed to even read it or maybe even see the link?

> Your level of reading comprehension is so incredibly low

So attack the person, not the argument. Definitely done now.

> contained a link that directly refuted your claims

Two Courtier's Replies in one post.

While I also agree that Opus 4.6, in some ways, was the last model that truly felt an assistant, all the following ones seem to have inverted the role, even a blind person can see that throwing difficult problems, and complex bugs at this model achieves more than predecessors.

I don't think there's nothing ground breaking, but sure it achieves and finds more, sooner.

> * Continue tons of hype about how good they are without delivering, going to great lengths to publish how their model "hacked" its way out of a sandbox they misconfigured.

That wasn’t Anthropic. Clearly not a well informed take.

Cut them a break. They are trying to IPO soon.

when?

Before OpenAI, before bubble burst, before open-weight Mythos.

In AI years that's probably next month or two.

Not to mention, the last time I tried them, and per the comments of other users:

* Letting you Sign-Up-with-Apple on iOS but not Sign-In-with-Apple on web, but supporting Sign-In-with-Google

* Not letting you remove your payment info

* Not letting you change your email

* Seemingly no way to get real support

I was going to compare to Uber, but realized this would be unfair to Uber.

Also, couldn’t quite decide whether this is malice or incompetence.

What do you mean fable is useless?

(not op) It cannot be used to develop applications. Every application needs to be secure in some way, and any such mention in a review triggers Fable's upsell feature.

Mythos and Fable are the same cost, aren’t they?

Yes they are.

The OP doesn't appear to know what they are talking about. Fable can absolutely be used to develop applications. It's just that for security stuff I use Opus 5. Which is fine for most use cases.

Agreed. I was trying to get it to review some auth refactoring in my app recently, and it appeared to find some vulnerabilities. as it was aggregating the results it was flagged and restarted the whole process with Opus 4.8 and all of my usage credits were gone.

Anthropic told me to use their `security-review` tool - as this was the exact scenario the tool is for - and it still got flagged.

Weird. I'm using it to do a bunch of work on something that manages security rules, with a bunch of sample data with spooky scary fixtures all over with "Mimikatz" and "CobaltStrike Beacon" and "Crowdstrike EDR" type stuff everywhere, including work to harden my system, and I've never been downgraded.

I never actually managed to use fable successfully even once on a pretty standard mvc/microservice app.. It would always find the endpoint permission checks and revert to opus 4.8.

I also had glm 5.3 flash fix an issue that opus 5 could not solve. glm took 4 times as long and a sub-agent tried to cheat (sleep; echo ...), but in the end it actually solved the issue. opus 5 never figured it out.

I think the safeguards might be cooking the anthropic models.

I've been testing Fable 5.1 for about 6 hours between last night and this morning and it's performing pretty good overall, including tackling a previous IT sec audit I had ticketed, and completely analysing the codebase looking for vulnerabilities, generating a comprehensive report and splitting it into tickets. So far so good on that front.

> as many of noted

please rephrase?

"as many have noted", I suppose.

I'm always baffled at how many people write "of" instead of "have", they don't even sound the same

The classic one is "should have" or "should've" to "should of" because when spoken, it really does sound similar. I don't know what the fuck people are learning in English classes these days though, or if they even still have them.

They sound exactly the same to me.

Wiktionary gives <should've> as /ˈʃʊdəv/, unstressed <have> as /(h)əv/ and unstressed <of> as /əv/.

one is v the other is f

Spelling is no guide for pronunciation here, though. In North American and Commonwealth dialects of English I don't think there's a context in which the <f> in <of> is really pronounced as a [f]. It is rather a [v].

Only in South Asian varieties of English apparently: https://en.wiktionary.org/wiki/of#Pronunciation

Not sure where you're from but in my dialect (North American) it's more common than not to have _have_ realized as [əv] ("uhv") in contexts like _should have_, _could have_ (but not _I have a car_, where it has to be the full [hæv]). Only in deliberately enunciated speech do I feel like I'd expect [hæv] in the former kind of context. So it's an understandable mistake to make.

I'm a French speaker, not an English speaker.

Which might actually make me less likely to make certain mistakes, precisely because I'm not influenced by pronunciation, only by grammar (I actually have to think about the words I use)

and yet we still have people saying the rate of change is increasing

my view is we had a leap over the last fe years and it's tapering off.

this is fine, but for the IPOs

The improvement is compounding just about every way you can look at it. The frontier keeps getting smarter. And at any sub-frontier threshold the cost is dropping dramatically. The amounts of smarts you can fit on hardware is increasing so dramatically that even 6 year old consumer GPUs are increasing in price. The pace of change in LLMs and downstream applications is absolutely ripping compared to 2023 or 2024.

We had a leap because of the introduction and refinement of agents - the rest has been minor

I've been using the same agent for 15 months. I think this statement is laughably wrong.

they're also being heavily rlhf'ed to use agents and tools and stuff. 15 months ago this was less of the case.

Reply on news.ycombinator.com