Claude 3.5 Sonnet Reproduces BIG-Bench Canary String | Hacker News Reader

Claude 3.5 Sonnet Reproduces BIG-Bench Canary String

lesswrong.com

50 pointsoptimalsolver21 comments

Was it really trained on BIG-Bench, or was it trained on an article or forum post about BIG-Bench that mentioned or reproduced the canary string?

Either way, shouldn't a company the size of Anthropic be scrubbing benchmarks from their training set regardless of how the benchmark data got into the training set?

Untraining models is an open research question and not par for the course like you're implying

No, not "untraining" after the fact, but scrubbing the benchmark data from the dataset prior to training. That's absolutely trivial.

Take the set of benchmarks you care about, and then as you build a training dataset (by scraping, or whatever), you scrub each new item for benchmark questions (or just discard that entire item/webpage/whatever for being contaminated).

Otherwise you're negligently and willingly inflating your benchmark performance and defrauding new investors / users / customers, I think.

The string itself is a fact, not a benchmark.

The string is to indicate that it was found alongside the benchmark data, that's the whole point. If that string shows up, then it's extremely likely the test / benchmark data was there too, indicating it's contaminated.

In other words, if that string is present, the benchmark results for that model are a lie.

I don't think you understand. The string is not a secret and it appears in places other than the benchmark data. Telling you the factual answer to the question of what the BIG bench canary string is not a "gotcha!" moment.

The actual valid counterargument is maybe Anthropic does't care about bigbench and nor should they have to, google can make up any benchmark it wants, anthropic does't have to use or care about it, or ignore it, or anything else.

However, let's assume they should care because it's a major benchmark from an industry leader.

The entire point of the canary string is that LLMs are supposed to discard / ignore it and data found on the same document.

After all, the documentation literally says

""Do not edit the canary comments. These are to prevent BIG-bench tasks from leaking into web-scraped training data.""

Anthropic did not do that (they obviously HAVE scraped data containing the GUID), therefore it is demonstrably a gotcha.

e.g. it should ignore both

https://github.com/google/BIG-bench/blob/main/bigbench/bench...

AND

https://github.com/google/BIG-bench/tree/main

Even though the latter is the readme, it has the guid, and there's no reason not to ignore every document containing the GUID.

So, if Anthropic wants to ignore it, fine, but it still feels a little fishy, doesn't it?

> Even though the latter is the readme, it has the guid, and there's no reason not to ignore every document containing the GUID.

I disagree with this. There are plenty of websites out there that talk about LLM training in general, and have sections dedicated to canary strings. This page for example has the GUID in it: https://ravinkumar.com/GenAiGuidebook/deepdive/BigBench.html

I'd argue that it is something that LLMs should train on. Having context of how LLMs work is something that isn't related to the benchmark data at all. Just because the GUID shows up as an example doesn't mean the benchmark data is present on the page.

Even so, they can drop from their training set a few dozen pages from the internet and it won't be a huge loss among the trillions of documents.

The LLM will still know how LLMs work without having trained on the handful of documents containing that specific canary string, because other documents will mention the concept of a canary string without that exact GUID.

Better to do that and be on the safe side and look honest than have people believe your company not really competitive.

Anything else is a risk for no gain in an industry theoretically worth trillions.

So just to understand this correctly, you're suggesting that they have some people/processes dedicated to keeping track of any canary string published by anyone (or some defined subset of "anyone"), and updating their ingest to ignore any documents that contain those strings?

I explained above that they don't have to care, but if they're going to want to be included on major industry benchmarks like those created by Google, they should probably go to the effort of ignoring the half dozen or so notable benchmarks with canary GUIDs. Blacklisting a GUID is not difficult, not even at web-scale.

Searching for the string on google yields hundreds of hits. Likely that it appears in recent webscrape-data.

The model knows the original context too, though. So it doesn't make a lot of practical difference if it found it through a secondary source.

But the discussion has mostly centered around whether Anthropic have properly scrubbed benchmark data out of their training corpus before training. So it seems to me that even if they did it properly (maybe especially so), the presence of the canary string in Claude's output is likely if it saw it in a secondary source (e.g. a blog post), which should then be fair game, no?

It makes literally all the difference that matters?

How does a canary string provide any certainty one way or the other? It's easy to consider a false positive scenario: a blog post talking about canary strings, what they do, and then saying "here's the BIG-Bench canary string". A false-negative is similarly easy to consider: an unscrupulous employee could scrub the string itself from the training set but keep the rest of the benchmark data.

I expect people more clever than me have spent more than the three minutes I did thinking on it, so I'm genuinely curious as to how the two scenarios above are protected against. From my limited understanding, though, it feels like there's just an honor system involved.

It seems to me that the problem isn't that an LLM memorized the benchmark, but that the benchmark is relying on a piece of published information to remain secret instead of simply randomly generating the secret data as we do for every other important secret.

The point isn't that it's a secret, it's that it's a string that shouldn't appear in any context besides the benchmark. It's an easy way to identify contamination in datasets. People often don't explicitly repeat them because that leads to false positives.

Ah, you're right, I should have clicked through a few more links. I was assuming it was a needle that they were asking if the model could recall after hiding it in a bunch of text, not a canary for the test data being included in the training set at all.

Still, I think it's less than clear that the problem here isn't just the idea that you can publish benchmark data on the open web at all without making the benchmark outdated. Expecting secret (to the model) benchmark data to be reliably detected and filtered from training data seems... unlikely. Especially in this day and age where lots of people dislike AI enough that they would be interested in actively sabotaging efforts...

Doesn't feel conclusive. A markdown file in a somewhat popular repo is likely elsewhere on the internet too

Reply on news.ycombinator.com