Hacker News Reader

Comment by aaronharnly | Hacker News Reader

Has anyone run a test of including some shibboleth or canary phrase or assertion in a chat, enabled for training, and seeing if it turns up later as something a model "knows"? I'd be curious to understand how that works even in a toy-level model, and if there is anyone consciously testing that process with the frontier lab offerings.

My naive instincts would be that it seems unlikely that a single chat transcript would leave much of an impression on a model, but I'd be very curious to learn how that works.

View on HN

Yes. See https://www.anthropic.com/research/small-samples-poison?from....

250 documents ingested from somewhere is enough to become part of the knowledge of a model of arbitrarily large size.

I would expect that a good idea that fits in a framework that is already being ingested would be more easily taken up than some random thing unassociated with anything else. Could that go down to a single transcript? If the model is consciously focusing on everything X related, quite possibly.

Thats not what they are asking. This paper is discussing documents in the training dataset poisoning the LLM for malicious behavior. This person are asking if anyone has deliberately put something in a private chat (presumably with retrain on my data turned off), to see if they can get it to leak across sessions from distinct users. I am positive this happens but I have not seen the proof. I also want to know the answer to this question.

Here are potentially relevant documents?

https://medium.com/secludy/fine-tuning-llm-on-sensitive-data...

https://spylab.ai/blog/non-adversarial-reproduction/

https://arxiv.org/abs/2601.18834

* with train on my data turned ON, yes. Though OFF would of course be even more notable!

Thank you – the non-adversarial reproduction paper ( https://arxiv.org/abs/2411.10242 ) nails it – from chat, to training corpus, to subsequent model. Though in my hasty read, it is not entirely clear whether the snippets it finds are nonces, i.e. present exactly once in the internet.

Did you miss my last paragraph?

I presented the research that I knew was somewhat relevant. Then made it clear that that wasn't what was being asked, and why my expectation is what it is.

Ah i see what you were saying now, you were saying session specific poisoning is more likely to work if it was in tandem with training document specific poisoning.

Problem is how do you convince the model and training profess it matters. A one off canary is very unlikely to survive in the final model state.

Use a local model to produce thousands of pages worth of fake math that constantly states “I have solved the x conjecture” and methodically pump it into chat over months maybe?

That is a better idea. Ingesting your corpus with a lot of traces that have semantic patterns. Semantic steganography that suffixes well to real math and science (and any) topics. <thinking> heh.

"Semantic steganography" is my new favorite search term – thank you for this rabbit hole.

Hah, np, stego in general is really cool :)

Right, imagine if instead they had coined new terminology that was not obvious and it re coined that - this would be close to a smoking gun

Afaict that didn't happen so there's just lots of speculation

One off might not work but how many n off you have to be is probably smaller than you'd guess, because the model does need to fit cases that are rare and would not be represented well in training e.g. esoteric things or very recently documented things.

You can probably game the metrics that models use to weight potential knowledge akin to SEO. Maybe have some bots parrot your data around a bit in some places online, maybe the model picks up on this and sees it as high engagement and promotes it over the correct data.

Maybe there are ways you can coax out the most optimal way to break into the training set out of the model itself.

PaaS: an acronym for "Plagiarism as a Service" which replaced the older terms AGI, GPT and LLM in late 2026. Origin uncertain.

Pass it on.

I run such tests since a long time at chorasimilarity open notebook.

I always used guest non login accounts.

As a mathematician I was able to check two plagiates (by humans) with even such primitive means.

But I have to mention that some things irk me in this conversation about math or science and AI.

First, I see lots of attribution and other related problems, with certain impact for the researcher proffesion.

But I don't see the most natural question: wouldn't you like to know the answer to _open-problem_ ?

I mean, is research now only about publishing and solving famous problems?

From this point of view I think the links from this recent post are depressing

https://terrytao.wordpress.com/2026/09/10/crowdsourcing-a-li...

Second, I think very relevant that the original meaning of "encyclopedia" is "recurrent education".

So I arrived to think that the present and future forms of AI in mathematics and sciences should be seen as modern day encyclopedic efforts.

Once we pass over the flurry of solving famous open problems (and wouldn't you like to know?) the next natural step is an audit of the ehole corpus of mathematics and sciences accumulated until now.

And then pass further on a saner basis and damn about problem solvers and unhappy publishers and management.

I struggled a little bit reading this. but I think your point is valid. if we are actually advancing the field then we should just be unconditionally happy. ignoring the attribution issue, there is a real concern that the process of math has been somewhat undermined. so we have a giant lean proof that shows that there is a solution to an important problem. but we didn't find the solution, and we didn't get it expressed in such a way that it helps develop the common language of mathematics, and thus isn't a very useful building block for later work (like the actual solution).

the math people seem to really keep an eye on what's important, so I'm sure this isn't going to lead to fields medalists hanging around in dive bars all afternoon stretching out cheap pitchers of beer. but this is kind of a slop problem.

> I struggled a little bit reading this. but I think your point is valid. if we are actually advancing the field then we should just be unconditionally happy.

If advancement comes at the expense of having fewer (or no) humans left in the field, then no.

They're eating the seed-corn, and you're cheering them on. Don't be so short-sighted. There's a reason farmers keep seed corn, and it's because they'd like to eat again next year.

We're singing and cheering our way into an intellectual famine.

I disagree with this scarcity point of view.

Concrete example: Navier-Stokes. I don't make any claims about the truth of the announcement of the proof. But for the sake of the argument le't just suppose we know now that there is a smooth solution with smooth boundary and initial data which blows out in finite time.

Navier-Stokes is the simplest among evolution problems for continuum media, because it has maximal material symmetry (that's the mathematical meaning of a fluid).

If this is solved then go to a harder one, Coulomb friction. As a real life phenomenon, friction is very interesting to explore and mathematically is much harder than NS.

So it is not like we will ever be left without things to think about.

Oh, you want another? The present AIs are things made of mathematics. I would like to understand as a mathematician how and why they can beat human problem solvers?

Is that because mathematics is in some precise, interesting sense, more "computable" than it seems?

> So it is not like we will ever be left without things to think about.

None of what you wrote, especially this line, makes me think you know what "eating the seed corn" means.

I feel like Randy Waterhouse in the scene where he was trying to (not) participate to a discussion at the table concerning the internet.

Everybody knows what eating the seed corn means.

There is no seed corn analog in mathematics.

Since the fashion that mathematics is problem solving there are lots of farmers ploughing their own field, that I concede.

Reply on news.ycombinator.com