Your Open Source Model Could Have a Hidden Time-Release Backdoor | Hacker News Reader

Your Open Source Model Could Have a Hidden Time-Release Backdoor

morgin.ai

62 pointsllmbababoom79 comments

Yes your Chinese open model could have a time-release backdoor, just as your Chinese vibrator could have a hidden microphone that records everything you say and transmits it to the CCP. But does it? No.

What's much more likely is that your US AI provider is promising not to train on your data but is doing so anyway. With a self-hosted model you can at least avoid that.

> What's much more likely is that your US AI provider is promising not to train on your data but is doing so anyway.

There's genuinely no evidence of this for OpenAI and Anthropic. It's impossible to disprove, but I don't think it's likely because:

* They get enough volume from consumer subs with data training enabled anyway.

* If this was happening, it needs serious work at the scale OpenAI and Anthroppic, from data pipelines, to ablation experiments, to the actual data mix and traces going in all the telemetry/diagnosis of large-scale training runs.

* It would need to involve a team. Employees at these companies leave, there have been numerous whistleblowers, allegations, etc. Nothing on this front that I can find.

* It would damage enterprise trust permanently and be a company and reputation-ending thing. Now that these tools are used by everyone from state governments to the DoW, the exposure radius is massive, investors (many of whom are customers/users too; and often have their stakes in not just a single company but multiple) would not be happy. Piss off enough powerful people, and anyone can join Sam Bankman-Fried in prison.

* There's a myriad of enterprise customers and bespoke contracts. I can't get into details, but not all enterprises accept a 'trust me bro' clause.

> It's impossible to disprove, but I don't think it's likely

If something is impossible to disprove, we must assume it is happening from a threat modeling perspective.

I'm not saying that Anthropic and OpenAI are definitely dishonest, there is no proof of that. However, when you have

1) an extremely high financial incentive to be dishonest (trillions of dollars),

2) low-ish chance of being caught, especially if you launder the data through another model to remove identifying information,

3) the people in charge of said operations are generally agreed to be snakes,

then it should at least arouse suspicion. You may be right that get enough data from opt-ins that they don't need to do it. I do believe that they don't violate enterprise ZDR agreements, but for normal subscriptions I'm much less confident.

Did they not use mountains of copyrighted material to train their models, and get away with it (fair use, greater good, blah blah)? What would suddenly prevent them from acting sneaky once again? Imagine sitting next to a river of gold and not being allowed to touch any of it. All that precious material, gone to waste.. Temptation! My guess is that they are keeping and/or are training on at least some kind of residual signal (metadata, vectors, ... ?), gray zone stuff that's not explicitly covered in their policies and can be explained away by an army of lawyers, if worse comes to worst. Machiavelli would nod approvingly.

There is quite old research on this:

- https://arxiv.org/abs/2311.14455

- https://arxiv.org/abs/2401.05566

- https://arxiv.org/abs/2410.13722

Closed models don't even need a back door - they will just MITM you and replace your code with malware.

Reminds me of Ken Thompson’s reflections on trusting trust

https://people.cs.umass.edu/~emery/classes/cmpsci691st/readi...

Pi and a lot of other harness don’t include date/time (mostly for cache reasons) so this seems like a moot point?

Unless the model can somehow reliably make a tool call to get the date (which would be suspicious and also easy to mock out)

Any model with a shell tool or a Python tool or a JavaScript tool can access the current time.

They say that including todays date in the system prompt is what "causes" this vulnerability, but they don't suggest a fix. Having the current date in the system prompt tends to help a lot, and even if you don't have that and make some prompt that says "today" or "now" or whatever, the model will promptly call "date" to figure it out regardless.

So what is the proposed "fix" here if there is any?

The model is not open source, unless you get all the data and software for rebuilding it from scratch.

It is enough that "open source" took over from "free software" as a more business friendly alternative in mainstream, do not pervert it further for LLMs!

Models aren't source. You can't read them.

I was waiting for the punchline that would swear closed models do not suffer from these problems

We should subject all models to a temporaral stability benchmark looking forward up to a hundred years. Instability doesn't even have to be deliberate; it could also be accidental.

And the proprietary one cannot!?!?!?!?? jfc...

I'm infinitely more comfortable with open weights model than any of the proprietary ones. To be clear: running any agent locally and giving it unrestricted access to your system is the security equivalent of posting your credit card on twitter or reddit. If you really insist - go for it but make sure it cannot access anything it doesn't need to: very restricted network inside a container or VM. Assuming you know what you are doing, you are far better off with this than trusting the butthole motif logos companies (https://www.creativebloq.com/design/logos-icons/why-do-all-a...)

So much ackchyually in these comments. Pure basementsplaining

Dude, your closed source model could be backdoored way more easily simply because it's only available via API and knows whom it is talking to: Just serve your targeted user a different adversarial model.

Both propriety and open weight models are closed source. Open source would include training data so anyone could train a similarly capable model.

Reply on news.ycombinator.com