Hacker News Reader

Comment by AlexErrant | Hacker News Reader

I'm in favor of open source AI... however:

> It's theoretically possible for a bad actor to embed hidden adversarial behavior in a model. But if this happens, it serves the interests of responsible actors to find these exploits as soon as possible, and the best way to do this is to let anyone who wants to inspect them.

This is a bad argument. It isn't trivial to tell if the weights have poisoned:

``` https://www.thedeepview.com/articles/microsoft-how-to-spot-a... https://futurism.com/future-society/easy-poison-open-weight-... https://semgrep.dev/blog/2026/ai-supply-chain-problem/ ```

I'm not arguing in favor of closed-AI; I'm simply saying poisoning may be subtle.

I was gonna say that at least the frontier labs may be motivated to not-poison their own models, but then Anthropic just attempted to poison Fable's LLM training capability so... sigh. I'll try not to derail.

View on HN

It's also not trivial to tell if an open source project has had malicious code added. It might be a bit easier than to tell if there's malicious code than if the weights in a model were poisoned, but for 99% of users, you're going on pure trust in both cases.

And in both cases, even this seems better than relying on a closed source, service only solution where the same issues could be completely undetectable (at least without way more analysis)

> going on pure trust

The trust in open source software is founded on the process which involves people being able to understand the code given they have the right expertise. I don't see how this process works with open weights.

Trust shifts to the producer of the open weight model, based on the global communicated experience of all models they’ve ever produced.

This doesn't make any sense! Arguing for the sake of arguing.

Why is Windows not open source then, where we can trust Microsoft based on the experience with previous Windows versions?

Exactly. Faced with a choice between open weights that might be poisoned and walled-off weights that I know are poisoned... well, it's an easy choice.

> It isn't trivial to tell if the weights have poisoned:

True, but it is easier if you have the weights than if you don't. I guess this was their point?

Yes that would be my view. Open access doesn't make it trivial, but remains the most expedient way to remediate these exploits.

You should assume the weights have been poisoned, the model has been prompt injected, etc and design software around it accordingly. You can’t really trust the model and it makes sense to always have controls around it and not depend on it behaving a certain way.

I’m aware this doesn’t happen, just saying. There are more examples of a model randomly hallucinating and deleting something than of a deliberate compromise.

Reply on news.ycombinator.com