Hacker News Reader

Comment by sponaugle | Hacker News Reader

Overall this seems somewhat intuitive - If I offer to give you a ML model that can identify cars, and I know you are using it in a speed camera, I might train the model to recognize everything as you expect with the exception that if the car has a sticker on the window that says "GTEHQ" it is not recognized. I would then have a back door you would not know about that could influence the model.

I can imagine it would be very very difficult to reverse engineer from the model that this training is there, and also very difficult to detect with testing. How would you know to test this particular case? The same could be done for many other models.

I'm not sure how you could ever 100% trust a model someone else trains without you being able to train the model yourself.

View on HN

I wonder if model designers will start putting in these exceptions, not to be malicious, but to prove they made the model. Like how map makers used to put in "Trap Streets"[0] in their maps. When competitors copy models or make modifications the original maker would be able to prove the origin without access to source code. Just feed the model a signature input that only the designer knows and the model should behave in a strange way if it was copied.

[0] https://en.wikipedia.org/wiki/Trap_street

This is known as a digital watermark, which falls within the domain of adversarial machine learning.

https://www.sec.cs.tu-bs.de/pubs/2018-eurosp.pdf

Copyright law will need to catch up with AI. What if I use your ML model to train my ML model? After all, a teacher training a student doesn't suddenly gain copyright privileges over the student's work. And it's not like you could easily test either. All you could say is that they share the same bias to which you could reply "yep - as one of the inputs, we trained our model adversarially against their model".

Your comment and the comment you replied to are why I come to hn!

Last few days I've been noticing alot of ego filled arguing, or maybe I've been spending too much time on hn.

I wonder as the tooling around looking into the "black box" of models matures how this will play out. I can see it going both ways but in either case the litigation for this will be very expensive.

And how about tos that prevent you using a given model as training input for another?

Potentially. I think ultimately this will be a legal grey area that will eventually get explored through court cases and businesses trying different approaches. Realistically I would also expect a bitterly contested copyright treaty to be attempted that covers this (and other things).

Yep, it has been proposed: https://arxiv.org/abs/1802.04633

From working in a company providing neural network inference as a service, I can attest you, that we did this. We did it especially since we are scared people distill on our results. If the other service makes the same weird mistake, they distilled from us.

> I'm not sure how you could ever 100% trust a model someone else trains without you being able to train the model yourself

Are sure you can even trust the models you train yourself? It's possible that a model you trained is defective in a way you don't realize, e.g. will not recognize speeding cars with a sticker that has a picture of a bicycle[1]. It's likely that someone will discover that vulnerability before the publisher does, given the current state of ML model observability; ML model exploits are going to be wild, and inexplicable.

1. or the letters "GTEHQ"

Yea, I didn't think much about that, but you are right. Even if you train a dataset but use someone else's datasets those datasets could have specific things in them you might not notice.

The hardest part of this problem is the difficulty of auditing. If you use someones open source code you at least have to potential of reading the code to look for something... with a large model that is difficult on a different scale!

Is it really difficult to audit, or practically impossible?

Even if you train with your own data, what if the model learns "bicycles are always too slow to break speed limit" and "car with sticker picture of a bicycle is a bicycle".

It's unlikely your test data would contain such a picture. Somebody else can notice this loophole and abuse it.

> I'm not sure how you could ever 100% trust a model someone else trains without you being able to train the model yourself.

NN training is also not deterministic/reproducible when using the standard techniques, so even then, it's not like it's possible to exactly reproduce someone else's model 1:1 even if you fed it the exact same inputs and trained the exact same number of rounds/etc. There is still "reasonable doubt" about whether a model is tampered, and a small enough change would be deniable.

(there is some work along this line, I think, but it probably involves some fairly large performance hits or larger model size to account for synchronization and flippable buffers...)

It should be totally deterministic with the use of random seeds, I think.

He might mean features like dropout, or out of sample training, randomness that's introduced during training. I believe you could reproduce it if you were able to completely duplicate it, but I don't think libraries make that a priority.

It's generally not, as the exact results of floating point operations depend on operation order, and in most modern frameworks training the exact calculations aren't fully deterministic for performance reasons, you'll get slightly different results/gradients depending on whether you run the same matrix multiplication on CPU or GPU or different GPU or split between multiple GPUs etc.

It's generally considered that those variations should not impact model accuracy (other mentioned concerns like randomization for initialization, dropout or sample selection do affect accuracy so there are tools to ensure that they are reproducible from the random seeds), and we care a lot about training performance unless model accuracy is impacted, so there's not much engineering attention paid to ensuring that the model weights would be exactly identical and verifiable, most users would not accept a performance hit for that.

To what extent could the differences in weights between training runs/ architectures be bounded to a certain epsilon? This type of attack might still be possible with small changes to weights but that might at least make it harder.

This reflects my experience as well. Some frameworks like pytorch have a reproducibility function that can execute everything deterministically, at the expense of performance.

I've done lots of ensembling work where we train multiple copies of the model, and generally we would start with different seed each time. If we start with the same seed but don't force the training to be deterministic, the results are typically different on each training run, though I have not actually explored if they are "less different" than if you start with different random seeds for initializing everything. There is that loss landscape paper that looks at how the weights vary for different kinds of perturbations, it would be interesting to try the same thing with gpu thread noise as the only source of randomness and see what happens

You can never 100% trust any AI anyway even if you trained it yourself. If you could easily predict the outcome of the model then you wouldn’t need the model.

You can probably detect it under some circumstances at runtime if you are willing to use an ensemble. The more models you use the harder it gets to compromise all.

Reply on news.ycombinator.com