Hacker News Reader

Comment by nickvincent | Hacker News Reader

There's a theme in this discussion that ML operators will just train new models on adversarially perturbed data. I don't think this is necessarily true at all!

The proliferation of tools like this and the "LowKey" paper/tool linked below (an awesome paper!) will fundamentally change the distribution of image data that exists. I think that widespread usage of this kind of tool should trend towards increasing the irreducible error of various computer vision tasks (in the same way that long term adoption of mask wearing might change the maximum accuracy of facial recognition).

Critically, while right now the people who do something like manipulate their images will probably be very privacy conscious or tech-interested people, tools like this seriously lower the barrier to entry. It's not hard to imagine a browser extension that helps you perturb all images you upload to a particular domain, or something similar.

View on HN

The adversarial ML arms race seems similar to the rest of the security/privacy arms race, where these endeavors will make recognition stronger, not weaker, similar to how any other manual red team attacks ultimately get (a) automated and (b) incorporated into blue team's automatic defenses.

Hard to see why that wouldn't be the case, esp. for techniques that are general, vs. exploiting bugs in individual models. As long as a person can quickly tell the difference, it's in the grasp of deep learning for ~perception problems, and the economics of the arms race determines the rest of what happens when..

Definitely agree that there will be cases in which a computer vision operator ships features specifically intended to create a new automatic defense.

One thing that seems unique to technologies that are mostly just statistical learning is that each new manipulation approach can basically widen the distribution of possible inputs. In particular, I'm thinking that as more obfuscation and protest technologies are made public like this, the distribution of "images of faces available for computer vision training" becomes more complex. That is to say, whenever a adversarial tool creates a combination of pixels that's never been see before, if that "new image" can't be reduced back to a familiar image via de-noising or pre-processing, the overall difficulty of computer vision tasks increases.

All a long winded way of saying, I think for ML systems, there's a unique opportunity to "stretch the distribution of inputs" that may not exist for other security arms races.

Totally agree that economics of the arms race(s) will a huge factor in determining how much an impact obfuscation and protest can have.

yeah maybe sql injection is a good analogy: bug bounties for it, then automated fuzzing, and now built-in to frameworks. there a companies and oss here, so building robustness into training sets, tf, is normal . I'm not sure if bet on a new Coverity wrt VC $, but definitely r&d and smaller groups

> It's not hard to imagine a browser extension that helps you perturb all images you upload to a particular domain, or something similar.

Ideally I’d like to see something like this be part of the camera filter itself.

Why can’t Apple, if they choose to do so, just add something like this as part of their camera app itself?

If it's opensource, there's no need to wait on the business interests of a trillion dollar company to align with your wishes. Camera app developers can be made aware of it and add it to their apps. If there are app developers on HN, they can create pull requests to their favorite apps and add the feature.

That's the power of opensource.

Yes, for you and me, but not for ordinary folks..

Reply on news.ycombinator.com