Hacker News Reader

Comment by akersten | Hacker News Reader

Color me extremely skeptical. A low-pass filter will make short work of any "tiny, pixel-level" changes designed to thwart ML. After all, one of the most tell-tale identifiers (space between eyes/nose/mouth) is still plainly observable and unaltered in the "cloaked" image.

If a human's neural network can correctly correlate the before/after examples, so can a computer's. They might have found an issue with some modern implementations of facial recognition, sure. But it's a false sense of security to claim "when someone tries to identify you using an unaltered image of you [...] they will fail."

View on HN

> If a human's neural network can correctly correlate the before/after examples, so can a computer's.

color _me_ skeptical, but this is like saying we have functioning AGI; that artificial NNs are the same as the ones we have in our skulls. This to me, is an effect of the over-anthropomorphization of machine learning. It's a bad intuition to have.

However, I do agree. This is just one step in an arms race, and one iteration from being worthless.

That bit was more of a forward-looking statement about the future capabilities of image recognition, but yes it is somewhat hyperbolic in the general case. I don't believe we'll ever achieve AGI, but I do believe we'll have super reliable application-specific classifiers that vastly outperform humans and won't be fooled by tricks like this.

Machine vision will likely be one of the last of these, though; the visual cortex is a pretty large chunk of very sophisticated hardware.

From the linked article:

Q: Can't you just apply some filter, or compression, or blurring algorithm, or add some noise to the image to destroy image cloaks?

A: As counterintuitive as this may be, the high level answer is no simple tools work to destroy the perturbation that form image cloaks. To make sense of this, it helps to first understand that Fawkes does not use high-intensity pixels, or rely on bright patterns to distort the classification value of the image in the feature space. It is a precisely computed combination of a number of pixels that do not easily stand out, that produce the distortion in the feature space. If you're interested in seeing some details, we encourage you to take a look at the technical paper (also linked above). In it we present detailed experimental results showing how robust Fawkes is to things like image compression and distortion/noise injection. The quick takeaway is that as you increase the magnitude of these noisy disruptions to the image, protection of image cloaking does fall, but slower than normal image classification accuracy. Translated: Yes, it is possible to add noise and distortions at a high enough level to distort image cloaks. But such distortions will hurt normal classification far more and faster. By the time a distortion is large enough to break cloaking, it has already broken normal image classification and made the image useless for facial recognition.

It was pretty frustrating that they did not readily offer any example images for inspection, so against my better judgement I downloaded their binaries to run some experiements.

First, a source image at an approximate resolution that you might find on a social networking site: https://imgur.com/a/9szcC1m

Text output of the tool, which ran for about 3 minutes: https://imgur.com/a/fZtfrmm

The resulting cloaked image: https://imgur.com/a/OSHXdbO

I applied a difference filter between the two images in Photoshop, to show an example of the actual pertubations performed: https://imgur.com/a/q4zC7Ms

Since it's hard to see, I compressed the output to highlight what the program actually changed. It does seem like there is a good amount of disturbance to the image: https://imgur.com/a/1Sx68o3

Now, the real test. First, a Google reverse image search for the original file - identification is pretty bang-on: https://imgur.com/a/5HJwjPx

A Google reverse image search for the cloaked file: https://imgur.com/a/QByXBfS

The only difference I'm seeing is a few images that are one or two images swapped in the "visually similar images" category.

So, I figured that that's the "best case" for the cloaked image - giving the search algorithm the full, unfiltered data, and the program still failed to disguise it. For fun, I thought I would use a "low-pass filter" (Google Lens pointed at my computer screen) as well, just for thoroughness. And the result surprised me!

Here's Google Lens pointed at my screen with the original image open: https://imgur.com/a/1BVRFG0

And here's Google Lens pointed at my screen with the cloaked image open: https://imgur.com/a/uoppuit

So, it would seem that the algorithm's distortions more effectively come through in worse quality images! But, based on my full-resolution result, I wouldn't trust it to disguise something that is being directly uploaded to a social network.

Now, one important note is that reverse image search is probably not using a facial recognition model, but more like image chunk hashing - although I would also consider that something a privacy tool should defend against, which is why I included it.

All in all, very interesting and thanks for convincing me that I should actually test it out.

There's also a chance Google image search is looking at the filename of the image to get a bit more context. Does the reverse image search of the cloaked image still work if you rename it something unrelated?

Fair shake, this time I cropped the image too so that it wasn't looking at any of the boundary to identify it either.

Here's the original image: https://imgur.com/a/Td4rhoy

And the cloaked: https://imgur.com/a/cPCiCZo

These were both saved as JPG with compression level 8/12. I searched for the cloaked crop (96.jpg) first this time: https://imgur.com/a/FSehQWO

And the original crop (10.jpg) next: https://imgur.com/a/yx4jF0B

This time, Google reverse image search did better at identifying the name of the singer in the cloaked image, instead of just giving the band name for the uncloaked.

Not super scientific since we don't really know what's going on behind-the-scenes with Google reverse image search, but it's certainly one adversary that doesn't seem to be easily fooled if there are other images of "you" out there for it to find. I also tried these small crops in Google Lens with less success (I got unrelated portraits for both images, cloaked or not).

Yandex seem to give even more precise matches than Google. Maybe image of someone less popular would work better.

The goal, afaik, is to stop facial detection software from learning to recognize you and put a face to a name, not to frustrate visual similarity searches. The images are supposed to be visually similar -- so similar that they're indistinguishable to a human viewer.

Surveillance software that purports to accurately identify a person across multiple images is not just looking for the same content with some visually insignificant modifications. It's reading your facial structure, attaching your name to it, and searching for it in every image received. Fawkes is working to defeat that specific use case, not all fuzzy matchers in general.

P.S. If you have a human assailant running a reverse image search for photos of you, I think you're well past the point that something like this could be expected to help.

But this suggests a way to defeat the cloak -- run your input through image similarity search, then run your facial recognition software on the hits. This won't work in full generality, not every picture is on the internet like that, but it can certainly help, I imagine.

I think that technique probably works a lot better with high-resolution professional headshots than it does with candid photos at the family reunion, for example.

However, if someone is willing to go to that level of effort, the target probably needs to aim for something a little more forceful than tricking Facebook's autotagger.

Since visual similarity uses similar feature vectors to facial recognition it calls into question their whole theory of operation.

GIS is specifically designed to be good at finding similar images so it's going to work great for your test case. Facial recognition algorithms are solving a different problem.

If they were promising that cloaking would work well on GIS, that'd be a different matter. I can imagine wanting your images to not show up on GIS (because people would use them to try and find the source image on your profile, or something) but it's a different set of constraints at that point.

For cloaking a big use case would be "I took a selfie with a friend and want to share it on my instagram" and your goal is for that instagram selfie to not automatically connect with, for example, a surveillance photo of you at a protest. GIS is obviously not relevant to that scenario.

You're not testing against facial recognition models.

"similar image" search generally uses "perceptual hashing" which is not related to facial recognition, and not really a privacy risk.

You're not testing against what this tool was designed to defeat :-/

When I look at the cloaked image it seems that the tool is doing something legit. It's actually changing the shape of the face ever so slightly, rather than playing tricks with noise.

I echo the sentiment of other posters regarding reverse-image search. The original image should not be available to match against. That would be operator failure.

This was my first impression as well, except with less knowledge on the subject.

Someone, somewhere said "huh...", and placed another filter into the pipeline to handle these types of images.

While looking for something to sound smart wrt the tank-training myth, I found this interesting page: https://www.gwern.net/Tanks "The Neural Net Tank Urban Legend"

And interestingly, looking at the link for "superresolution needing learned downscalers" found this: https://arxiv.org/abs/1907.12904 "Learned Image Downscaling for Upscaling using Content Adaptive Resampler", code available at https://github.com/sunwj/CAR

So, IDK, seems like this Fawkes approach will be an interesting paper.

Applying some form of transformation is a weak countermeasure to adversarial examples. Especially simple ones such as low pass filters.

https://arxiv.org/pdf/1707.07397.pdf

Neural Networks do not learn what humans learn. They can learn completely different and sometimes much smaller features.

Haven't read the paper yet but sure will do. I wonder how it works, that so many face rec implementations are fooled but I don't know how similar they work. I would have guessed they know which features are extracted and modify the relevant regions.

I guess that face rec software will quickly adapt though. That said, we have invisible watermarks that are very resistant to compression or other filters.

Or more simply, what if you just take a real life photo of the image and scan it back in and use the same classification techniques?

Reproducing the image in this way is essentially a manual low-pass filter (although with little control over the parameters), so it's certainly one valid data point with which to test the hypothesis.

This can be thwarted fairly easily https://arxiv.org/pdf/1607.02533.pdf

Reply on news.ycombinator.com