Hacker News Reader

Comment by taylorius | Hacker News Reader

Computer vision systems are not as robust as human vision systems. There are subtle changes to pixel values that can utterly flummox a computer vision system, but seem undetectable to a human looking at the image.

View on HN

I've read about this

https://www.nytimes.com/2023/02/13/technology/ai-art-generat...

How do subtle pixel changes cause the image of a dog to classified as a cat rather than as gibberish?

This is a guess (I have read some of the paper but to me it does not seem to explain), but presumably you work backwards from the result you want. There is some vector that comes out of the classifier model that would represent cat, and another that would represent dog; you get the difference between these two models, and work back through the layers of matrix multiplication finding places where small perturbations in the input data force the classifier to make the exact errors you need.

They claim that the poisoning is transferrable across different models, which would not be the case if that was true.

OTOH, given this group's track record with their earlier poisoning effort (Glaze) vs. the adulatory press that they managed to arrange, I don't expect much from Nightshade despite the similarly adulatory press. Great media relations game, though.

what is the nature of these pixel changes?

It varies. Specially crafted noise added to the pixel values. Looks like random noise, but obviously isn't. TBH I'm not an expert, but as I understand it, It is "trained" using the vision network, with a loss function that is some combination of being low amplitude, and reducing the strength of the correct image identification.

Reply on news.ycombinator.com