Hacker News Reader

Comment by extrapolate | Hacker News Reader

Great read! Time to troll some of my machine learning co-workers.

"Can you classify this image of a panda? Oh, curious, it thinks it is a vulture..."

There are tons of anecdotes i remember from AI classes about neural nets learning something other than what you expected them to learn from your training set.

For example, one story involved training a classifier to recognize an overhead image with tanks vs without. It turns out it ended up learning which days were sunny and which were overcast.

This sort of thing happens when training people from examples too: From the mundane cases in school, to the AA587 crash in Queens NYC.

> ended up learning which days were sunny and which were overcast.

The earliest source I can find for this is https://neil.fraser.name/writing/tank/ but it says it "might be apocryphal".

Actually, there's also the 1992 paper "What Artificial Experts Can and Cannot Do" [1] which gives this anecdote and calls it a "legend".

[1] http://www.jefftk.com/dreyfus92.pdf

Nice sleuthing! I don't even recall where I first heard this story myself. The paper was an interesting read and still largely applicable even now in this 3rd or 4th coming of NN's.

Expanded this into a post: http://www.jefftk.com/p/detecting-tanks

I don't think it'll nessesarily work, if you take an adversial example for one network and plugs it into another.

surprisingly, if you read the paper, the adversarial examples actually do generalize to some extent if you try them on another network!

Here is the table from the original paper. Note this is for MNIST, not more complicated datasets like Imagenet.

http://i.imgur.com/fJ35PTc.png

It's kind of confusing, but table 2 shows what percent of these adversarial images trained on one networked worked on another. It varies quite a bit, and many networks aren't similar enough to each other for it to work reliably. But there is definitely some degree of generalization.

Reply on news.ycombinator.com