What I learned from competing against a ConvNet on ImageNet | Hacker News Reader

What I learned from competing against a ConvNet on ImageNet

karpathy.github.io

32 pointsHoushalter5 comments

Based on the sample of images I worked on, the GoogLeNet classification error turned out to be 6.8% (the error on the full test set of 100,000 images is 6.7%). My own error in the end turned out to be 5.1%, approximately 1.7% better.

To put this in perspective, the task is basically "given a picture, identify the correct class out of 1000 categories" (it gets into specific breeds of dog, for instance). Humans turn out to miss some of these, and neural networks are now nearing human level performance.

It's fun to note that about 4 years ago I performed a similar (but much quicker and less detailed) human classification accuracy analysis on CIFAR-10. This was back when the state of the art was at 77% by Adam Coates, and my own accuracy turned out to be 94%. I think the best ConvNets now get about 92%.

Things are moving quickly in this field.

(My personal) ILSVRC 2014 TLDR: 50% more teams. 50% improved classification and detection. ConvNet ensembles all over the place. Google team wins.

To be clear - this is a human viewing the images, and only narrowly beating the neural net.

Furthermore, the human spent some time 'training' himself on the images, plus he used a neural net to reduce the possible answer space - he didn't come to it cold.

"Then I organized a labeling party of intense labeling effort only among the (expert labelers) in our lab. Then I developed a modified interface that used GoogLeNet predictions to prune the number of categories from 1000 to only about 100. It was still too hard - people kept missing categories and getting up to ranges of 13-15% error rates. In the end I realized that to get anywhere competitively close to GoogLeNet, it was most efficient if I sat down and went through the painfully long training process and the subsequent careful annotation process myself."

Go to the demonstration labeling interface here: http://cs.stanford.edu/people/karpathy/ilsvrc/

Hit the "Use hard course".

On the left hit "Show google prediction" consider the answers for a few seconds, then hit "Show answer".

My reaction to the Google predictions is "yeah, that's reasonable". Often my reaction to the actual true label is "wtf, that's not obvious at all."

I get the feeling the Google image net produces better tags than the validation set labels :)

There are some good TLDRs here. But the actual article includes a link to a UI with the actual ImageNet decision task created by the author. You can compete against the winning ConvNet too..

This is very impressive. I hadn't realized that image classification was so good.

I'm not sure I understand the results though. On the demonstration site, there's a group of images that apparently failed to be classified. But, when I tried it this morning, google's prediction shows a correct classification, for some of them.

Anyway, where is this service? It would be extremely valuable to developers, even in it's current state.

Reply on news.ycombinator.com