Hacker News Reader

Comment by Eridrus | Hacker News Reader

This paper is definitely quite interesting, but before this thread turns into a bunch of NN hate, here is another take:

"Do CIFAR-10 Classifiers Generalize to CIFAR-10?" - https://arxiv.org/abs/1806.00451

They use the same procedure used to construct CIFAR-10 to construct a new test set and test a bunch of state of the art results on the new test set.

They see a generalization gap, but the relative order of SoTA results remains (roughly) the same.

So, yes, test set validation is an overestimate of real-world performance of these systems, but progress on test sets is indicative of progress in real-world settings.

And to remember this in context, no-one is asking "do traditional computer vision systems understand images", because they were explicitly just looking at image statistics.

View on HN

As you point out, that the relative ordering remained stable means progress being made is not simply overfitting to the test set. The negative finding from the paper you link to is in how much accuracy drops considering how slight modifications to the test set were. In their own words:

> We view this gap as the result of a small distribution shift between the original CIFAR-10 dataset and our new test set. The fact that this gap is large, affects all models, and occurs despite our efforts to replicate the CIFAR-10 creation process is concerning.

> Nevertheless, the accuracy of all models drops by 4 - 15% and the relative increase in error rates is up to 3×. This indicates that current CIFAR-10 classifiers have difficulty generalizing to natural variations in image data.

It remains to be seen if others can think up ways to maintain performance that the authors did not manage.

Here is the new dataset: https://github.com/modestyachts/CIFAR-10.1

How do we reconcile these results with, e.g. YOLO performance in videos like this? https://www.youtube.com/watch?v=_kxX09i4fds

Those models don't seem to be THAT brittle.

test set validation is an overestimate of real-world performance of these systems, but progress on test sets is indicative of progress in real-world settings

I think this is important point to make and talks to the real life limitations of run of the mill SL techniques. If you can't validate with noisy "real world" data, then you're basically staying inside the box and hoping that you've curated well enough to mimic real world conditions.

I still like RL system for the improvement on this, however it's much more difficult in practice.

Reply on news.ycombinator.com