Comment by jph00 | Hacker News Reader

Comment by jph00 | Hacker News Reader

I don't understand why this seems to be getting so much attention. There are plenty of small image datasets around, and wide recognition of the issues with MNIST.

I see no evidence at all that this particular dataset is better than MNIST. None of the issues they themselves list with MNIST are discussed with relation to their proposed replacement.

The benchmarks they provide are entirely useless - sklearn does not claim to be a platform for computer vision models. A quick WRN model gets 96% of this dataset (h/t @ajmooch on Twitter), suggesting that it doesn't deal with the "too easy" issue.

The images clearly don't deal with the problem of lack of translation invariance.

On the downside, they don't have the same ease of understanding of hand-drawn digits, which is extremely helpful for teaching, debugging, and visualizing.

I completely agree. My company does image recognition in fashion and this dataset is both way too easy and not useful. I'd love to see a client with such an easy problem (telling between shoes and bags in inventory images).

We regularly get high 90's accuracy on real world images where the system has to auto-crop all on its own. Inventory images are far too easy.

What's more, the actual hard cases (like tiny shorts vs tiny skirt, or long blouse vs short dress) would be near impossible to do at such low resolution.

Wide residual networks are not exactly hello-world networks. For teaching (I teach a course on DL), I think this is great. It adds variety for me in terms of an introductory tutorial for DL, and it encourage students to use a GPU as error rates will be high on their laptops.

> sklearn does not claim to be a platform for computer vision models.

There are more than a dozen image classification and segmentation examples on the scikit-learn gallery:

http://scikit-learn.org/stable/auto_examples/index.html

Yes but at this stage they're really toy examples. IIRC the best sklearn result they showed on this benchmark had 20% error, whereas the little WRN network was 4%.

IIRC a simple convnet will get higher than 99.0% on MNIST, so maybe it is more difficult than it seems at first glance :)

Reply on news.ycombinator.com