Hacker News Reader

Comment by janhenr | Hacker News Reader

I do really like the idea of an MNIST alternative to very quickly verify ideas. However I have a few nitpicks:

1. 10 classes is way too small to make meaningful estimates as to how well a model will do on a "proper" image datatset such as COCO or ImageNet. Even a vastly more complicated dataset like CIFAR-10 does not hold up.

2. I feel like CIFAR-100 is widely used as the dataset that you envision MNIST-1D to be. Personally I found that some training methods will work very well on CIFAR-100 but not that well on ImageNet so TinyImageNet is now my go-to "verify new ideas dataset"

View on HN

Genuine question: are there real-world image recognition tasks that requires training on more than, say, 10 or even 100 classes? I'm personally aware of only one that might come close, and that's because it's an image-based species detection module, and it's whole purpose is actively trying to be able to recognize a large number of very specific subgroups. But most of the other I can think of get maybe a couple dozen, and sometimes even as few as 4 or 5 classes where they're useful, and the accuracy within those cases is much more important than the sheer number of possibilities themselves.

I guess I'm just asking if COCO or ImageNet-trained networks are actually noticeably superior for most real-world tasks, or if it's just a metric that's used because the performance differences only show up in the long tail of the distribution.

> I guess I'm just asking if COCO or ImageNet-trained networks are actually noticeably superior for most real-world tasks, or if it's just a metric that's used because the performance differences only show up in the long tail of the distribution.

Given that for any real-world vision task you start from a pretrained model om those datasets they will in fact be noticably superior on the real world task after finetuning. Just because the quality of the features extracted through the backbone is better.

Reply on news.ycombinator.com