Scaling Down Deep Learning
greydanus.github.io
greydanus.github.io
Fashion MNISt is not better - it shares the same issue with MNIST. At the very least, use non-MNIST (letters A-H from various fonts). Instead, I wholeheartedly recommend Google Quickdraw - hand-drawn doodles; more samples, more engaging, and more diverse. And images of the same size as MNIST.
See an example of usage for someone's first neural network: https://github.com/stared/thinking-in-tensors-writing-in-pyt...
The foundational discoveries in genetics centered on far simpler organisms such as peas, molds, fruit flies, and mice. To this day, biologists use these simpler organisms as genetic “minimal working examples” in order to save time, energy, and money. A well-designed experiment with Drosophilia, such as Feany and Bender (2000), can teach us an astonishing amount about humans.
The deep learning analogue of Drosophilia is the MNIST dataset. A large number of deep learning innovations including dropout, Adam, convolutional networks, generative adversarial networks, and variational autoencoders began life as MNIST experiments. Once these innovations proved themselves on small-scale experiments, scientists found ways to scale them to larger and more impactful applications.
> Many good ideas will not work well on MNIST (e.g. batch norm). Inversely many bad ideas may work on MNIST and no[t] transfer to real [computer vision].
It might be considered Drosophilia of machine learning, though. Take a look at this beautiful table of the results of various classifiers http://yann.lecun.com/exdb/mnist/.
Unless your dataset is more similar to mnist than to “real computer vision” (eg galaxies or piano rolls) and you still want to use deep learning to classify it.
Galaxies are not like such (more diverse, more noise, you don't start with centered images, usually more data - in size and channels).
I suspect a chess position evaluation would suffer from batchnorm just as much, if the intermediate feature maps remain sparse.
What an amazing find, this blog, especially wing optimization (as you pointed out). I hope this guy gets the resources to run free with his work and just create incredible things.
Other datasets I tried:
Fashion MNIST has so many mislabels and also is pretty trivial to separate with UMAP alone
Google's QuickDraw is better than MNIST, but I also haven't tested it against e.g. log. regression.
Of course there is CIFAR but those images hardly look like images. I can't classify half of them.
It's been a minute since I last touched ML, but that seems like a fairly extreme claim. Am I wrong in thinking this?