1. Straw man tweet by some non-practitioner which is used to set up the straw-man argument.
2. The whole Digits example is ridiculous, statisticians "love" toy problems to prove theorems & make "arguments" etc. ML is empirical and not just the performance but the entire pipeline from data to application matters.
Let me illustrate: If your aim is to predict 1 vs 0 from images of digits. As an ML researcher I would write a program to synthesize images in all different combinations of fonts/font-color/background color/ location available. The data would easily be more than ~100,000 images. At this point one cannot use LASSO on top 10 pixels (due to jittering), and a Deep Models would be necessary. But in reality my model will outperforms because the thinking process as an ML researcher was not to make an "argument" but to "solve" the problem of detecting 1 vs 0.
3. But the biggest flaw is the following argument """The sample size matters. If you are Google, Amazon, or Facebook and have near infinite data it makes sense to deep learn."""
This is an another issue with Biostatisticians (The author of this post is Bio-Stats professor), is that they are fundamentally unable to recognize importance of programming and ability to collect data. Even if you are not Google, Amazon, Facebook you can easily collect data, even labeled data in scale of terabytes can be collected in within days or a week. Every single PhD student I know is not limited by size of the data but rather computational power and storage available to them. I personally have several terabytes of video and data from YFCC 100M that I would love to process and build models on but I am only limited by the computational power & AWS costs. If you want a concrete example, see the Google PlaNet paper [1] I today have enough data (~5 Tb) to replicate it and build open source geolocation model, the only hurdles are storage and computation costs.