However, the article is misleading about the initial gif being any sort of argument against Bayesian probabilistic approaches, as it's precisely an argument for them. It's an argument against using point estimates and summary stats, as they obviously loose a lot of information and can be misleading (you can get the intuition from a CS perspective as well: compressing a 2D image into 4 numbers is necessarily an extremely lossy compression). Fully Bayesian approaches would maintain the distributions all the way through the calculations, rather than collapse them at some point to one or a few numbers summarizing the distribution.
Take a look at Kaldi or any state of the art platform for sound to speech and NLP analysis, or take a look at deep belief machines/networks. They all roughly extend probabilistic graphical models with deep architectures and some nonlinearity to do what couldn't be done before. (See also the Neural Hawkes Process [https://arxiv.org/abs/1612.09328].)
The article seems to primarily be about promoting his book instead of having a cohesive argument.
Yes, nonlinearity matters, but I'm not remotely convinced that tossing probability is a good thing.
I want to dig deeper into this but two things come to mind:
1. George Sugihara's and Floris Takens' works on quantifying causation in non-linear systems. Correlation doesn't imply causation, so why not look at causation? Here's a writeup on it: https://www.quantamagazine.org/chaos-theory-in-ecology-predi...
2. Andrei Khrennikov argues that quantum models can be expressed with Kolmogorovian (classical) probability theory. He points out that many probability theorists have glossed over an entire subtlety of Kolmogorov's work: that probability spaces need not be finite dimensional, that they can be subspaces of higher dimensional, even infinite dimensional probability spaces. Khrennikov's philosophical convictions are not widely shared by quantum theorists, but his math is solid (he seems to be in pursuit of quantum theories with "hidden variables", like Einstein was looking for). http://www.sciencedirect.com/science/article/pii/S0375960103...
Even in the gif shown in this article, you might suppose that each point set could be modeled in higher dimensional terms, for example adding some kind of discrete curl-like measure. But even without going there: something as simple as a changing to a polar basis should reveal different statistics.
And isn't this kind of what deep learning is all about? Going between higher and lower dimensional representations?
Therefore it shouldn't be surprising that that you can find probability distributions that share the first two moments, but differ wildly in the higher moments (skewness, kurtosis) or even the number of modes (bimodal, trimodal, etc.).
Not everything in the world is Gaussian, but we knew that.
The paper is actually really interesting, as it provides a useful insight into what summary statistics mean and the importance of data visualization.
The second part of the article makes a claim that is worth discussing -- that probabilistic inference does not work well in nonlinear domains.
I think this is true in one sense, but in many practical situations we can usually to get something to work well enough for it to be useful. Assuming we can (approximately) split up the domain into multiple regions and get probability distributions for each region (with transition functions between them), we can continue to make inferences under nonlinearity. That said, this is a concession. Perhaps there are techniques in deep learning that more general in nature.
Again, simple summaries of complex things removes nuance.
Things work until they don't.