Machine Learning Is the New Statistics
danielmiessler.com
danielmiessler.com
The standard text in ML, "The Elements of Statistical Learning" is authored by statistics Professors.
Statistics is the new statistics. The rest is marketing bullshit.
But let me try to push back.
If we're trying to draw human-consumable wisdom about the state of the world from data, simply capturing a snapshot and then applying some basic analysis is one thing.
Creating self-improving mechanisms for doing this is another, even if the latter use statistics in the process. Perhaps in a similar way that Statistics uses Algebra yet is distinct in its description and capabilities.
I'm not convinced you're wrong here, just trying to talk through it.
Saying that applied physics will be the new physics because it changes the real world, is nonsensical.
In earlier days Google called their translation algorithms statistical, then they changed it to ML and now AI has come in favour again, so that is the word being used.
To be fair, I think "Machine Learning" was an academic marketing term coined by Computer Science departments. It seems that the term "Statistical Learning" was coined in response by Statistics departments. Other similar marketing buzzwords used by various factions (comp sci, stats, actuarial science, industrial engineers, etc) include "Data Science, "Predictive Analytics," "Data Mining," "Knowledge Discovery," "Knowledge Engineering," "Soft Computing," "Artificial Intelligence," "Big Data," "Deep Learning"... To be honest, it's tiresome and troubling to see academic departments invent and adopt overlapping and vacuous marketing buzzwords.
Well, each random variable has a distribution. And there are a few distributions that are common so are taught. Then, presto, bingo, too many students conclude that an important first step is to find a distribution. However, commonly in practice, with just samples and without more in mathematical assumptions, finding a distribution is from not very promising to hopeless. Hopeless? Yes, consider a random variable that takes values in 50 dimensional Euclidean space.
But there is a lot of statistics that is distribution-free, where we make no assumptions on probability distributions. E.g., I published such a paper in Information Sciences. In addition with some meager assumptions, say, the random variables have expectations, the squares of the random variables have finite expectations, etc., can do more.
For model fitting, if can assume that the data has Gaussian distribution, is homoscedastic, have independent and identically distributed (i.i.d.), etc., then can get some more results, e.g., know that some of the results of the computations have Gaussian or F distribution, etc. Then can do a lot of classic hypothesis tests, confidence intervals, etc.
But, with just meager assumptions, commonly can still proceed and know that are still making a best L^2 approximation. Then can drag out the classic result that a sequence of (such) random variables that are Cauchy convergent in L^2 do converge in L^2, that L^2 is complete (i.e., a Hilbert space), and that some sub-sequence converges almost surely. That's a lot -- might be able to take that to the bank. And made no more than meager, general assumptions about distributions.
Really, often we get some of the well known distributions from some theorems, not analysis of empirical data. E.g., get a Gaussian assumption from the central limit theorem. Can get an exponential distribution from the Poisson process (e.g., E. Cinlar's text), and can get that from the very general, even astounding, renewal theorem (e.g., W. Feller's second volume).
In fact the birth of statistical learning theory was Vapnik's (rather unintuitive) insight that although the theoretically optimal strategy in a classification task is the class conditional distribution, estimating the class conditional distribution from the data is not a promising way to do it. Estimating distributions is freaking hard, you may not (ever) have data to estimate that distribution, however you can short circuit the process and solve the classification problem directly (without explicitly modeling the class conditional distribution).
That said, machine learning has models that do not rely on a stochastic sampling process. The sampling process can be adversarial too. In such situations the algorithms guarantee that you wouldn't be too far off (even in finite time) from the strategy that would be optimal in hind-sight. These are non-asymptotic guarantees. I am of course sweeping large swathes of theory under the carpet, because this is hardly the forum for that.
The beauty of CLT notwithstanding, what is surprising is how rarely the Gaussian assumption holds in practice. Many distributions are too heavy tailed to have a finite variance, and in such scenarios CLT does not yield you a Gaussian but yields a stable distribution (Gaussian is just one, in fact the only one in the stable family with a finite distribution. Gaussian distribution was not discovered by Gauss and neither is it that normal). To make a tongue in cheek claim, the prince of mathematicians sort of got away with it.
Machine learning theory was never far from statistics and probability. How can it, when its traditional theoretical bedrock (PAC) is explicitly a statement about convergence in probability with all its epsilon and delta in your face. Where the two differ are in the amount of focus on prediction (as opposed to parameter estimation), familiarity with optimization and algorithms, and lack of fascination for test of hypothesis, Gaussian distribution and asymptotic normality.
I don't see how assuming the data are IID from a Gaussian is a meager assumption.
It's not. Somehow we have failed to communicate accurately.
I wrote
> But, with just meager assumptions, commonly can still proceed and know that are still making a best L^2 approximation.
In that sentence, I didn't suggest that those "assumptions" were the Gaussian i.i.d. of the previous paragraph. Instead, I left the "assumptions" unspecified. Why? Because there is quite a variety available. But lots of the options are "meager".
Typically with more assumptions, can get more results. But for model fitting, building, constructing, discovering, whatever, can still get a lot with next to nothing in assumptions.
Walk into a statistics department in a university, and you can feel the differences in culture. It seems to boil down to things about what one considers sacred. Confidence Intervals, Hypothesis Testing and Rigor are things statisticions hold near and dear to their hearts. Want to be a statistics phd? You'd better understand all the differences between convergence in probability and convergence almost surely. And things like Probability Approximate Correct Learning, VC Dimension, Random Forests, and even Deep Learning should sound like heresy to a true, card carrying, statistician.
Another odd cultural difference. Statisticians love citing old papers. You can call a paper about 20 years old "the classical work of" in ML. Stats has a much higher standard for this. The classics go way, way back to a time before I was born. Its gotta be an issue of pride.
But yea, I do feel for the most part, I feel the fields are converging, even if it seems like ML is consuming statistics, not the other way round. Stats majors are publishing in places like NIPS rather than the traditional journals, not the other way round. Just a feeling though. I have no stats to back up this claim.
But there's no point saying "A is a subset of B" unless you want to spice up your happy hour conversations in the pub.
Statistics are tallied numbers and represent actual measured values. The field of statistics is concerned with tallied numbers collected, probability is concerned with the likelihood of those numbers being produced under specific assumptions, and machine learning is a process that uses statistics to verify and adjust the probability model being used for study.
Those are all definitions used by mathematicians and statisticians (who are a subset of mathematicians), not 'sociological artifacts'. Things don't sound like heresy to a statistician unless he is arguing implicit versus explicit logic. That is regardless of how it feels when he walk into his department.
There is no need to prescribe artifacts' if we can just keep the correct definitions clear and not conflate them.
To make the case that ML - stats isn't empty, consider neural networks and singular value decomposition (Netflix winners used SVD): both are wildly successful ML techniques – and neither is remotely statistically sound. Their correctness is at best heuristic, yet, partly because they can be efficiently applied to huge amounts of data – far more than classical statistical methods can handle – they are very effective.
Statistics has traditionally focused on making the most of a modest amount of data – because in previous eras, data was the limiting factor. Huge data sets today render most statistical tests useless since no hypothesis is strictly true given enough precision, and the focus on eking the most out of every data point is overkill. It turns out that a naive, imprecise method with a huge amount of data is often more effective than a sophisticated method with far less data. As ML gets better at building statistically sound models and stats gets better at scaling, the two disciplines are converging slowly – but we are still very far away from inclusion in either direction.
Mind, huge data sets and huge amounts of computing power also make the use of Bayesian methods feasible, which don't have the NHST problem that anything is significant with enough data-points.
The central concept of Machine Learning is self-improvement of the models that are used, based on data alone.
That is NOT the central concept of Statistics.
So yes, Machine Learning may be related to, part of, semantically linked, a subset of, or whatever you want to say there, but the fact that Machine Learning is designed to self-improve is (in my opinion anyway) the reason it should be considered a "new" way of evaluating the world.
Saying it's all just Statistics sounds a lot like calling consciousness "just another information processing mechanism" (yawn). I know it's not that big of a difference, but it's similar.
Self-improvement of data analysis models matter enough to warrant the separate name and the attention that comes with it.
We can recognize objects, recognize speech, at almost human level accuracies. That's a big milestone when you think about it.
Also technology improves exponentially when a ton of smart people funded by crazy fuck you money work on pushing it forward.
It's a claim, nothing more.
The difference between ML and statistics is entirely semantic. Is logistic regression a ML method or a statistical method? It is both!
That's the part that's going to have the impact---automated improvement of the way Statistics is applied to data analysis.
Is that not major enough to be considered and discussed separately?
So, I actually agree to an extent. Much as computers can be seen as the "next logic". Only, it is such a "builds on" relationship that I think calling it next is dubious.
[1] Variational Inference: A Review for Statisticians, https://arxiv.org/abs/1601.00670
[2] A Summary of Machine Learning Papers from IJCAI-85 http://web.engr.oregonstate.edu/~tgd/publications/mlj-ijcai8...
[3] Chunking in soar: the anatomy of a general learning mechanism, http://repository.cmu.edu/cgi/viewcontent.cgi?article=2552&c...
[4] Explanation-Based Learning: An Alternative View, http://www.cs.utexas.edu/~ml/papers/ebl-mlj-86.pdf
Deep Learning wasn't even possible until recently though, data and compute power have made it possible as opposed to just proportionally better [1]. There have also been a lot of breakthroughs in Reinforcement Learning riding on the wave of Deep Learning, and both of those (DL and RL) are more than applied statistics.
1. I think of this as a 0 to 1 innovation versus a 1 to n innovation if you're familiar with Peter Thiel's terminology on that.