Hacker News Reader

Comment by MAXPOOL | Hacker News Reader

Deep learning machine learning models are statistical and probabilistic models. You can categorize Deep Learning under both computer science and statistics. For example stat.ML and cs.LG in Arxiv.

Machine learning and statistics are closely related fields, both historically and in current practice and methodology.

View on HN

My working definition is that statisticians choose and engineer models while machine learning searches a vast space of models.

That doesn't seem right to me. In both cases you have a model and are just searching for optimal parameters considering the bias/variance tradeoff. There may be a few instances of a neural network or other ML model being set up to dynamically change it's architecture during training but that seems to me like it would not work out well at all.

If anything in specific cases the statistical model (if Bayesian) is more comprehensive in that it doesn't try to find a point estimate of the parameters but instead forms a full distribution around the plausibility of the parameters.

You can have Bayesian DNNs.

Not easily. I don't think the literature is very incredible on this - how do you define a prior over all of the parameters of an NN?

There’s a vast literature you can read.

I have read some of the literature - ie. the bayes by backprop method, etc.

It doesn't seem like getting a posterior over the parameter space of a neural network is tractable as of now.

It ain't tractable in general either.

So I was thinking like this - if you have a data set with 200000 variables and 70,000 examples how would you find a model without using a search process? On the other hand if you have 20 variables and or if you understand which of the 200000 variables are the ones to worry about then you can build a model by hand. I guess that also statisticians are working to summarise or create insight about data while ML is working to create a prediction (although statistical models can be used to do that too).

Why do you think that statisticians only build models by hand? Dimension reduction techniques are employed in statistical learning as well.

Another way I've heard it framed is that statistics cares about the parameters, and ML cares about the answers.

ML cares about predictive generalization, stats cares about understanding and interpreting drivers

That framing seems a wee bit dismissive of statistics, esp. applied stats -- I'll hazard a guess that this came from the ML camp. :)

There is nothing dismissive about it, they are just different things.

If we have strong theoretical understanding of the physics/model of a problem, but are unsure about some parameters, it makes sense to develop methods to find those unknown parameters accurately. This ability is a big deal in traditional statistics. People working in this field try and prove results that their proposed method can actually do this. If the model happens to be largely correct and the method robust, this even allows prediction.

Often, however, we do not know the model and the 'parameter' is a piece of fiction anyway. If we are interested in prediction alone, its fine to let go of an ability to accurately estimate the parameters as long as predictions are accurate. Think epicycle models of planetary motion. ML folks try to prove that their methods have good prediction properties and are happy to sacrifice on parameter recovery.

Nonparametric statistics and prequential statistics are somewhere in the middle.

Sometimes it does seem though that people are trying to out do each other, coming up with methods that estimates the spectrum of a unicorn's rainbow more accurately than the best known result in research literature. This may look odd because the unicorn and his rainbow are pieces of fiction.

Fair points made, and your unicorn analogy is wonderful. :)

As someone with degrees in statistics who now works in ML, all of these simplistic definitions are pithy, but useless. The reality is that the overlap between the two fields is so enormous that trying to come up with a clean separation for the two is a losing battle. To say that statisticians are uninterested in “the answers” (I.e. predictive accuracy) is foolish and rarely true. To say that ML doesn’t care about parameters (I.e. inference) ignores the vast amount of effort in ML researching topics like causal inference, feature importance, and yes, inference.

At this point I’m convinced the debate regarding what is statistics vs what is ML is largely just tribal warfare. The two parties are fighting over shared territory, and while both may have had some claim to uniquely identifying techniques and research, the lines have long since eroded.

Reply on news.ycombinator.com