Comment by theothermkn | Hacker News Reader

Comment by theothermkn | Hacker News Reader

This may sound a little like trivializing, but don't we have to know what "statistics" are and what "machine learning" is to say anything about the difference(s) between them?

Looking at this through even the lens of multinomial logistic regression, or of econometrics generally, I don't think that "statistics draws population inferences from a sample, while machine learning finds generalizable predictive patterns" even makes sense as a difference. Any prediction is an inference about the population of future events, or of contemporary events not present in the sample. You plug 1000s of events each described by 100 columns into a logistic regression, and you're hoping to get something predictive out of it. Further, as nice as the idea is that you can tease out "factors" from your 100 columns, you don't have to look at "3.3375905e-5 x (spent five years before age 18 in a smoker's home)" for very long to wonder how much 'explanation' you're getting out of the terms in the exponents of your probability functions.

I still can't resist tweaking ML enthusiasts and data scientists: Statistics is what people who know what they're doing are doing. Machine learning is the rest!

Everybody owes it to themselves to read Breiman's Two Cultures[1]:

> There are two cultures in the use of statistical modeling to reach conclusions from data. One assumes that the data are generated by a given stochastic data model. The other uses algorithmic models and treats the data mechanism as unknown. The statistical community has been committed to the almost exclusive use of data models. This commitment has led to irrelevant theory, questionable conclusions, and has kept statisticians from working on a large range of interesting current problems. Algorithmic modeling, both in theory and practice, has developed rapidly in fields outside statistics. It can be used both on large complex data sets and as a more accurate and informative alternative to data modeling on smaller data sets. If our goal as a field is to use data to solve problems, then we need to move away from exclusive dependence on data models and adopt a more diverse set of tools.

The difference between inference and prediction can be illustrated with something like a decision tree or a random forest. Statistical inference is "the theory, methods, and practice of forming judgments about the parameters of a population and the reliability of statistical relationships, typically on the basis of random sampling." If you look at something like linear regressions, it makes a lot of assumptions about the data[2]: the distribution of residual errors is normal, there's no multicollinearity, etc.

Random forest don't care. Random forest is a set of steps. You follow the steps, you get an answer. You made no assumptions about your data, about the distribution of it, any of it. You just followed an algorithm.

Algorithms are powerful! Not everything needs to be an inference problem. We're better off when we have lots of tools. You don't have to choose _between_ the two camps. But the "Everything is Statistics!" and the "Everything is Machine Learning!" points of view rob us of ways of thinking about our tools that helps us understand what those tools are.

1) https://projecteuclid.org/download/pdf_1/euclid.ss/100921372...

2) https://thestatsgeek.com/2013/08/07/assumptions-for-linear-r...

Nice comment, but there's more nuance than you claim.

> You made no assumptions about your data, about the distribution of it, any of it. You just followed an algorithm.

You made implicit assumptions that you are now unaware of, which might come and bite you later (eg: using zip codes or names as a proxy for race in credit scoring models, neural networks overfititng to texture and classifying a leopard print couch as a leopard). This means that you are liable to overfit to the data, and generalize poorly, or in ways you are not supposed to.

This is what leads to the perspective that "machine learning" might end up being a powerful tool for laundering bias: https://idlewords.com/talks/sase_panel.htm

I think what you're saying is orthogonal to what I'm saying.

Yes, ZIP codes can be a proxy for race in models dealing with credit scores (and recidivism, which is also a really bad place to put racial bias), as an example. But if I put it in a mixed-effects model, it shows the same bias, and a mixed-effects model is just an extended version of linear regression. Both statistical and ML models suffer from the problem you're stating.

What you have not made any assumptions about in a random forest is about the distribution of the data you're looking at. One example of a case where the assumptions that bog-standard OLS makes about your data can cause you problems is zero-dominated data -- data with a lot of zeros in it. Basically any time you're trying to make predictions about things that are rare in your measured population.

OLS does a bad job on zero-dominated data. If you throw a zero-dominated dataset into a random forest, you will get back better answers than if you use OLS on zero-dominated data.

To be clear: there are strategies for dealing with zero-dominated data using statistical inference. You don't have to resort to non-inferential learning just because you have data that doesn't look like a bell curve. But machine learning is a powerful way to get pretty good results on a lot of problem spaces without having to understand the probability function involved (or in cases where the probability function is too complicated to be tractable computationally, like the probability function that determines the color of pixels in a dataset where you're classifying dogs versus cats).

> machine learning is a powerful way to get pretty good results on a lot of problem spaces without having to understand the probability function involved

The NFL theorems essentially state that for every dataset on which an algorithm generalizes well, there is another on which it generalizes poorly. So, there are always implicit biases, even for a random forest model (eg: if you try to model Boolean functions with a random forest, those functions which can be approximated effectively with few trees will form a set of very small measure. Specifically, it is my intuition that those which have many terms in the sum of products form might need many trees to approximate). It then becomes a question of whether the bias of your models are compatible with the dataset/domain under consideration.

See this Minsky-Sussman koan: http://www.catb.org/jargon/html/koans.html#id3141241

> Both statistical and ML models suffer from the problem you're stating.

And that is precisely why I don’t see how ML algorithms are “more powerful” than statistics in any way, as per your claim.

> And that is precisely why I don’t see how ML algorithms are “more powerful” than statistics in any way, as per your claim.

Well I'm glad you don't see that, but I'm a bit confused why you think I said it because I didn't say it and don't believe it. I said that ML is "powerful," not "more powerful."

That seems to paint random forests as too magical. Rather, to me, random forests are just assuming that there is a BDD to get your answer. You aren't making statistical assumptions of the data, per se. However, you are assuming you can keep doing some sort of split on the data. Such that lower levels of the tree should be somewhat interpretable. Same for the highest levels. It is just the walk from high to low that is hard to fully explain. Right?

I don't think random forests are "magical" at all. I think you can walk a non-technical audience from decision trees to bagged and boosted forests on a whiteboard in under an hour. (I don't think you can do this nearly as well for maximum likelihood estimation, honestly.) But there's still a difference between a mathematical function that you're trying to find the optimal values for and something that's defined by an algorithm that there's no single function to optimize.

But, at the end of the day, you can get some easy to understand statistics from building a decision tree. Based on population makeup, you'll know solid numbers on how many will be expected to hit each outcome.

Such that optimizing some other function based on the application of a decision tree is effectively a statistical question. A natural one.

Indeed, I'm not sure I see the difference. One is just explainable based on understandings of distributions and trusting that they hold. One is explainable based on understanding of decisions, and trusting those remain the best decisions.

I used my bike ride as an example earlier. It is made up of many decisions to get home. Knowing what all of those decisions are can help build up a solid range of when I'm likely to get home. For the level of accuracy I typically care about, so does just knowing the rough distribution of how long I typically take.

How are both of those not statistical in nature? One is just more fully exploring a space with ridiculous computational power, whereas the other is generalizing to a much quicker answer. Right?

Breiman is right and wrong at the same time.

Powerful tools like nonlinear polynomial models, OLS, HMM and RBMs were implemented and devised by statisticians. And not from tribe 2. The difference is the data model is explicit but general.

Have fun trying to model extremely nonlinear physical effects with statistics

Oh yes, we just include all the possible nonlinear effects and pick the ones which look meaningful [0][1]

Then we go back and perturb the model to do some hand wavey guessing at real actionable insights

but hey clients are impressed by overfitting so who the fuck cares as long as the money is coming in and the press releases are going out!!!!

[0] https://www.featuretools.com/ [1] https://scikit-learn.org/stable/modules/generated/sklearn.pr...

Reply on news.ycombinator.com