Machine learning is not just glorified statistics | Hacker News Reader

Machine learning is not just glorified statistics

towardsdatascience.com

45 pointsxTWOz62 comments

I've been meaning to ask if "townrdsdatascience" is a serious website, but I guess this answers it.

It seems in order for the argument here to make sense, you have to say that predictive modelling is outside the field of statistics, and that "a class of computational algorithms" (presumably including classic ML algorithms such as decision trees, random forest and support vector machines) are also not statistical algorithms.

The author gets close to having a point when saying that in reinforcement learning you may not even have a dataset, so what is there to make statistics on? Well, you make statistics on the generated data.

I think he is right once he starts talking about approach and not knowing "variance of a population, or to define marginal probability" is not necessary to perform ML. I mean, you can perform ML without knowing what variance is, just as well as you can perform psychology experiments without knowing the variance, but I get his point:

Classical statistics is very much focused on explaining a dataset, whereas ML is very much focused on making future predictions. And you can combine and build a lot of predictive models, without the knowledge of the other half of statistics, and vice versa. But this argument is like saying topology isn't math because it is about shapes and not numbers. Or that NLP is not neither machine learning or statistics, because it is about language.

>future predictions.

But what is a future prediction really? You have a model and a dataset (possibly generated by previous input to the model), and then you ask it "IF x then what, IF y then what."

It's the exact same thing as turning data into an approximate line, and then inputting data into the function to predict the output value, within a certain confidence.

> Or that NLP is not neither machine learning or statistics, because it is about language.

Except NLP is actually not machine learning. Some NLP tasks use machine learning but a lot of NLP is not ML-based (for instance rule-based TA systems).

> Classical statistics is very much focused on explaining a dataset, whereas ML is very much focused on making future predictions.

True enough. Could you slightly change this and say statistics is about understanding the past and ML is about predicting the future? The only way to predict the future is to understand the past, or be very lucky.

One branch of statistics is about understanding the past. Statisticians are very much interested in predicting the future and have methods that look nothing like ML.

Mostly just arguing that it's special and not 'statistics' because it's statistics being calculated at a much larger scale in automated fashions. Seems like unnecessary gatekeeping just for the purpose of inflating egos.

His bio indicates he does 'ML @ Harvard'. I would imagine there's more than just ego riding on this.

Your suggestion is that someone who did a degree specifically in machine learning at Harvard would be less likely to be defending their discipline out of a sense of ego?

Not merely a suggestion; I'm stating outright that when someone's career (both the value of an acquired scholarly degree and the earned income within the field) depends on a field being regarded as a valid specialization, that person is going to put effort into reinforcing that perceived validity.

Oh okay! I thought you were suggesting the opposite (which is nonsense). Thanks!

Indeed, he should have attached ©, ®, and ™ to that phrase.

"I havent learned classical statistics, but I did take a machine learning course, so what I did must not have been that thing I dont know much about."

>Throughout the class, my fellow students and I successfully trained models

On top of that, arguing that training models is some sort of contribution to the field?

In High School Statistics, we were taught how to do everything by hand, with arithmatic. Then after we demonstrated we understood the formula, how it worked, and why, the teacher would show us that the graphing calculator had a function to do it automatically all along. Rinse and Repeat over the duration of the course. From then on, we could use the prewritten function.

Training a model is skipping to executing the function, without first learning how to write it. Is training a model even a feat worth bragging about?

>Yet, if you had asked me, or most of the students in that class, how to calculate the variance of a population, or to define marginal probability, you likely would have gotten blank stares.

But if I showed you the automatic function in the calculator or excel, and you understood how to input the paramaters correctly, I would be disappointed if you COULDNT get it to work.

>Information theory, in general, requires a strong understanding of data and probability

Probability theory is the underlying foundation of statistics. If you are going to argue that ML isnt Statistics, youre still agreeing that they are both subsets of probability theory.

>How could someone with virtually no background in stats develop a deep understanding of cutting-edge ML concepts?

Because you either a) learned statistics under a different name or b) learned how to input data into someone elses black box formulas and take credit for the output. Overall, this was a pretty painful read. The writer should check out a Philosophy of Science or Logic course. And are you really claiming a deep understanding of the concepts, or are we back to you can input data into someone elses function, and think the output is novel.

> as if to distinguish themselves from the regular, less statistical kind. Furthermore, most of the hype-fueling innovation in machine learning in recent years has been in the domain of neural networks, so the point is irrelevant.

.....what is an example of this "less statistical kind" of Machine learning? Am I taking crazy pills, neural networks are still probabilistic models. Giving such significance to the difference between AI vs ML vs deep learning vs neural networks, probably demonstrates more than anything, not knowing the unifying and underlying probability theory that well.

>Batch normalization? ... (probably because they are not statistical techniques at all).

Are you really arguing that Batch normalization isnt an application of NORMALIZATION, you know, from statistics??

> "Training a model is skipping to executing the function, without first learning how to write it. Is training a model even a feat worth bragging about?"

do you really think its easy? go do kaggle competitions. if its really easy you should just be able to run a few models and quit your day job.

ML and statistics are very much the same and just as different. maybe its true about ML people as well, but statisticians have a dogma. they think there is a right model, and they came up with it. ML practitioners generally think they have a more useful model than what was previously done.

I had a PHD in statistics try to do deep learning on images. they never got far because they could not get over the idea of having more parameters than observations. I had stats people ask me what is the r^2 on the results of an xgboost model, which the best answer I could give them was that the mean was not a good representation of the data therefore any r^2 number could not be trusted, to which they hated.

it was a bit of hyperbole.

Or c) you didn't "develop a deep understanding of cutting-edge ML concepts."

Don't get me wrong, there is immense value to being an index of what tools are available to solve a job, which works best for what, what is on the horizon, how to put the pieces together in a way they compliment each other, when to wait for the next big thing vs jumping into the legacy way of solving a problem.

Just call a spade a spade, and admit youre more of a journalist/strategist/architect/historian than a scientist.

I dont understand what is to gain by trying to paint ML as a new field, vs being Applied Statistics, unless the end goal is to CREATE confusion and mysticism, which all in all seems very unscientific.

I would consider machine learning to be an application of statistics, in much the same way that mechanical engineering is an application of physics. The foundations of mechanical engineering are rooted in physical concepts, but mechanical engineers have formulas for all sorts of things, like calculating the fatigue life of gears, that you can't get from pure physics because they are empirically derived curve fits, rather than natural laws. The science is more pure, but the application of it is what's powerful in the real world.

So in a sense I agree that "machine learning is not statistics," but I strongly disagree with the tone of the article, which is "we're better than statistics." Don't shit on the shoulders you're standing on.

It may not be an "application of statistics" in the direct sense, but that depends on how one defines statistics, which brings one back to the original problem.

Statistics and ML often have similar goals, but ML emphasizes computational efficiency over trace-able accuracy. Thus, I view each field as having different weights on the same sub-goals.

>...you can't get from pure physics...

You could use natural laws, if you want to propagate a mess of uncertain parameters through a complicated forward problem. Interpolation within empirical data is wildly easier, but a masochist could do gear fatigue ab initio in silico.

Here's one way I think of it: in statistics vs. machine learning, there's a difference in your goals, which is reflected in a difference in your models.

- In statistics, the goal is to explain something. The models have few variables, and each variable should mean something, like the influence of the person's age or sex on the outcome.

- In machine learning, the goal is to make something work. This is apparently better done with millions of variables (neural network weights), and each variable is opaque and means nothing by itself.

He hints at this distinction in the blog post but it's not entirely clear.

I completely disagree, and this is a notion that seems to have become popular but is completely off base. For some reason, people seem to believe that statistics is for inference, and machine learning is for prediction, as if statistics hasn't been concerned with matters of prediction for hundreds of years.

Machine learning is closer to being a subfield of statistics, in which methods tend to be non-parametric, have an emphasis on automation, and tend to require (but not necessarily) large amounts of data.

I think the entire reason for this notion is due to CS folks wanting in on the field of predictive analytics, but not wanting to admit that they don't have even minimal understanding of statistics or probability theory. But if they call it machine learning and insist enough that it is a completely different discipline, then they can feel better about themselves and their models that they threw together.

I think you're on point except for cynicism in your last paragraph. Machine learning came from genuinely novel research into ai. Over time the connections between ml and statistics became apparent, ml tightened up it's statistical rigour, stats took on some new techniques, and data science was born.

I'm sure some have played the machine learning card cynically, as with any buzzword, but don't throw out baby with bathwater.

I absolutely am not disparaging the actual research in machine learning borne from academics studying CS and AI. Rather, I'm referring to the seemingly recent trend by people to insist on severing all ties between statistics and machine learning. And it consistently seems to be made in articles like the one above written by authors who almost seem happy to insist they know nothing, and need to know nothing, regarding statistics.

Perhaps I am being too cynical, but its hard not to see ulterior motives given the ridiculous amounts of hype regarding machine learning, the innumerable boot camps, and to-be data science influencers.

That's the difference between inference and prediction. Both statistics and machine learning can do either of those things.

There is a difference between 'can' and "is the central question". As I said in my comment [0], traditionally statistics is about making claims about the nature of the population from a sample. If one is operating with a parametric model, the running assumption is that one knows absolutely everything about the population except for a few parameters. Those are estimated from the data. One you have done that, one can indeed use those to do prediction and if you were lucky that the data did come from the distribution assumed, then you wont do too badly, otherwise you could perform very poorly.

ML on the other hand short circuits the step of making any claims about, or even trying to understand the population. ML straight away jumps to claims about prediction quality. Note there is non-parametric statistics, but even there (barring edge cases) the central object of inquiry is the population. There is also prequential statistics that also jumps directly to claims about prediction without making any claims about understanding the population. Prequential stats is fascinating, and it is what is closes to ML in spirit.

[0] https://news.ycombinator.com/item?id=18591143

Can you give some examples of both?

Statistics being used for prediction: https://fivethirtyeight.com/features/how-our-march-madness-p...

Statistics being used for inference: https://fivethirtyeight.com/features/stephen-curry-is-the-re...

As others have pointed out, in statistics the goal is understanding something. Once you have understood it you can predict its behaviour. However the reverse is not true.

This isn't true though. There are ton's of uninterpretable model methodologies in classical statistics that have little to no ability to allow for understanding but are aimed entirely at accurate predictions. Where is this narrative that statistics is only interested in understanding coming from?

> Where is this narrative that statistics is only interested in understanding coming from?

You may have heard of this guy called Fisher he might like to have a few words with you. He says he fathered modern Stats, that he connected what was a bag of recipes to math. Many seem to agree [0] despite the fact he does not seem to be the most pleasant bloke around. The British queen seemed awfully impressed with him though, YMMV.

[0] "a genius who almost single handedly created the foundations for modern statistical science" -- Hald, Anders, A History of Mathematical Statistics.

I'm sure you were attempting to make a point, but you failed. But if you are trying to claim that R.A. Fisher only cared claims that statistics is only about inference, the point is moot because:

1. R.A. Fisher isn't the Almighty Statistical God just because he did laid a lot of foundations in early statistics.

2. Fisher has a long history of seriously stupid personal-beliefs including but not limited to: refuting anything and everything relating to Bayesian statistics, attempting to discredit the studies done showing a link between smoking and lung cancer, and advocating for eugenics and the idea of superiority/inferiority between races.

So just because Fisher may have claimed something doesn't make it so.

Machine Learning is not just statistics. David Donoho spells out the history of the whole thing in his 50 Years Of Data Science:

https://courses.csail.mit.edu/18.337/2015/docs/50YearsDataSc...

The clearest statement of the difference I've found is Leo Breiman's "Statistical Modeling: The Two Cultures:"

https://projecteuclid.org/download/pdf_1/euclid.ss/100921372...

The abstract has a succinct explanation:

> Abstract. There are two cultures in the use of statistical modeling to reach conclusions from data. One assumes that the data are generated by a given stochastic data model. The other uses algorithmic models and treats the data mechanism as unknown. The statistical community has been committed to the almost exclusive use of data models. This commitment has led to irrelevant theory, questionable conclusions, and has kept statisticians from working on a large range of interesting current problems. Algorithmic modeling, both in theory and practice, has developed rapidly in fields outside statistics. It can be used both on large complex data sets and as a more accurate and informative alternative to data modeling on smaller data sets. If our goal as a field is to use data to solve problems, then we need to move away from exclusive dependence on data models and adopt a more diverse set of tools.

The "stochastic data model" camp covers most of what people think of when they think of traditional stats -- everything from OLS through to more sophisticated techniques. Things like neural networks and random forests are algorithmic techniques that make no assumptions about or conclusions about the distribution of the underlying data.

The premise of the abstract is flawed though. Who says statisticians aren't using algorithmic models? Perhaps the author doesn't but random forests, clustering, PCA, gaussian processes, and even neural networks are standard fare for many statisticians.

I mean, the author certainly used random forests, seeing as he invented them:

https://www.berkeley.edu/news/media/releases/2005/07/07_brei...

I can't help but notice he died in 2005. Perhaps the premise of the two cultures was more relevant a few decades ago, but in my more recent experience, its absolutely not the case.

There is certainly cross-pollination between the two, and practitioners these days often adopt both. But the snide "machine learning is just statistics practiced by people who don't know what they're doing" comments you tend to see on HN ignore that machine learning was something that sprung up (largely in CS departments) to address challenges that statistics departments weren't addressing.

"..I get it — it’s not fashionable to be part of the overly enthusiastic, hype-drunk crowd of deep learning evangelists."

I think it is very fashionable. The people who are not part of this crowd are perhaps older and boring like me.

Yet, if you had asked me, or most of the students in that class, how to calculate the variance of a population, or to define marginal probability, you likely would have gotten blank stares. That seems a bit inconsistent with the claim that AI is just a rebranding of age-old statistical techniques.

So the author is asserting that... ML is statistics for people who don’t know statistics? I wouldn’t necessarily argue with that, but I wouldn’t brag about it either...

I don't love the term 'mansplaining', but if there is a term that describes essentially the same idea but in the context of academic fields, its exactly how I would describe the central thesis of this blog post and a trend I've encountered frequently in the last couple years. There is a rising tide of CS people who have just latched onto the hype of data science, and now go around letting statisticians know that no, they aren't actually interested prediction, they don't actually know how to work with large data, and don't actually work with non-parametric methods. It certainly comes as a shock to all of the statisticians in the world who have indeed been working on these types problems for a long time now.

> It certainly comes as a shock to all of the statisticians in the world who have indeed been working on these types problems for a long time now

Yup on high dimensional data of dimension as fantastic as 12. I feel bad for them though but they have only themselves to blame -- got too comfortable within their small world and lost touch of what the next set of interesting problems are.

Its only after getting kicked in the nuts that I see a course correction and that's enriching both ML as well as Stats.

If one computes stats on 600 data points with 10 dimensions and feels king of the hill, they can continue, but there is likelihood that some one else will be eating your lunch and you will be left behind. Sadly enough, this has already happened and is quite evident if one steps out of the stats bubble. Statistics could have been what machine learning and datamining is now, been the main driving force, the owner of initiative. On the contrary other communities are using statistics and probability motivated approaches but engineering them well to grab (funding) attention, well deserved in my opinion. It is them who got the ball rolling again.

https://news.ycombinator.com/item?id=17687303

Your response is so entirely off base I don't know where to begin.

> Yup on high dimensional data of dimension as fantastic as 12

Who says that has been the limit of classical statistics.

> If one computes stats on 600 data points with 10 dimensions and feels king of the hill, they can continue, but there is likelihood that some one else will be eating your lunch and you will be left behind.

Again, why do you have this impression? You clearly have no experience in the field if this is what you think statistics is. Unless your intent is to simply construct strawman arguments.

> Statistics could have been what machine learning and datamining is now, been the main driving force, the owner of initiative.

This is entirely based on the assumption that machine learning and datamining and statistics are distinct and separate, which isn't the case and is my entire point.

> On the contrary other communities are using statistics and probability motivated approaches but engineering them well to grab (funding) attention, well deserved in my opinion. It is them who got the ball rolling again.

Seriously, wtf are you talking about?

You basically went line by line to take too many words to say “you’re wrong”. That's how much content is left over after you filter out the insults.

> Who says that has been the limit of classical statistics.

The professors writing the grants. No not the limit of statistical methods per se but the limit of what they want to consider. Yeah I used to get involved in the review on rare occasions.

> You clearly have no experience in the field if this is what you think statistics is.

Does 18 years count ? I may be wrong about this but you sound like a newish grad student in statistics. If that is true I go back a bit more than you do and know about the state of affairs at the stats departments, their funding/projects/budget woes. I feel glad that the statistics departments got shaken a bit by ML and datamining for statistics to try and become relevant again.

If you follow the link in my parent comment you will see a vigorous argument (presumably by a student of statistics) that there is no reason why statistical packages even need to support 64 bits. This kind of thinking was pervasive, thankfully things are a bit better now and that would not have happened on its own.

“The world’s frontier of technological progress and innovation.” What does that even mean? All other progress and innovation is somehow trailing behind ML? What?

What a silly/confused article. To say it’s not stats because someone can implement a GAN without deep stats knowledge is ignoring everything that’s lead to that point. It’s like arguing there’s no assembly in programming because I can write and understand hello world in Python. Your fancy GAN is not coming from a vacuum. It’s been built on top of a ton of math+stats, and then you end up implementing it in tensorflow which itself is a bunch of abstractions done for you.

By definition machine learning is Statistics + Decision Theory. Statistics simply tells you what information you have about the world, not what to do with that information.

Yes it is. Having a bigger model state makes it more complex. It doesn’t make it not statistics.

Quote:

“One of our assigned projects was to implement and train a Wasserstein GAN in TensorFlow. At this point, I had taken only an introductory statistics class that was a required general elective, and then promptly forgotten most of it. Needless to say, my statistical skills were not very strong. Yet, I was able to read and understand a paper on a state-of-the-art generative machine learning model...Yet, if you had asked me, or most of the students in that class, how to calculate the variance of a population, or to define marginal probability, you likely would have gotten blank stares. ”

I agree that the GAN might be implementable without much stats knowledge, but I would be very surprised if someone who did not know population variances and marginal probabilities would be able to follow the Wasserstein GAN paper. Just see for yourself: https://arxiv.org/abs/1701.07875

This is a terrible post, coming from someone who doesn't know statistics to claim something is not statistics.

his is a terrible post, coming from someone who doesn't know statistics to claim something is not statistics.

But the author went to Harvard, doncha know. Harvard.

Yes, it is.

No, it's worse. It's statistics on steroids used by people who don't have a clue about statistics.

>At this point, I had taken only an introductory statistics class that was a required general elective, and then promptly forgotten most of it.

"No, Computing is not just glorified math". At the end of the day, yes it is, but I can understand the sentiment of those who don't need/care to look at it any deeper.

But, so is the brain.

So, I know a whole bunch of ML types who make this argument. Is this something that neurologists think?

I mean, obviously, being neither an expert on statistics, ML, or the brain, I'm unqualified to comment, but I really doubt that ML experts have a good handle on... how my biological brain works.

We think..

This sentence sums it up perfectly:

> In many cases, these algorithms are completely useless in aiding with the understanding of data and assist only in certain types of uninterpretable predictive modeling.

Statistics is a crucial component of the scientific method. It is the tool with which scientists check whether their theories agree with empirical evidence.

Machine learning is about building mathematical models that are apparently "right", in the sense that they have predictive power, but that don't necessarily improve our understanding of the data-generating processes involved.

I was listening to Lex Fridman's AGI Podcast where Vladimir Vapnik came to talk about Statistical Learning[0] and his take on deep learning as compared to statistics/mathematics was dismissive. To paraphrase from memory, the problems deep learning is solving is not hard enough.

There has been a rift between the statistical community and ML community for a while and I see it similar to the arguments one makes when it comes to sciences vs engineering.

[0] https://lexfridman.com/ai/

I have to frequently work with Data Scientists to learn their ML models and scale them in our production environments. A lot of times the models are not even statistics, it's just linear algebra. They have a tendency to go for approximate algorithms when an exact algorithm could be written. I largely blame the company for this and not the data scientists. Every company wants to be in the ML game but may not have the volume and variety of data to warrant a use case. When your favorite hammer is ML then every data problem is a nail.

The post makes two main claims

"Machine Learning Does Not Require An Advanced Knowledge of Statistics"

Lets take it at its face value, even then it does not preclude ML from being glorified stats. Depending on how advanced the 'advanced' is in that statement I would agree with it. To use ML tools well you do need some familiarity with understanding how uncertainty effects the results and that is as up one can be statistics alley. I say this although I am firmly in the ML side of the tribe. BTW I would even claim this

"[Practicing] Machine Learning Does Not Require An Advanced Knowledge of Machine Learning"

The other claim that the post makes is

"Machine Learning = Representation + Evaluation + Optimization"

Whoah! big blind spot there. Dealing with and reasoning about uncertainty, generalization is a big deal in ML. Sure, it helps to argue that ML and stats is different if one ignores that bit.

I do think ML and Stats is different but not for those reasons.

Yeah ML brings to bear some tools that card holding statisticians have traditionally not used in anger before, for example, advanced and large scale optimization, algorithms, data structures.

Using new tools to address the same question does not qualify as a deep difference in my books. For example, even the fields of optimization and algorithms themselves use tools that are different from what the tools were 50 years ago. I think the main difference is in the questions that ML and Stats wants to answer, and here there indeed are differences.

Stats (barring edge cases) is primarily interested in going from a sample to making claims about the population, or making claims about something via a claim about the population. ML is primarily interested in going from a sample to another sample. (Do note sample is a collective noun.) Now, there have been statisticians (fewer in number) and a body of statistics literature that has focused on prediction as opposed to parameter recovery, but that's not main stream in Stats.

And finally in pseudo-quotes "I managed to train a model without knowing what variance is. I know variance is statistics. ML is not Statistics ... QED ". Ah I see, nevertheless, not what I would call a brilliant case of logical deduction".

If you start your article with the assumption that ML is being a subject to memes because "it's not fashionable to like it" -- which also includes "liking" it, as if it's some kind of aesthetics art... Then your argument falls apart from the get go.

Perhaps the author should use ML itself to find out why people started mocking it!

Reply on news.ycombinator.com