Comment by Konohamaru | Hacker News Reader

Comment by Konohamaru | Hacker News Reader

> Traditionally, it’s a cardinal sin in academia to use parameters like these because you can’t say anything interesting about the parameters, but the trick in machine learning is that you don’t need to say anything about the parameters. In machine learning, your focus is on describing y-hat, not β-hat.

This kind of philosophy will cause future generations to see machine learning as something worse than a fad, almost as something in between a fad and crank science. If this encapsulates how all (generally speaking) machine learning operates then we will enter big trouble, if we have not already.

> In machine learning, bad results are wrong if they catastrophically fail to predict the future, and nobody cares much how your crystal ball works, they only care that it works.

This has moved from Cargo Cult Science into numeromancy. It's leveraging the occult (=hidden, incomprehensible parameters) for predicting the future. Because there exist no first principles, nothing can be further interpreted. Only more of the occult can be leveraged in order to make more predictions not amenable to interpretation, which will in turn require MORE occult to make MORE inscrutable predictions, until the heat death of the universe....

And appealing to 80's AI (neural networks) as case precedence further harms the author's case. If ML operates like how AI neural network technology went, then this whole rigamole will go tits up by case precedence as well.

I think essentially the opposite is true, and academia's notion of cardinal sins have held back the analysis of data by decades. And I say this as an academic.

You make predictions all the time, and you don't know how. You don't know how you walk, how you drive a car, how you know the rules of English grammar. Your mind is a black box. And yet, you can do things that are still beyond the power of what we can understand using first principles. In many domains, first principles have achieved nothing. Years and years of effort by some of the smartest people in the world, and we pushed the ball one inch down the football field.

Supervised machine learning asks the question "To what extent can we predict this outcome, given these predictors?" This is a perfectly valid question, one that we can try to answer given enough data. And we can do it under the exact same conditions we can do standard statistical inference.

Ultimately, we want to answer "why" questions. But sometimes we can't even answer "what" questions. Most data is so complex that it's hard to say what the data even says, let alone why it says it. We could have been using what as a stepping stone to why, but our own provincialism as statisticians prevented us. I hope now we are learning to do better.

How does one interpret what a black box means? In truth, any black box must remain beyond interpretation as a consequence of the fact that Verificationalism failed.

There exist hidden verificationalist presuppositions embedded in your conclusions. If verificationalism doesn't hold, then by-first-principles stands as the best approach.

> This kind of philosophy will cause future generations to see machine learning as something worse than a fad

Why do you think so ? Parameters are a piece of fiction, no one has actually seen them. Sometimes they are an useful piece of fiction, but not a very falsifiable notion.

Prediction accuracy, well, that I can definitely measure without resorting to pieces of fiction that are epistemologically unknowable.

I agree from the perspective that often complex models that cannot be interpreted are created because Neural Networks sound fancy, when much simpler models could be used instead.

But on the other hand complex models such as Neural Networks are being used for very wide datasets with incredible amounts of parameters, so that understanding a single parameter's contribution is not useful in the real world.

For example, using mouse movement data to classify users into age groups. Knowing that some movement vector adds 0.000067 to the probability of a user being in the 16-25 age group, and so being allowed to watch a movie rated 16+, is not very useful.

> But on the other hand complex models such as Neural Networks are being used for very wide datasets with incredible amounts of parameters, so that understanding a single parameter's contribution is not useful in the real world.

In the near future AI and statisticians will need to cooperate by:

1. Finding means for extracting rules/general principles from neural networks.

2. Creating new fields of statistics that can handle multi-dimensional data sets, such as by representing them as small-dimensional datasets that interact (?), which converges to the same multidimensional model when the modeler sutures these interactions (?) together according to a topological structure.

We know that we can do this because somehow human beings reason about complex systems successfully.

>human beings reason about complex systems successfully

Have you SEEN economics?

There’s a whole field around 1 called explainable ai. Interestingly, one of the SOTA techniques, SHOP values, comes from economics (game theory).

Regarding the article, I think it was a good read, but as a data scientist on a research at Amazon, there’s a reason our interviews have shifted from stats-heavy to more cs-heavy: the ability to actually implement and maintain analytic products is just more useful. (Note, this isn’t across the board - we still hire PhD-level candidates to do research).

It's conceivable that humans minds are not perfectly modeled by software neural networks.

If neural networks are not perfect models of our minds, then your last paragraph does not hold.

Have you considered that useful predictive models for reality are simply irreducible to being comprehensible by swollen savannah monkey brains? It's not magic. It simply cannot both be explained to a human and be worth anything.

By the Good Regulator Theorem, if models of reality were incapable of being made comprehensible, then we would have went extinct. What we need to figure out then becomes how to externalize what we know unconsciously.

Sounds interesting, but I'm not sure if I understand.

From Wikipedia: It is stated that "every good regulator of a system must be a model of that system".

How is that related to comprehensibility? Or going extinct?

A lot of smart people seem to be surprised that anything can be comprehended at all:

https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness...

The lack of mathematics effectiveness in description of intelligence seems unsurprising to me.

Can a swollen savannah monkey brain drive a car safely on public roads, using nothing but its attached eyes and ears? Yes (most of the time). Can AI do it? Not yet...

Sure you can drive, but can you explain in details HOW you do it? Probably not and that's why a machine cannot do it yet (until it learns by itself, as we all did).

Not an AI in a publicly available release that can be obtained by Joe consumer.

Supposedly the development versions we the general public can’t get yet are getting pretty good and would meet your “most of the time” bar on public roads.

But we don’t really know how good AI is at any given moment, because those that are leading in any AI area sometimes have reasons for not telling yet. Or because we don’t believe them when they do tell us.

Well, statistics is for those who understood that showing your work on an exam question was the whole point of the exam. ML is for those who just wrote down the final answer and dismissed showing their work as a waste of time. There's no need to understand the 'why' if you already know the 'what'.

I love this and I'm stealing this but I think you maybe didn't take it far enough. To extend your analogy,

Statistics is for places where making meaning of or taking action on the answer must (for practical or normative reasons) include the process

Machine learning is for places where meaning or action can be done without regard for the process.

So, you're saying ML is for situations in which type II errors are considered irrelevant?

Given that every semester is have at least one student who can't conceptualize type II errors and at least one who just can't accept type I error...I don't even k ow anymore

Reply on news.ycombinator.com