Comment by reify | Hacker News Reader

Comment by reify | Hacker News Reader

All thats wrong with the modern world

https://www.ibm.com/think/topics/linear-regression

A proven way to scientifically and reliably predict the future

Business and organizational leaders can make better decisions by using linear regression techniques. Organizations collect masses of data, and linear regression helps them use that data to better manage reality, instead of relying on experience and intuition. You can take large amounts of raw data and transform it into actionable information.

You can also use linear regression to provide better insights by uncovering patterns and relationships that your business colleagues might have previously seen and thought they already understood.

For example, performing an analysis of sales and purchase data can help you uncover specific purchasing patterns on particular days or at certain times. Insights gathered from regression analysis can help business leaders anticipate times when their company’s products will be in high demand.

While I get your point, it doesn't carry too much weight, because you can (and we often read this) claim the opposite:

Linear regression, for all its faults, forces you to be very selective about parameters that you believe to be meaningful, and offers trivial tools to validate the fit (i.e. even residuals, or posterior predictive simulations if you want to be fancy).

ML and beyond, on the other hand, throws you in a whirl of hyperparameters that you no longer understand and which traps even clever people in overfitting that they don't understand.

Obligatory xkcd: https://xkcd.com/1838/

So a better critique, in my view, would be something that the JW Tukey wrote in his famous 1962 paper: (paraphrasing because I'm lazy):

"better to have an approximate answer to a precise question rather than an answer to an approximate question, which can always be made arbitrarily precise".

So our problem is not the tools, it's that we fool ourselves by applying the tools to the wrong problems because they are easier.

That particular xkcd was funny until the LLMs came around

Aren't LLMs also a pile of linear algebra?

And a pinch of ReLU

That's the point, yes. "Piling up more and more data then stirring it until it works" stopped being a joke and turned out to be a practical approach.

This can be seen as another occurence of the "bitter lesson": http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Uh, it's funny because it works. It came out at a point where that approach was already being used in plenty of applications.

Thanks for the link to The Bitter Lesson.

I indeed find the lesson that it describes unbearably bitter. Searching and learning, as used by the article, may discover patterns and results (due to infinite scaling of computation) that we, humans, are physically uncapable of discovering -- however, all those learnings will have no meaning, they will not expose any causality. This is what I find unbearable, as it implies that the real world must ultimately remain impervious to human cognizance; it implies that our meaning- and causality-based human reasoning ultimately falls short to model the world, while general, computation-only methods (given ever-growing computing power) at least "converges" to a faithful (but meaningless) description of the world.

See examples like protein folding, medicine research, AI-assisted diagnosis, self driving cars. We're going to rely on their results, but we'll never know why those results work. We're not going to reject self-driving cars if those cars save lives per same distance driven and/or same time driven; however, we're going to sit in, and drive, those cars blind. To me, that's an unbearable thought, even apart from the possibility that at some point the system might break down, and cause a huge accident inexplicably. An inexplicable misbehavior of the system is of course catastrophic, but to me, even the inexplicable proper behavior of the system is an unsettling thought -- because it is inexplicable.

Edited to add: I think the phrase "how we think we think" is awesome in the essay. We don't even know how our reasoning works, so trying to "machinize" those misconceptions is likely bound to fail.

Arguably, "the way our reasoning works" is probably a normal distribution but with a broad curve (and for some things, possibly a bimodal distribution), so trying to understand "why" is a fool's errand. It's more valuable to understand the input variables and then be able to calculate the likely output behaviors with error bars than to try to reduce the problem to a guaranteed if(this), then(that) equation. I don't particularly care why a person behaves a certain way in many cases, as long as 1) their behavior is generally within an expected range, and 2) doesn't harm themselves or others, and I don't see why I'd care any more about the behavior of an AI-driven system. As with most things, Safety first!

Well I'd say that prompt engineering is still exactly this?

My maxim of statistics is that applied statistics is the art of making decisions under uncertainty, but people treat it like the science of making certainty out of data.

That sums it up exceptionally well.

Reply on news.ycombinator.com