Comment by levocardia | Hacker News Reader

Comment by levocardia | Hacker News Reader

Parent Full thread

I see a lot of comments here assuming "linear model" means "can't model nonlinearities." Absolutely not the case. Splines can easily take care of that. The "linear" part of linear model just means "linear in the predictor space." You can add a non-linear predictor easily via spline basis (similar/sometimes identical to "kernels" in ML).

My series of lm/glm/gam/gamm revelations was:

1. All t-tests and ANOVA flavors are just linear models

2. Linear models are just a special case of generalized linear models (GLMs), which can deal with binary or count data too

3. All linear models and GLMs are just special cases of generalized linear mixed models, which can deal with repeated measures, grouped data, and other non-iid clustering

4. Linearity is usually a bad assumption, which can easily be overcome via splines

5. Estimating a spline smoothing penalty is the same thing as estimating the variance for a random effect in a a mixed model, so #3 and #4 can be combined for free

And then you end up with a generalized additive mixed model (GAMM), which can model smooth nonlinear functions of many variables, smooth interaction surfaces between variables (e.g. latitude/longitude), handle repeated measurements and grouping, and deals with many types of outcomes, including continuous, binary yes/no, count, ordinal categories, or survival time measurements.

All while yielding statistically valid confidence intervals, and typically only taking a few minutes of CPU time even on datasets with hundreds of thousands / millions of datapoints.

Widely known in the controls engineering world for the last hundred years as 'anything is linear if you zoom in far enough'

Well, subject to certain constraints.

You aren't guaranteed that your equilibria behave the same in the linearization of a nonlinear system if your Jacobian has any eigenvalues with real part of 0.

https://en.wikipedia.org/wiki/Hartman%E2%80%93Grobman_theore...

It's amazing how many ideas I've seen being "rediscovered" these days from control theory.

Alternatively, anything is linear if you add enough dimensions.

Wait isn't that just an application of Taylor expansion which analysis has known about for several hundreds of years?

Taylor is only for some functions, must be continuos and differentiable to n grades. In control theory you break the functions, so discontinuities can also be considered.

In the 70's in CS grad school at USC, I wrote an adaptive least squares cubic spline fit routine. Kept subdividing intervals and fitting a least squares cubic spline in each interval until a criteria was met.

Fun times.

Don't have the code or writeup any more.

Can you recommend some resources to go from step 3 to 4?

So far I have been successfully using GLMMs where appropriate, but then jumped to implementing completely arbitrary models by fitting them to the data (plus bootstrapping).

But that takes ages for most problems as you say

GLMs are a non linear transformation on an output followed by linear modelling. They are referred to as “linear models” but you might as well then also consider NNs as linear models, or any model at all which ends with addition as the final step.

This sounds really cool but was hard to digest for me as a ML Engineer who came into work just around deep learning and DNNs.

Is there some go-to practice material I could look at?

Splines I haven't touched since numerical computing exercises in school.

Simon Wood's Generalized Additive Model book.

Or, for a more approachable treatment, Semiparametric Regression with R by Harezlak, Ruppert, and Wand. A middle ground between Wood's book (which is comprehensive but can dip into math that's way over my head at times) and H/R/W is Semiparametric Regression by Ruppert, Wand, and Carroll.

I have also heard great things about Frank Harrell's Regression Modeling Strategies which uses a slightly different approach (still spline-based though), but I haven't read it. His other writing is fantastic though.

> Frank Harrell's Regression Modeling Strategies which uses a slightly different approach (still spline-based though), but I haven't read it.

Very very good indeed, with the exception that he basically ignores compute time and efficiency. I learned a bunch, but applying his approach to the kinds of datasets I deal with (much larger and with pretty strict compute budgets) was very difficult.

If you know DNNs, you'll find the introduction of splines in Semiparametric Regression in R very intuitive - splines are introduced using what the authors call a "truncated line basis" but you already know it as the RELU function, just with a bias. Indeed, even the penalization of splines will look extremely familiar: it's basically just L2 regularization to induce smoothness.

You might also enjoy reading the "Neural Additive Model" paper from Hinton's lab, which is basically GAMs using a separate DNN as a "spline basis" for each input variable.

> Splines can easily take care of that.

within set boundaries.

"You can make linear model non linear by adding non linearity" is kind of vacuous statement, isn't it

No no, linear just means some expression of the form

    y = b1*x1 + b2*x2 + ... + bp*xp

which is in the heart of the model (perhaps more specific description would be linear combination). Whatever you call it, it's just some quantity y, that is constructed via an additive process from components x1...xp, and each component is multiplied by some constant (the coefficient of the linear combination).

This linear combination "core" of the model can be used directly (in which case it is geometrically a hyperplane), but you could also use non-linear transformation of the inputs or the outputs.

E.g. x1 could be a spline basis function, or log-transformed data, or anything.

Similarly, the output y can be passed though some non-linearity to force a particular output (like in logistic regression). Hence the statement is not vacuous

Well yeah, but what is the non trivial statement here? Linear model is linear, and if you add to your model non-linear steps (such as pre- or post- processing) then it is not longer linear and can handle non-linear data. In the same spirit, you could say that GPT4 is a linear model (token logits) with a very large preprocessing (transformer).

Maybe what OP meant is that particular non linearities (splines) allow to keep some goodies from the linear model in non-linear settings. Still it's not clear for me in which settings exactly such models are good enough

Reply on news.ycombinator.com