The Truth About Linear Regression (2015)
stat.cmu.edu
stat.cmu.edu
Another one I would add that is very important: Human beings, especially in groups, can only reasonably make linear decisions.
That is, when we are in a meeting making decisions for the direction of the company we can only say things like "we need to increase ad spend, while reducing the other costs of acquisition such as discount vouchers". If you want to find the balance between "increasing ad spend" while "decreasing other costs" that's a simple linear model.
Even if you have a great non-linear model, it's not even a matter of "interpretability" so much as "actionability". You can bring the results of a regression analysis to a meeting and very quickly model different strategies with reasonable directional confidence.
I struggled communicating actionable insights upward until I started to really understand regression analysis. After that it became amazingly simple to quickly crack open and understand fairly complex business processes.
For simplicity, I’m going to assume each variable in the model is independent of every other variable.
We can interpret the coefficients in linear models. This relationship holds for the model for the range of values it is based on. This relationship is the same for the whole range of the model. (We can’t extrapolate outside of what’s been modeled.)
y = c1x1 + c2x2 +…+ cnxn (excuse the poor formatting)
The sign tells you the direction (+ means it will increase the value of y, - means it will decrease the value of y), the value of the coefficient tells you how much the y will change for a given 1-unit change in the x value.
Since this is linear, you get the same change to the output for the relevant increases no matter your starting point.
So, the regression model would say x1, x3, and x5 have positive coefficients and variables x2, x4 have negative coefficients. If you want y to increase, either start doing more of x1, x3, x5 or do less of x2, x4. Depending on what these are and your limited investment budget, for example, you may pick doing x3 if that is the largest positive coefficient.
Again, since this is linear, you can keep on putting resources into the largest coefficient and get the same increase up until your model is no longer valid.
For non-linear models, you can still interpret the coefficients, but the interpretation depends on your starting conditions and where you are on the graph.
There may be asymptotes in your non-linear model, so there is a point of diminishing returns where if you keep putting resources into a variable with a positive coefficient, this will not keep getting you commensurate results.
Sorry I don’t have any actual examples here and I don’t have time to go digging through my old textbooks to look for any.
What I think is the important part, is that it is better to ask decision makers for decisions on setting a continuous parameter, than to make binary yes/no or go/no-go decisions. When it's a decision by committee, I can see why that is.
No, that's not true. Human groups are very able to make discrete decisions. Actually, often they tend to go for discrete decisions, when something continuous (and perhaps linear) would be a lot better.
(Just to be clear: if you force your linear models to make discrete predictions, they are no longer linear in any sense of the word. That's why linear optimisation is a problem that can be solved in polynomial time, and integer linear optimisation is NP complete.
Even convex optimisation, which is no longer linear but still continuous, can be solved in roughly polynomial time.)
Often people demand more decisive decisions, of 'yes'/'no' or concrete action, not shades of grey and fiddling at the margins.
Getting people to even appreciate linear models is already a step forward. Like it or not, your business strategy meetings are already a step ahead of what most people would naturally be inclined to.
And working with these people is so painful.
In my concrete cases I mostly saw that in the direct sense of being able to deploy more mathematics and operations research, eg for netting out (partially) offsetting financial instruments for a bank.
But by introspection you can come up with more example. Eg that's a common selling point for running your servers on AWS instead of building your own hardware.
a) carefully choose the two most important dimensions of concern (as Alan Kay said: the correct point of view is worth 80 Iq points)
b) make them binary: are we happy here or do we need to change?
In a way similar to the pareto ratio, you keep a surprising amount of value in something “so simple it cant be possibly so useful”.
There are entire statistics textbooks devoted to multilevel linear models, you can get really far with these.
Shrinking through information sharing is really important to avoid overly optimistic predictions in the case of little data.
[1] https://cran.r-project.org/web/packages/ashr/index.html [2] https://github.com/mengyin/vashr
R's mgcv package (which does all of the above) is probably the single reason I'm still using R as my primary stats language.
There are absolutely decisions that need to get made, and do get made, that are not linear. Step functions are a great example. "We need to decide if we are going to accept this acquisition offer" is an example of a decision with step function utility. You can try to "linearize" it and then apply a threshold -- "let's agree on a model for the value at which we would accept an acquisition offer" -- but in many ways that obscures that the utility function can be arbitrarily non-linear.
That 0/1 input variable could also have arbitrary interactions with other variables, which would also amount to “step function “ input effects.
See for example the autism/age setup down thread.
But the parent comment is not talking about constrained optimization, just gradient following.
In the context of this post, that’s just “which of these N discrete variables, if moved from 0 to 1, will increase the quantity of interest according to the linear model?” “Which will decrease it?”
The question is not, “if I can only set M of these N variables to 1, which should I choose?”
That’s a good question, and it leads to problems in NP, but that’s not what the comment was referring to.
Yes, you are right in that abstract setting.
If you always have the full hypercube of available, the problem is as easy as you describe. But if there are constraints between the variables, it gets hairier.
If all the numbers that appear in your constraints are rational (p/q with finite p and q), then any solution is also a rational number (with finite nominator and finite denominator).
(Well, any finite solution. Your solution could also be unbounded, then you might have infinities in there.)
A computer can represent finite rational numbers just fine. See eg https://docs.python.org/3/library/fractions.html or https://hackage.haskell.org/package/base-4.20.0.1/docs/Data-... for some libraries.
Though in most cases, people just use floating point numbers in practice, but that's of no philosophical concern.
This seems to be getting a lot of attention. I couldn't agree more, we assume linearity all the time because reasoning non-linearly is exceptionally difficult. Yes we can do it sometimes, but it is not the default. Reasoning linearly has its flaws, and we should recognize we are making an imperfect decision, but it is still extremely useful.
gam(temperature ~ te(long, lat) + s(year) + ti(long, lat,year))
You still can't do arbitrary high-order interactions like you can get out of tree-based methods (xgboost & friends) but that's a small price to pay for valid confidence intervals and p-values. For example, the model above will give you a p-value for the ti() term, which you can use as formal statistical evidence to say -- at what level of confidence -- a spatiotemporal trend exists.
This Rmarkdown file (not rendered sadly) shows how to do this and other tricks https://github.com/eric-pedersen/mgcv-esa-workshop/blob/mast...
A common problem I encounter in the literature is authors over-interpreting the slopes of a model with quadratic terms (e.g. Y = age + age^2) at the lowest and highest ages. In variably the plot (not the confidence intervals) will seem to indicate declines (for example) at the oldest ages (example: random example off internet [1]), when really the apparent negative slope is due to the limitations of quadratic models not being able to model an asymptote.
The approach I've used, when I do not have a theoretically driven choice to work with) is using fractionated polynomials [2], e.g. x^s where s = {−2, −1, −0.5, 0, 0.5, 1, 2, 3}, and then picking a strategy to pick the best fitting polynomial while avoiding overfitting.
Its not a bad technique; I've tried others like piecewise polynomial regression, knots, etc [3],but I could not figure out how to test (for example) for a group interaction between two knotted splines). Also additive models.
[1] https://www.researchgate.net/figure/Scatter-plot-of-the-quad...) [2] https://journal.r-project.org/articles/RN-2005-017/RN-2005-0... [3] https://bookdown.org/ssjackson300/Machine-Learning-Lecture-N...
[1] https://cran.r-project.org/web/packages/gratia/vignettes/gra...
I'd love to see tools in the ecosystem around extracting relevant features that then can be used on a lower cost, more predictable model.
i've met smart people that cant wrap their head around how it's possible to create linear model where the number of parameters exceeds the number of data points (that's an OLS restriction).
or they're worried about how they can apply their formula for calculating the std error on the parameters. bruh, it's the future and we have big computers. just bootstrap em and don't make any assumptions.
Linear models have many solutions fitting the data exactly in that parameter regime, many more fitting it approximately for any metric still satisfying the idea that identical outputs are preferable, and sometimes multiple solutions even with more data.
So.....not just for OLS, but for most metrics (where you'd prefer to match or approximately match the data), the parameters are underconstrained.
How much that matters depends on lots of things. If you have additional constraints (a common one that's particularly easy to program is looking for a minimum-norm solution), that trivially solves the problem. Otherwise, you might still have issues. E.g., non-minimum-norm solutions often perform badly on slightly out-of-distribution samples (since those extra basis vectors were unconstrained and thus might be large).
Is there something I'm missing where 'linear models' are used to represent something wildly different than I'm used to? Are people using norms with discontinuities or something in practice? Is the criticism of OLS perhaps unrelated to the overparameterization issue? I think I'm missing some detail that would relate all of those.
> If you want to convert people into loving linear models (and you should), we need to make sure that they learn the difference between 'linear models' and 'linear models fit using OLS'
Help me understand the pitch. What linear models are you referring to here that aren’t estimated with OLS? How should I wrap my head around having more parameters than observations?
Models that can cope with more parameters than observations: - Ridge - Lasso
Point 3 — just pick the right basis — is very difficult outside a handful of kernels that are known to work. And how are you going to extrapolate your spline for prediction for example? Linearly is usually the answer…
Point 4 — sure for differentiable functions, but most people are fitting data not functions, and if you know it’s generating function why would you bother with a linear model?
Take a simple linear model involving a test score, their age in years (age range 7-16 years), and a binary categorical variable autism diagnosis (0=control,1=autism): score = age + diagnosis + age:diagnosis score = (X1)age + (X2)diagnosis + (X3)age:diagnosis.
If the X2 is significant, the naive student would say, "look a group difference!!", not realizing this is the predicted group difference at the intercept, which is when participants were 0 years old. [[ You center age by the mean, or median, or better yet, the age you are most interested in. Once interactions are in the equation, all "lower order" parameter estimates are in reference to the intercept.]]
They might also note a significant effect of age, and then assume it applies to both groups, but the parameter X1 only tells you what the predicted slope is for the reference group (controls), while the interaction tests if the age slopes differ between groups...moreover, even if the interaction isn't significant, the age effect in the autism group might not significantly differ from zero...the data is in the wish washy zone, and you have to be careful in how one interprets the data.
To some here all this will seem obvious, but to many, getting their head firmly into the conditional space of parameters when their are interaction terms takes work. (note: for now I am ignoring other ways of coding groups (grand mean vs one group being the reference) but the lesson still remains. Understand what the intercept means and to whom/what it refers.
[1] https://www.bioconductor.org/packages/release/bioc/html/Expl...
No, the way you wrote the equation, X1 is for the entire group. You didn’t make a proper dummy variable.
X1*age*isControl+X2*isControl+X3*isAutism+X4*isAutism*age+X5*age
A significant loading on diagnosis (X2) does not tell you anything about the effect of diagnosis at any particular age (except age 0).
You’d have to recenter the model about the age of interest.
A big weakness of linear regression that I had to learn the hard way is that the academic assumptions for valid interpretation of the coefficients are easy to construct for small educational datasets but rarely applicable to messy real world data.
With regard to other assumptions, e.g. normality of the residuals, linear models can often deal with some degree of violation against those. But I agree that it's always good to understand the influence of those violations, e.g. by using simulations and making p-value histograms of null-data.
In a related problem, covariance matrix estimation, variants of shrinkage is popular. The most straight forward one being Linear Shrinkage (Ledoit, Wolf).
Excepting neural nets, I think most people doing regression simply use linear regression with above type touches based on the domain.
Particularly in finance you fool yourself too much with more complex models.
Call it 2000 liquid products on the US exchanges. Many years of data. Even if you approximate it down from per tick to 1 minutely, that doesn't feel like you're struggling for a large in sample period?
These may be valid assumptions, but even if they are, "sample size" is always relative to between-sample unit variance, and that variance can be quite large for financial data. In some cases even infinite!
Regarding relativity of sample size, see e.g. this upcoming article: https://two-wrongs.com/sample-unit-engineering
There is no big mystery I'm afraid, there is no big reveal. It's as Jim Simons described in the Numberphile video interview: a slow painstaking accumulation of weak signals, plus crafting and improving various boxes of the system. (the interfaces between them are largely known) The fitting method used does not buy that much in the grand scheme of things - as long as it does not ruin things, that is.
(I've not been at Citadel but been quant R&D&trading last 20yrs)
For my cs phd I looked mostly at regression problems using deep learning models. I didn't look at this specifically but I still think it would be neat if there is some way to translate the rigid proofs and theorems for classical linear models to deep regression models.
https://www.stat.cmu.edu/~cshalizi/ADAfaEPoV/
It is rightly over-weight on linear and additive models and simulation. 90% of the book is useless without a computer but that is a modern truth.
https://arxiv.org/abs/2303.14151
Double descent is a surprising phenomenon in machine learning, in which as the number of model parameters grows relative to the number of data, test error drops as models grow ever larger into the highly overparameterized (data undersampled) regime. This drop in test error flies against classical learning theory on overfitting and has arguably underpinned the success of large models in machine learning. This non-monotonic behavior of test loss depends on the number of data, the dimensionality of the data and the number of model parameters. Here, we briefly describe double descent, then provide an explanation of why double descent occurs in an informal and approachable manner, requiring only familiarity with linear algebra and introductory probability. We provide visual intuition using polynomial regression, then mathematically analyze double descent with ordinary linear regression and identify three interpretable factors that, when simultaneously all present, together create double descent. We demonstrate that double descent occurs on real data when using ordinary linear regression, then demonstrate that double descent does not occur when any of the three factors are ablated. We use this understanding to shed light on recent observations in nonlinear models concerning superposition and double descent. Code is publicly available
I particularly like chapter 6, visual diagnosis. Very well done.