Yeah but let’s not go crazy. Linear models perform very badly on partition-able tabular data where tree models excel. They are also obviously no replacement or competition in deep learning related tasks.

Point 3 — just pick the right basis — is very difficult outside a handful of kernels that are known to work. And how are you going to extrapolate your spline for prediction for example? Linearly is usually the answer…

Point 4 — sure for differentiable functions, but most people are fitting data not functions, and if you know it’s generating function why would you bother with a linear model?