The basic concept of least-squares modelling

This is a nice interactive example, but it doesn't explain _why_ one would prefer to use a least-squares estimator over any other arbitrary method. For example, one could measure the error in any other L_p norm - e.g. L_1 or L_{\infty} instead of L_2. What's so special about this one?

To answer that, we have the Gauss-Markov theorem [1] -- provided certain assumptions regarding the data are met, least squares is the best linear unbiased estimator.

Note, however, that no-one is forcing us to choose an estimator that is either linear or unbiased. What if by dropping one of those two somewhat arbitrary constraints we could get a better estimator?

One example of this is the James-Stein estimator [2], which is a biased estimator that has the amusing property of dominating (i.e., always performing no worse than, and performing strictly better than in some case) the bog standard least-squares estimate when one is dealing with data in three or more dimensions (and a few more common assumptions).

Crudely speaking, the James-Stein estimator produces estimates that are biased toward the origin, when compared to estimates produced from ordinary least squares. It is also possible to define a James-Stein estimator that biases toward an arbitrary fixed non-origin vector -- these variants also dominate the bog-standard ordinary least squares estimator! Personally I find this result both intriguing and also reasonably offensive (which means my mathematical intuition needs adjustment).

[1] - https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem

[2] - https://en.wikipedia.org/wiki/James%E2%80%93Stein_estimator

@shoo: Pretty interesting, Stein. Thx. Taleb advocates abs. deviation, mean abs. dev. and the like.

OP named reason to pick least squares: "easy to calc" IIRC, the rest of the rationale didn't exist back when it was invented and picked....

When explaining the least-squares, why not drawing the damn squares in the chart then?