An Introduction to Gradient Descent and Linear Regression

Agreed. Someone might actually implement linear regression in the way described! That would be embarrassing.

Another thing worth mentioning is the behavior of gradient descent for problems where the Hessian is not well-conditioned (i.e., highly eccentric elliptical contours in the cost function). This would be well-illustrated by a simple example - I think a case where the

  y = a x + b 

line had a very large a would do the trick, because then a and b would be tightly coupled (choose the wrong "a" and the intercept would change a lot, due to the large slope).

The number of iterations will skyrocket, which is why gradient descent is avoided if possible in favor of conjugate gradient or related approaches.

Also, he states that "Lines that fit our data better will result in lower error values". Strictly speaking, that is not true. The error function is a mathematical model for reality; the result is a 'best fit in a linear least squares fit' sense only. I.e you are fitting your model, not necessarily reality - what the data represents.

Sure, that is all implied in the article, and I don't believe that the author is confused at all by this point. But I witness so many people just running a linear regression, claim it's "optimum", and go on to conclude grossly incorrect things. So I think it is worth stressing (repeatedly), that this is 'optimum' only in the very strict sense of minimizing your error function.

Let's take a problem I am working on at the moment - tracking an object in flight. The data is noisy, but in a complicated way. Yes, you have the typical Gaussian measurement noise, but then you also have outlier measurements that have nothing to do with the object you are trying to track. Blithely applying least squares unduly weights the bad measurements because of the distance squared amplifying the influence of points that lie far away from the real trajectory. I could run gradient descent on the data (we tend to use Levenberg-Marquardt), but boy, the output will not model reality very well. It's optimum, but it is also wrong.

So, for my data, the sentence should read "Lines that fit our data better will result in higher error values". Well, the relationship is more complicated that that, but you get the idea.

I'm not dismissing the article (I liked it), just discussing the next steps you have to start thinking about. If you know you have linear data with Gaussian noise you'll get a meaningful minimum using this approach. If that is not true, then do not trust this to give you meaningful information. Get thee to a University, go (which is what the author also suggests at the end).

You need it for l1 and l2 regularization which is popular these days don't you?

Author here. These are all great points. I chose linear regression because it's simple and easy to understand. There is obviously a simple closed form solution to this problem. However, as already mentioned, gradient descent has some nice advantages like scaling. Further, a "close enough" solution/approximation might be sufficient depending on the application.

Actually, although the direct method is often faster than gradient descent, it is not faster than conjugate gradient.

And conjugate gradient is more numerically stable. So the question is, why would anyone competent use the direct method for linear regression?

This was my introduction - I used the techniques described to write an interplanetary trajectory timing optimization tool. (finding good launch windows between various celestial bodies). It was an excellent read and eminently transformable into functional code.

Correct... if there is only one minimum, it is unimodal, but not necessarily convex. For instance, if there are inflection points, it will not be convex.

Yes - see e.g. "On the importance of initialization and momentum in deep learning" from ICML 2013 (http://www.cs.toronto.edu/~fritz/absps/momentum.pdf) for an overview in the context of NNs.

Accelerated gradient descent is pretty common everywhere.