For the first section, where he uses a Hessian and finds he really only needs one step for linear regression -- isn't he basically doing a newton method, and it works more or less perfectly b/c with the normal squared error term (L2 norm) his problem is exactly quadratic? I haven't unpacked his math carefully but I think (quasi)newton methods are often worth a look if (a) your dimension is small enough that the the D^2 hessian is ok, and (b) your data is small enough that you can afford to us exact derivatives rather than mini-batch estimates.