A First Course in Linear Algebra

PD: If you really do like or need the algebraic approach, Shilov's book is the way to go after this course... I think the first chapter is about determinants and he builds the rest of the book from there. It also includes a chapter on tensor algebra.

As a math professor, I think you can get by without defining vector spaces -- I think it is a little bit difficult to come up with examples, other than R^n or C^n, which seem motivated and interesting to the beginner. The truly important example (IMHO) is R^n without a choice of basis -- but I think this can only be well motivated after you've seen a lot of linear algebra, not before.

Mechanics of matrix operations are not pleasant to teach, they make the subject seem like a bunch of contrived and confusing examples. Same for the "row echelon" stuff. Alright, you now have an algorithm for solving systems of equations... but presenting linear algebra as an algorithm sells it short.

What is really important in my view is the geometry of the subject, which is already very interesting in two dimensions. Problem: Here is a linear transformation, given as a 2x2 matrix. Draw a picture which illustrates what this does to the plane. If you ask me, this is much more important than most of the crap that gets taught and tested in most courses in linear algebra.

Another resource I've used is Patrick JMT's excellent videos found at: http://patrickjmt.com. He really goes over problems slowly. Best math teacher I've ever had.

http://hbpms.blogspot.com/

btw, I couldn't use Singular Value Decomposition in numeric.js for PCA because the method, numeric.svd, uses the "thin" algorithm, and throws an error if there are more columns than rows. I calculate way more features (50-200+ columns) than I have training samples (rows, 10-30 written manually). without svd I had to use the "covariance method", which I guess can sometimes present approximation issues, but seems to be working well for me.

The purpose of the PCA is dimensionality reduction (google "curse of dimensionality"). I had used Mahalanobis Distance as a p-value score to detect outliers (p-val < 0.05), and it worked well when there were only 6 features. Curse of dimensionality makes MD useless when there are 50 or 100 features, and PCA reduces them to 3-10 features which carry the most information with the rest approaching zero. So I do MD on the projected (reduced) features, and its working great.

If I do it all over again I might try a "one-class SVM" (which, sadly, I had not heard of until only recently and very late in the project). SVM's are non-linear like most machine learning algos, but the linear PCA can still be used to complement other methods, eg to do pre-processing before feeding to a neural network.[1]

For deeper linear methods, check out the PCA-related extensions like Fisher Linear Discriminants or Projection Pursuit.

1. "Many neural networks are susceptible to the Curse of Dimensionality though less so than the statistical techniques. The neural networks attempt to fit a surface over the data and there must be sufficient data density to discern the surface. Most neural networks automatically reduce the input features to focus on the key attributes. But nevertheless, they still benefit from feature selection or lower dimensionality data projections."

It is very good and explains all the proofs and definitions very well.