How are PCA and SVD related?

For those looking for a more succinct answer: https://stats.stackexchange.com/questions/134282/relationshi...

And here is another interesting connection between PCA and ridge regression: https://stats.stackexchange.com/questions/81395/relationship...

I don't understand why people create these webpages just re-explaining stuff that can be read in a book, lecture notes (usually available freely online), or wikipedia. It just adds more noise to the internet. Is it a kind of marketing thing to show their customers that they know what they are doing?

6 word answer

PCA is the SVD of A'A

PCA is a statistical model -- the simplest factor model there is. It deals with variances and covariances in datasets. It returns a transformed dataset that's linearly related to the original one but has the first variable with the highest variance and so on.

SVD is a matrix decomposition. It generalizes the idea of representing a linear transformation (with same dimensions in domain and codomain) in the basis of its eigenvalues, which gives a diagonal matrix representation and a formula like A = V'DV.

SVD is like this, but for rectangular matrices. So you have two matrices to diagonalize: A = U'DV.

That SVD even performs PCA as noted in the algorithms is a theorem, albeit simple one usually given as an exercise. But hey, even OLS regression can be programmed with SVD if you want to.

I've always understood PCA as SVD on a whitened matrix. Is this too simplistic of a view to take wrt implementation?

https://en.m.wikipedia.org/wiki/Whitening_transformation

The connection between these two has always been hazy to me. I often mixed up the two when talking about each of them independently.

This article was well-written, exactly precise enough, and cleared up the confusion. Thanks for sharing!

SVD is the decomposition of a matrix into its components.

PCA is the analysis of a set of eigenvectors. Eigenvectors can come from SVD components or a covariance matrix.

source: http://www.eggie5.com/107-svd-and-pca

Great article, the lecture comparing the two from John Hopkins, part of the Data Science Specialization on Coursera also offers a great explanation.

"Because vectors are typically written horizontally, we transpose the vectors to write them vertically". Is there a typo in this sentence or is to just too early in the morning for me to read this?