Linear Hammer: PCA is not a panacea (2013) | Hacker News Reader

Linear Hammer: PCA is not a panacea (2013)

danluu.com

53 pointsPaulHoule20 comments

> They're useful. But I keep seeing people try to apply the same four linear methods to every problem in sight.

Nowadays I see the reverse. People loading in huge huggingface models, when a simple linear regression would have been fine.

While PCA can be useful for some classification problems, it is intended to be a dimensionality reduction technique, not a classification technique.

If you're trying to make a classifier, have a crazy high dimensional space, and don't have a lot of training examples, dimensional reduction can be a lifesaver.

With simple text classification problems (is this a biomedical abstract or celebrity gossip?) and bag-of-words features I've seen dimensional reduction cut the error rate in half.

That’s what confused me.

The problem with this line of argument is that this dimensionality reduction is being done without paying any heed to the end goal better linear separation of the classes. One can get lucky and get a low-d projection that separates well but thats pure luck.

Let me see if I can draw an example

           ++++++
       ++++++++++++++++
             +++
           ---------
       -----------------
            -------

The '+' and '-' denote the data points of the two different classes. In this example the PCA direction will be the along the X axis, which would be the worst axis to project on to separate the classes. The best in this case would have been the Y axis.

A far better approach would be to use a dimensionality reduction technique that is aware of the end goal. One such example is Fisher discriminant analysis and its kernelized variant.

Where does the the idea that PCA is optimal come from?

I understand the second example (girlfriend taking a phone screen): some people never took statistics past a linear models course, so they use what they know. Understandable, if inefficient. But they don't seem to be claiming linear models are optimal all the time, they're just stuck in a local maximum of trying to speed up the system they have.

But the PCA guy I really don't understand.

PCA is pretty close to optimal in situations where you want to preserve global structure and variance-covariance matrices are highly informative. This comes down to the min-max theorem for Rayleigh quotients; I won't grandstand going over the math here. It won't work well for gene sequences or text embeddings because the desired structure there is local (UMAP seems to reign supreme these days for that).

An older explanation for PCA is that it's a sort of default factor model prior to rotations that introduce structural assumptions. (Varimax etc. factor analysis is really underrated in exploratory statistics; but then, by now data science training never introduces people to the FWL theorem, identification, etc. With large enough deep learning pretty much anything is possible -- also because using deep learning implies having truckloads of data -- but the median xgboost guy is way out of his element.)

"and variance-covariance matrices are highly informative"

So - a distribution which is linear?

I don't think that the distribution needs to be linear... simply the correlation between N elements needs to be linear (or 1st order dominant). It works great for estimating multiple (e.g. different causes) small deviations (over multiple elements) from a global response. If you're looking at those larger eigen values, then you're looking at the dominant mode shapes of the eigen vectors of correlation.

As others have said, it's better for estimation (not for classification).

You keep using that word.

A multivariate Gaussian, for example.

PCA is optimal in that it returns orthogonal axes in directions of maximum covariance. It can be cast as an optimization problem.

It is definitely not optimal in a classification sense.

(2013 - had to check, there was no way talking about LSI could be recent)

So what do people do nowadays

In general these are referred to as embedding techniques PCA/SVD is still common along with other linear methods like ICA, CCA, PLS t-sne is pretty popular statistically inspired with some manifold learning non-negative matrix factorization is useful recommender systems topologically inspired techniques are gaining traction, like UMAP, LLE the author notes neural embeddings, like autoencoders.

Around the time that this article was written I was putting the finishing touches on a neural autoencoder that was part of a search engine for patents. (The training was done with assembly language SIMD instructions so we weren't in a hurry to make any changes in the network, training algorithms, ...)

The search engine worked amazingly well. When we first demoed it we heard from the USPTO in the first week, they wanted to buy a license...

So what do you work on now if you were that ahead?

UMAP.

Obligatory Lior Pachter tweet (https://twitter.com/lpachter/status/1431325969411821572) [doesn't mention issue of initialization explictly but it's probably buried in the paper or the cited tweets]

Yeah. I was kind of infatuated with the UMAP paper when it first came up (and working through it to build intuition led me to persistent homology and other fun stuff), but it's far too complex to work as a "plug-in tool" that people naive to statistics can blindly use.

This is the power of ANOVA, CCA, linear regression and factor analysis, etc. It's much, much, much, much harder to shoot yourself in the foot, particularly with experimental data (usage in the social sciences is another bag of johnsons.)

Reply on news.ycombinator.com