Hacker News Reader

Comment by civilized | Hacker News Reader

Now that we've all said "feature selection is not dimensionality reduction" to our hearts' content, could we return to the point of the article?

Regardless of whether you're doing feature selection or dimensionality reduction, the point remains that, if you're doing supervised learning, PCA is just compressing your X space, without any regard to your y. It could be that the last principal component of X, containing only 0.1% of the variance, contains 100% of the correlation between X and y.

Using PCA for dimensionality reduction in a supervised learning context means throwing out an unknown amount of signal, which could be up to 100% of the signal.

Now for unsupervised, exploratory analysis, PCA is definitely a candidate, but there are plenty of often-better alternatives there too.

View on HN

For me PCA is strictly in a bucket labeled "this comes up a lot in data science interviews but you never actually need to use it at work".

For me the contents of this bucket are things like SVM, Naive Bayes, and k-means. PCA is actually in a slightly higher tier than those three.

k-means are actually useful for bootstrapping statistical mixtures when you don't have good priors.

Just for this convenient use alone I would place them above PCA.

What does bootstrapping statistical mixtures mean?

Why not just bootstrap in the regular old non-parametric way? Why inject k-means into it?

When you try to approximate your data with a mixture of distributions, say Gaussians for instance, you have to initialize their parameters with some values before running your EM loop.

Finding good ones can be very problematic. The k-means process is a reasonable method to get good enough starting values without having to think/compute too much.

https://en.wikipedia.org/wiki/Expectation%E2%80%93maximizati...

OK, so for you, "bootstrapping statistical mixtures" means "warm-starting EM mixture modeling".

Personally I never would have guessed that, in retrospect it makes sense... to each their own!

You've never had to any kind of factor analysis in your work or done any searching for latent variables that map to customer/stakeholder question? Given the number of people I've worked with that are interested in modeling "engagement", I find this hard to believe.

PCA is an incredibly valuable tool that I've used in most jobs I've had. It's just a terrible idea as a default part of a feature engineering pipeline (which is what the author is talking about in terms of "feature selection"), for reasons outline in this article.

I suggest you don't be quite so quick to dismiss important concepts in this area, and before criticizing this post, at least read through it (I noticed your comment about misunderstand what the author is discussing by "feature selection" is the top comment here).

> You've never had to

Nope, never. I'm not dismissing PCA altogether; I'm sharing my experience and pointing out that some topics come up much more often in interviews than on the job.

My experience is the opposite of yours. It has never come up in a job interview but I have used it for real work several times.

If you're looking for the part of the signal in X that covaries with y, the first thing to try is partial least squares.

Yes. In the simplified linear setting we're considering...

If you want to do dimension reduction and only have X, use PCA.

If you want to do dimension reduction and also have a "y", consider PLS or CCA (canonical correlation analysis).

Some background: https://www.cs.cmu.edu/~tom/10701_sp11/slides/CCA_tutorial.p...

You nailed it.

Reply on news.ycombinator.com