Hacker News Reader

Comment by tel | Hacker News Reader

In practice if you're working in high dimensions and don't do PCA then you have very little chance of building a good model. It certainly isn't linearly valid if to you valid demands scale invariance, but it's an essential tool for unsupervised feature engineering.

View on HN

I definitely agree that you need dimensionality reduction, though there are methods other than PCA (e.g. auto-encoders.)

Auto-encoders have some big advantages over PCA, but they suffer the same shortcoming (sensitivity to units of measurement) described in the original post.

In practice you have to scale data to [-1,1] or [0,1] for a neural net anyway, right? (Depending on the kernel function.)

Another common method is to scale to mean 0, variance 1. In my opinion this makes more sense since it handles outliers a bit better-e.g., consider a case where most of your values for a feature are scaled from 1 to 10 but there's one point with value 1,000,000.

I agree that you SHOULD normalize/scale data before running neural networks and autoencoders.... and this resolves the units issue in most cases (unless measurements in some units are non-linear functions of measurements in others).

But this scaling also resolves the issue for PCA. So, I don't see much difference between autoencoders and PCA with regards to original post's "dimensional invalidity" concern.

If anything, the scaling options you mention suggest "dimensional invalidity" isn't a big deal in practice for either method.

Regularization is an excellent alternative to dimensionality reduction in most cases.

To expand on this a bit (please correct me if I'm wrong), there are three major uses of dimensionality reduction:

1. Reduce overfitting

2. Train models faster

3. Take advantage of unsupervised data

Regularization handles case (1) quite well. It can also be used in conjunction with most methods of dimensionality reduction such as PCA/auto-encoders.

PCA covers all three, but isn't as effective at dimensionality reduction compared to auto-encoders.

Auto-encoders tend to yield a better compression than PCA but take more time to train and produce output that's harder to understand. There is a bit of analogy here, auto-encoders are to PCA what neural nets are to linear regression.

L1 regularization can also be used for feature selection[0] and hence dimensionality reduction. I've in found practice that the speed of this can compare favorably even to fast implementations[1] of svd.

[0] Feature selection, L1 vs. L2 regularization, and rotational invariance, Andrew Y. Ng. In Proceedings of the Twenty-first International Conference on Machine Learning, 2004. http://ai.stanford.edu/~ang/papers/icml04-l1l2.pdf

[1]"Augmented Implicitly Restarted Lanczos Bidiagonalization Methods", J. Baglama and L. Reichel, SIAM J. Sci. Comput. 2005.

Reply on news.ycombinator.com