Hacker News Reader

Comment by brendamorgan | Hacker News Reader

Curse of Dimensionality refers to non-intuitive properties of data observed when working in high-dimensional space, specifically related to usability and interpretation of distances and volumes.

View on HN

You can get around the curse a little bit by using shared nearest-neighbor methods, but that only helps you a little bit.

Although, the curse of dimensionality is no precise. The real problem is due to intrinsic dimensionality. There's a nice paper on how to estimate it http://www.nii.ac.jp/TechReports/14-001E.pdf

I remembered someone telling me that they prefer cosine similarity over euler many times. I was looking into reason why and found an interesting paper "On the Surprising Behavior of Distance Metrics in High Dimensional Space"[1] which at some point argues that fractional metrics are perform better than more widely used distance metrics on high-dimensional space

[1] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.23....

There is a cornucopia of distances to choose from. It's not as simple as "just use the best one." Cosine similarity assumes a certain projection of data (by ignoring distance of points are from the origin). It preforms well whenever that's a good projection. I think it's best to think of it as one way to reduce dimensionality, and should be used when you think that's the right way to reduce dimensionality.

Reply on news.ycombinator.com