Hacker News Reader

Comment by civilized | Hacker News Reader

You've almost grasped the point. Mixture distributions with a missing underlying feature are how outliers (in the meaningful, well-defined sense) are generated. And if you dig into this paper (I have not yet), I can almost guarantee you that this is foundational to the paper.

View on HN

It's kind of funny just how long I had to work with statisticians to see that stats people and physics people look at generative processes in two completely different ways:

"There are two cultures in the use of statistical modeling to reach conclusions from data. One assumes that the data are generated by a given stochastic data model. The other uses algorithmic models and treats the data mechanism as unknown. The statistical community has been committed to the almost exclusive use of data models"

I can tell you from extensive experience that domain knowledge is more critical to understanding such issues than this algorithm. Fundamentally, such causes can not be identified if they don't exist in the data and the algorithm doesn't "see" them. Like the previous example where the identity of the object being hurled is hidden and all the data shows is the distance. And the causes change a lot from field to field.

I don't believe that your extensive experience tells you about the utility of an algorithm you've never used that was developed by Amazon researchers most likely to help with problems at Amazon.

Reply on news.ycombinator.com