Hacker News Reader

Comment by ratsbane | Hacker News Reader

This is exactly one of the things that makes ML, AI, or whatever so interesting. Sometimes it does find correlations which a human might miss because the human thinks there is no way those two things are linked - but in fact they are.

"For example, a correlation may exist between changes in temperature in an obscure Australian town and price changes in the U.S. stock market. A person would know that the two events have no connection."

View on HN

Recommended reading: “Everything is correlated”

https://www.gwern.net/Everything

Indeed, I can imagine a few mechanisms for such a correlation.

1. El Nino / Southern Oscillation affects the temperature in Australia and (especially South) America which would affect farm yields and hence US stock market prices.

2. Extra hot temperatures at a uranium mine near a town in Australia might limit productivity of that mine, causing uranium prices to change.

Just two off the top of my head. The effects might be only very mildly correlated, but not zero.

If you're willing to entertain that many hypotheses, you need a huge amount of data to not be fooled by noise. For instance, there might be 1B hypotheses of the form "temperature in some town is correlated with the price of some stock". You'd need p<0.000000001 to not have false positives. Even the stock market doesn't have enough data for that.

My objection is to the absoluteness of the phrase

> A person would know that the two events have no connection.

When a negative statement like this is presented as absolute and categorical it's very easy to disagree with it.

I'm genuinely confused if you're saying ML is better or worse for finding such correlations, because humans are also extremely good at finding patterns where there's none, as this website hilariously demonstrates:

https://tylervigen.com/spurious-correlations

And sometimes it cheats, when a criminal matcher turns out to be a smile and skin tone detector.

Reply on news.ycombinator.com