Hacker News Reader

Comment by foobarian | Hacker News Reader

I'm not sure. The author says that adding a common component to two random time series doesn't make them correlated. But that's not true, by construction, at least using any of the simple correlation tests. It's a complicated subject explained in a confusing way.

View on HN

I don't get it either. The author is adding a common dependency to two independent series makes them correlated, which to me seems trivially true. He then goes on to say that this form of correlation is uninteresting, because we're only interested in whether variations are correlated, which doesn't seem so trivially true: after all, a linear trend is a first-order variation, and if two things both increase at the same rate, then their relationship is worth at least a second look.

It looks like what the author is really trying to say is that we should pass the data through a high-pass filter, eliminating any 'expected' trends such as inflation, and instead observing if the noise of the two datasets is correlated. This is an observation that has some value, but is certainly not trivial to pick a threshold for the high-pass (certainly it's not always just a linear trend), and the mutually dependent variable can have as much noise (if not more) as the two measured data sources, so you still might get "false" correlation.

It seems to me that in the end it all comes down to your actual hypothesis whether and what is actually interesting …

Adding a common deterministic trend to two random time series does not make them correlated. Intuitively, you still cannot predict the deviation from the trend of one timeseries using the deviation from the trend of the other timeseries.

His statement (after adding the constant trend) is misleading:

"Now let’s repeat the same tests on these new series. We get surprising results: the correlation coefficient is 0.96 — a very strong unmistakable correlation"

What he's calculated is the correlation between a set of points from y_1 and y_2 - and that will be large, of course - their (deterministically increasing) means have correlation 1.

The quantity that qualifies as correlation for predictability purposes is actually the correlation between the deviations from mean.

This is all fairly clear if you use the actual formula (Pearson correlation):

E[(y_1 - mu * t)(y_2 - mu * t)] / (sigma_y1 * sigma_y2)

Increasingly I'm finding that any description of causation/correlation which (a) isn't written by a more or less formally trained statistician of some variety or (b) invoking Pearl-like causal diagrams directly has a coin-flip's chance of being actively dangerous to read.

Causation is super tricky business. You probably have an okay model of it intuitively once you've done a little stats work, but to explain it to someone else or tackle it in more exotic or hypothetical domains is an exercise is insanity if you're not armed with the very best tools.

Causation is dead simple---if you can directly manipulate the putative causal variable. Otherwise, you're left with the old saw about correlation, and while you might be able to tell a story about why there should be causation, the best you can do in that case is eliminate other potential confounds.

It's simple in models where you assume you understand the causal structure. Then whenever things get murky people tend to try to operate without clarifying the causal structure and end up in a bind.

Of course what you say is true, but executing it in practice and having a sufficiently rich language to identify confounders well is what's usually lacking.

You missed the point. This has to do with "time series data" which means time is the same among all the data sets. Because you know this you should remove it to test for similarity. You could apply this to any automatic variable in your domain but time & change is very obviously a typical common variable for many data sets.

Please help me understand. Assume I have 100 years of atmospheric CO2 levels and 100 years of temperature data over the same span. These are both "time series data". What are you suggesting that I remove before testing for similarity? Does this help me to determine the degree to which one of the two variables is causing the other?

You should first break out the relationship between CO2 and time, and the relationship between temperature data and time. You effectively normalize both for time.

So, in effect, you model the correlation between CO2/time vs. temperature/time. The interesting correlation would be if every time CO2/time deviates you can find a correlation with a temperature/time deviation.

I think time - then you're left with data. If when removing this they still have the same shape they do in fact correlate. I'd be curious to see the results of that experiment to be sure.

I believe the beauty is you're using time (x and x-1) to preserve the change from sample to sample but remove the time influence that broadly affects all and distorts the subtle differences that may in fact not be related.

It only matters if the commonality is far more influential and unimportant than the unique but import parts.

Had he stopped there I might have concluded it was just a poor explanation. But luckily he continued:

Put another way, we've introduced a mutual dependency. By introducing a trend, we've made Y1 dependent on X, and Y2 dependent on X as well. In a time series, X is time. Correlating Y1 and Y2 will uncover their mutual dependence — but the correlation is really just the fact that they're both dependent on X. In many cases, as with Jennifer Lawrence’s popularity and the stock market index, what you’re really seeing is that they both increased over time in the period you’re looking at.

What, pray tell, is the X for which the stock market index and Jennifer Lawrence's popularity depend on? Oh, you say they're both dependent on the underlying trend... What?

That's right. Sort of like factoring out common variables/values in algebra I'd say. Time series is the "easy case" but I suspect identifying less obvious commons is the real art of it all.

Yeah. A better example would be to add 2 different slightly sloping lines to Y1 and Y2 random variables.

I don't think that will actually change anything. Two positively slowing lines (of any slope) have, by definition correlation = 1.0. The "random data" he used is just a small amount of noise on top of the line(s).

Reply on news.ycombinator.com