Avoiding a Common Mistake with Time Series | Hacker News Reader

Avoiding a Common Mistake with Time Series

svds.com

161 pointsdavidkellis41 comments

I'm not sure. The author says that adding a common component to two random time series doesn't make them correlated. But that's not true, by construction, at least using any of the simple correlation tests. It's a complicated subject explained in a confusing way.

Yeah. A better example would be to add 2 different slightly sloping lines to Y1 and Y2 random variables.

I don't think that will actually change anything. Two positively slowing lines (of any slope) have, by definition correlation = 1.0. The "random data" he used is just a small amount of noise on top of the line(s).

I don't get it either. The author is adding a common dependency to two independent series makes them correlated, which to me seems trivially true. He then goes on to say that this form of correlation is uninteresting, because we're only interested in whether variations are correlated, which doesn't seem so trivially true: after all, a linear trend is a first-order variation, and if two things both increase at the same rate, then their relationship is worth at least a second look.

It looks like what the author is really trying to say is that we should pass the data through a high-pass filter, eliminating any 'expected' trends such as inflation, and instead observing if the noise of the two datasets is correlated. This is an observation that has some value, but is certainly not trivial to pick a threshold for the high-pass (certainly it's not always just a linear trend), and the mutually dependent variable can have as much noise (if not more) as the two measured data sources, so you still might get "false" correlation.

It seems to me that in the end it all comes down to your actual hypothesis whether and what is actually interesting …

Increasingly I'm finding that any description of causation/correlation which (a) isn't written by a more or less formally trained statistician of some variety or (b) invoking Pearl-like causal diagrams directly has a coin-flip's chance of being actively dangerous to read.

Causation is super tricky business. You probably have an okay model of it intuitively once you've done a little stats work, but to explain it to someone else or tackle it in more exotic or hypothetical domains is an exercise is insanity if you're not armed with the very best tools.

Causation is dead simple---if you can directly manipulate the putative causal variable. Otherwise, you're left with the old saw about correlation, and while you might be able to tell a story about why there should be causation, the best you can do in that case is eliminate other potential confounds.

It's simple in models where you assume you understand the causal structure. Then whenever things get murky people tend to try to operate without clarifying the causal structure and end up in a bind.

Of course what you say is true, but executing it in practice and having a sufficiently rich language to identify confounders well is what's usually lacking.

Adding a common deterministic trend to two random time series does not make them correlated. Intuitively, you still cannot predict the deviation from the trend of one timeseries using the deviation from the trend of the other timeseries.

His statement (after adding the constant trend) is misleading:

"Now let’s repeat the same tests on these new series. We get surprising results: the correlation coefficient is 0.96 — a very strong unmistakable correlation"

What he's calculated is the correlation between a set of points from y_1 and y_2 - and that will be large, of course - their (deterministically increasing) means have correlation 1.

The quantity that qualifies as correlation for predictability purposes is actually the correlation between the deviations from mean.

This is all fairly clear if you use the actual formula (Pearson correlation):

E[(y_1 - mu * t)(y_2 - mu * t)] / (sigma_y1 * sigma_y2)

You missed the point. This has to do with "time series data" which means time is the same among all the data sets. Because you know this you should remove it to test for similarity. You could apply this to any automatic variable in your domain but time & change is very obviously a typical common variable for many data sets.

Please help me understand. Assume I have 100 years of atmospheric CO2 levels and 100 years of temperature data over the same span. These are both "time series data". What are you suggesting that I remove before testing for similarity? Does this help me to determine the degree to which one of the two variables is causing the other?

I think time - then you're left with data. If when removing this they still have the same shape they do in fact correlate. I'd be curious to see the results of that experiment to be sure.

I believe the beauty is you're using time (x and x-1) to preserve the change from sample to sample but remove the time influence that broadly affects all and distorts the subtle differences that may in fact not be related.

It only matters if the commonality is far more influential and unimportant than the unique but import parts.

You should first break out the relationship between CO2 and time, and the relationship between temperature data and time. You effectively normalize both for time.

So, in effect, you model the correlation between CO2/time vs. temperature/time. The interesting correlation would be if every time CO2/time deviates you can find a correlation with a temperature/time deviation.

Had he stopped there I might have concluded it was just a poor explanation. But luckily he continued:

Put another way, we've introduced a mutual dependency. By introducing a trend, we've made Y1 dependent on X, and Y2 dependent on X as well. In a time series, X is time. Correlating Y1 and Y2 will uncover their mutual dependence — but the correlation is really just the fact that they're both dependent on X. In many cases, as with Jennifer Lawrence’s popularity and the stock market index, what you’re really seeing is that they both increased over time in the period you’re looking at.

What, pray tell, is the X for which the stock market index and Jennifer Lawrence's popularity depend on? Oh, you say they're both dependent on the underlying trend... What?

That's right. Sort of like factoring out common variables/values in algebra I'd say. Time series is the "easy case" but I suspect identifying less obvious commons is the real art of it all.

What the author is trying to explain are the concepts of cointegration and stationarity. A useful introduction here: http://www.uta.edu/faculty/crowder/papers/drunk%20and%20dog....

Great article, thanks very much for posting it. Nobody else seems to have captured the relevant terminology: "cointegration" and "stationarity". Those perfectly captured it for me.

Never use first differencing, that's crazy. It magnifies measurement error. Pass the signal through a high pass filter to remove trend

Or simply subtract a first-order fit. (What the matlab/numpy `detrend()` function does)

Edit: I'll note that this is the same thing as subtracting a very long-windowed low-pass filter, i.e., performing high-pass filtering.

Something I've repeatedly found useful is that when debugging and you have a conjecture, not only look for evidence that a correlation/causation is present; but also look for evidence that it isn't.

Doing a very quick A/B test helps too.

"Something I've repeatedly found useful is that when debugging and you have a conjecture, not only look for evidence that a correlation/causation is present; but also look for evidence that it isn't."

In my opinion, the absolute, utter core essence of science can be expressed simply as "Always be trying to prove yourself wrong." The human brain is extremely biased in the other direction, and it's darned good at proving itself correct. It can prove itself correct in absurdly powerful ways. Always be fighting it, always be looking for ways to prove yourself wrong. If you do it seriously, things like "the scientific method" will naturally fall out of your serious attempts and need not be surrounded by near-worship, whereas no amount of worshipfully-following a checklist of the "scientific method" without the true effort to prove yourself wrong will produce truth; the human brain is far more powerful than the "scientific method" or any such static methodology.

I have deliberately left the word 'theory' out of this post. This scientific mindset beyond that into engineering and any number of day-to-day activities.

Try to prove yourself wrong about believing it is better to be wrong all the time.

Insanity!

I agree with you for the most part. But there's a line. When it starts to destroy your personal identity and sense of self, you've crossed that line. Sometimes it's just better to be right and trust yourself.

Or not. That's zen, I think. Trying to have so much information about everything that you wind up overloading yourself with it and can't make heads or tails of it, because in every simple truth exists an infinite proof. Analysis paralysis!

Despite superficially resembling some self-referential paradoxes, if the heuristic can stand up to itself, it's not a paradox. You try to prove it wrong, and you fail, so you keep it, which, lo, is exactly what I've done. (Or perhaps more precisely, while it is not a perfect fit and I could quibble with details myself, it is the best one-sentence summary of true science in my opinion, and I'm familiar with most of the usual ones, and aware that I've only seen this variant in one or two other places even so.) No paradox, no contradiction, no insanity.

Alas, if you're looking for an excuse to tie either yourself or me up in some sophomoric philosophical conundrum, this doesn't do it. But there's no lack of such things if you look, so don't be disappointed. Keep on trucking. May I suggest Godel, Escher, Bach: An Eternal Golden Braid if you would like the industrial strength version rope, err, braid to tie yourself in?

I understand where you are coming from, but I'm not trying to engage you in a 'sophomoric' debate. I came to the existential philosophical argument from mathematics and the computer sciences, not the other way around. I started my interest in computing science through automating proof (mathematical and computational). I doubt you care, but it's more about looking at the big picture of software development versus the little picture. You can't build anything if you keep testing the first thing for every bug, including everything it was built with, including all the mathematics used to reason about it.

But I'm sure there are two ways to respond to my comment: pretend you know what you are talking about, or admit you could be wrong and care to engage me in a real conversation. I've been down the same road hundreds of times on the internet, and it is hard to find people to talk to about things and meet in some kind of scientific, logical middle. But if that's not your thing, oesn't bother me if all you came here for was to prove yourself right about the noble ethics of your science. At least I'm no longer trapping my mind in a paradox of it's own creation. (Or am I? I never really know).

Logic is very different from science, that's all I have to say. Rigor is for precision, not discovery.

"I understand where you are coming from, but I'm not trying to engage you in a 'sophomoric' debate."

Considering the entire rest of your message consists of you essentially trying to lord over me whatever superiority you seem to think you have, and not missing an opportunity to slide a snide comment in, which I'd also observe is consistent with your original post, I've come to the conclusion that the rest of your post belies this claim. You're being abrasive and abusive while trying to pretend to be so much more open minded.

No sale. I recognize that tactic and refuse to engage with it. I'm sticking with my original assessment; you're being too sophomoric to engage with, and I reject your attempts to psychology me into engaging with you in the presupposed frame of your superiority.

Apologies, I would not interpret receiving my post back to me in the same way (I do not think, I usually read my posts back to myself and pretend I am talking to myself to try to understand how another person may take it, which I realize is fairly biased given my potentially unique perspective on reality). Human socialization is very difficult for me.

I don't like psychology.

What worked for me is findig someone who is not involved in your analysis but has an analytical background and understanding what you are working on, and asking him to prove that you are wrong. My experience is that the most analyst are happy to have an opportunity to prove that you are wrong and they are right.

> Doing a very quick A/B test helps too.

But then there are all the ways people screw up A/B tests...

If you search long enough, especially in times of big data, you will always find some statistically significant correlation. That´s why it is getting even more important to validate your model/hypothesis. In our case over time. A correlation passing an adequate validation is high probably a causation. A not validated correlation is only an opinion.

Isn't the author exaggerating in the other direction? There is obviously correlation between the two time series. Sure, who's saying there is causation (as mentioned in the article there can be a third random variable that the first two depended on)? But also, who's to say there's no causation? Is it ok to always remove the correlated part of the two time series? What if that's the interesting part and the explanation you're looking for?

Even if you're looking for causation, detrending is usually necessary to obtain consistent estimators (in statistical terms). For example, Granger causality works with two stationary timeseries:

https://en.wikipedia.org/wiki/Granger_causality

This is called spurious correlation. It's well known in financial / economic time-series analysis. The lesson is that you never measure the correlation between the PRICE LEVELS of products, instead you measure the correlation between the daily/weekly/etc CHANGE IN PRICE LEVELS.

A famous example of this:

The tale of David Leinweber, which is related in the excellent new book "Quantitative Value," illustrates this point about "stupid data miner tricks." Leinweber sifted through a United Nations CD covering the economic data of 140 countries. He found that butter production in Bangladesh explained 75 percent of the variation of the S&P 500 Index. Not satisfied, he found that if he added a broader category of global dairy products, the correlation would rise to 95 percent. Then he added a third variable, the population of sheep, and found that he had now explained 99 percent of the variation in the S&P 500 for the period 1983-'99.

(http://www.cbsnews.com/news/what-butter-production-means-for...)

I ordered the book, Quantitative Forecasting Methods by Farnum and Stanton (PWS-KENT, 1989). it was only £2.81, sounds like money well spent.

Unless the book is junk, in which case you lost potential money by reading it, perhaps well in excess of the 3 bucks :-)

I got the last "cheap" one, at $3.94.

hmm, my purchase got cancelled and now they are all 10x the price.

I reckon they had a look at the competition before putting it in the envelope

Does this mean that if I apply this algorithm and that 2 or more time series data sets are still similar that they are in fact correlated? I find this test fascinating.

Wow, and the graphs on http://www.tylervigen.com/ are just incredible.

also: statistical tests on correlation coefficients don't test whether the correlation is "significant" or not --- they only test whether the correlation is reliably different than 0.00

So a small correlation (e.g. r=0.10) can still be "statistically significant" at p<0.001 but all this means is that r is reliably different than 0.00 --- it doesn't mean r is big

perform a dickey fuller if you are unsure if a time series is nonstationary, perhaps.

where are the scrollbars??

Reply on news.ycombinator.com