Hacker News Reader

Comment by _delirium | Hacker News Reader

The short version of the error: If in your data you find that A has no statistically significant effect, but B does have a statistically significant effect, this does not automatically show that B has, with statistical significance, more effect than A. To do that you have to do a statistical test on the difference in the effects.

I believe (?) this is normally done correctly in medical studies with placebos, where the typical analysis is to show that a drug has a statistically significant effect compared to the placebo baseline; it's not sufficient to show that the drug has an effect compared to a no-drug baseline, and, separately, that a placebo doesn't.

View on HN

You may be simplifying too much to be understood. In medical terms, if your drug is significantly more effective than doing nothing, but a placebo is not significantly more effective than doing nothing, your drug need not be significantly more effective than the placebo. (Yes, medicine typically avoids this particular error.)

(E.g. if doing nothing cures 10%, the placebo cures 11% and the drug cures 12%, only the difference drug/nothing may be significant.)

Or: if I compare three different distributions, it's possible for A&C to be distinct, but for A&B and B&C to overlap.

As the linked article says, the idea of statistical significance being a binary state is to blame here.

It's true. Many anti-depressant drugs have no significant effect when compared to placebo.

http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fj...

I don't believe the controversy over that is whether the statistics were done properly, but over: 1) replicability, in particular whether significance is still found when doing large-scale meta-analyses; and 2) whether blindness of the studies is compromised by "unblinding" effects where a drug's side-effects can tip off the doctor or patient about whether they got a placebo or not. But it's been a while since I waded into that flamewar of a debate...

No, according to this one controversial[0] study, anti-depressant drugs have no statistical significance compared to placebo except in cases of severe depression.

[0]: http://www.plosmedicine.org/article/comments/info%3Adoi%2F10...

A good paper on this error is from Andrew Gelman www.stat.columbia.edu/~gelman/research/unpublished/signif3.pdf

I'm curious about the example in the article. If the effects were measured at 30% and 15%, which are the change in some measured norm, then isn't the difference in differences actually 100% not 15%?

Often you still use units of % change relative to the initial quantity, though I suppose you could define it either way. For example, if we're comparing a country with an inflation rate of 4% and one with an inflation of 8%, we might say that the difference between the rates is 4%, in that |rate1-rate2| = 0.04, as opposed to measuring one in terms of percent of the other.

I'm asking because the author talks about the "difference in differences" and as early as highschool physics they drill into your head that the difference between two things is represented by the percentage difference not it's absolute difference. Saying 15% is negligible when it is actually a 100% change in the effect seems incorrect somehow.

Reply on news.ycombinator.com