Hacker News Reader

Hacker News

Comment by kaffeemitsahne | Hacker News Reader

Full thread

kaffeemitsahne

Potentially OT. Why do people write

>"While suicide rates for females increased after the show’s release, the increase was not statistically significant."

instead of

>"Suicide rates for females did not increase significantly after the show's release."

I have seen this quite often and it always seems a bit misleading to me.

Edit: I should have written the 2nd version as "No statistically significant increase in [...] was found". My point is not about the difference between "significance" and "statistical significance" but about mentioning an effect, while also saying the effect is not statistically significant. If it's not statistically significant, why mention it?

View on HN

IMO, your proposed second option leaves ambiguity as to whether "significant" means "statistically significant" or "materially significant." The original option is clear on that point.

However, I would argue that the original text is also misleading, because if the result wasn't statistically significant, there's no evidence that rates for females increased. If anything, the first option could perhaps say "While suicide rates for females IN THE STUDY increased after the show's release, the increase was not statistically significant, so we have no evidence that it's a real effect."

> if the result wasn't statistically significant, there's no evidence that rates for females increased.

That's not quite right. A conclusion of no statistical significance is not a conclusion that there is no evidence to support the hypothesis, but that there is insufficient evidence.

The study failed to disprove the null hypothesis, but "failure to disprove" does not prove it to be true either. A more powerful study (with a larger sample size, or one that includes this incident in a meta-study of similar media events) might be able to use this data to support a positive conclusion.

They observed an increase, but the statistical test they used couldn't distinguish it from noise with sufficient confidence to call it "statistically significant."

Statistical significance and a significant increase aren't the same. A statistically significant increase might be totally insignificant, and for a small enough study, a subjectively "significant" increase might be statistically insignificant.

I think it's to combat the colloquial definition of "significant" with the specific statistical definition.

All polls and samplings have an amount of error (e.g. Biden leading in the polls at 30%, 3% margin of error). If Biden polls at 32% next week, there was an increase. Colloquially some would say that's a significant increase (after all, his pie grew by 6% compared to the prior week!), but it's very likely to do with chance. Unless you survey the entire population, you don't know the true proportion in which something occurs.

A change from 30% to 35% is quite unlikely to be random chance.

This is a recurring issue because "significant" means very different things in different contexts. The first sentence reads as "rates increased, but in a way that is not distinguishable from noise", while the second reads as "rates increased, but not to an extent that matters".

The second reads as though there wasn’t even an insignificant increase, which is very different.

The former is technically correct and much more descriptive.

The latter is potentially-misleading. One might think that the rate did not increase at all, when in fact it did, or that it did not increase "a lot" (the colloquial understanding of significant). "Statistically-significant" is a keyword with technical meaning.

The former could mean that the rate increased by 2±2 units. A more-precise study might show that the rate did increase, or it might not. All we know from the former is that the rate increased, but that the rate was not statistically-significant.

Significance and statistical significance are two different things, and it's good for them to be distinguished.

Why not just say: Suicide rates for females did not change in a statistically significant way.

These mean different things.

You can have a 1% increase (which is not a "significant" increase) but with a large enough sample size you can be quite confident in it (statistically significant).

The former version is more specific and therefore accurate.

Because those statements are not equivalent.

Reply on news.ycombinator.com