Hacker News Reader

Comment by s17n | Hacker News Reader

> The researchers observed that the accuracy of news sources the Twitter users retweeted promptly declined by roughly 1 percent in the next 24 hours after being corrected. Similarly, evaluating over 7,000 retweets with links to political content made by the Twitter accounts in the same 24 hours, the scholars found an upturn by over 1 percent in the partisan lean of content, and an increase of about 3 percent in the “toxicity” of the retweets, based on an analysis of the language being used.

Rule of thumb, you can disregard any social science paper where the effect size is single digits.

View on HN

As imperfect measures as they are, statistical significance measures like the p-value are more epistemologically sound than any arbitrary rule of thumb-threshold on the size of the effect.

> statistical significance measures like the p-value are more epistemologically sound than any arbitrary rule of thumb-threshold on the size of the effect.

Keep in mind that the GP isn't saying the effect doesn't exist if it's in the single digits, but that it is inconclusive and/or insignificant. Insignificant in the human sense, not the statistical sense.

A 1% increase in this behavior? Irrelevant to almost everyone.

This, of course, is not even getting into the issue of the reproducibility crisis, much of which did rely on p-values. While I personally am happy to do p-tests, the skepticism of small effects is well founded. Were someone else to try to reproduce the effects and fail, the standard defense is that the results are sensitive to the methodology used. It's much easier to invoke that defense if your effect is 1% vs 20%.

P-values are based on a normally-distributed sampling distribution which presumes the samples are randomly-chosen. It's hard to see how random sampling can apply very well here.

Suppose we did a comparison between the p-values of biased researchers desperate to publish, versus a conservative heuristic that doesn't believe small effects. Particularly for social science experiments like this, which would you bet on being able to assess repeatability better?

Statistical significance is a good measure of how sure we are there is a correlation, or in this case with an RCT, causation. But any layperson can look at a 3% effect and conclude, yeah, that's probably not that big of a deal. Or not, depending upon your preferences! No judgement. It's not something that requires a degree to determine, just an assessment of one's own values and the effect size in that context.

P-values don't work (https://en.wikipedia.org/wiki/Replication_crisis).

My rule of thumb is to disregard snarky know-it-all comments by random people on the Internet.

Oddly enough, this comment seems to reinforce the findings we just read about.

But now you've created three Borromean competing claims.

I hadn't heard of Borromean Knots before you made this comment and got a great read out of it. Thanks!

For others following up on this thread, Borromean Knots seem to be an extension of psychoanalytic theory that tries to describe social interactions through an object oriented model that highlights the differences between symbolic, real, and imaginary claims:

1. https://en.wikipedia.org/wiki/Borromean_clinic

2. https://larvalsubjects.wordpress.com/2009/12/08/borromean-kn...

russell's paradox: the hn comment

Your comment does not provide any useful information at all. A merely personal attack.

My guessing is that their measurement of partisan lean of content and toxicity of retweets may have measurement error way larger than 1% or 3%, not to mention other noises in the stastistical modeling.

Why would real effects either be 0 or >= 10%? What would cause such a discontinuity across every single effect?

It isn't that simple. We want to know what percentage of tweets have property X given event Y. We can estimate that using a sample and error bars.

Note that the quotes specifically avoid this inference because the error bars would be silly. 500 million total tweets a day can't be talked about based on 7,000 hand categorized tweets.

But non staticians don't have any reference point for how powerful a predictor it is so a short hand is < 10% probably means in the error bars so less believable.

Taking your question non-rhetorically: because the error bars are wider than 10%[eg, 0, which finds a floor of 4% for anything based on polls], so anything less than that is consistent with p-hacking/random noise[eg see 1], and this is pervasive enough[2] that social science papers should be assumed garbage until proven otherwise.

0: http://slatestarcodex.com/2013/04/12/noisy-poll-results-and-...

1: https://www.explainxkcd.com/wiki/index.php/882:_Significant

2: https://en.wikipedia.org/wiki/Replication_crisis

It's not that no effects occur in this range, but if they do, they are not discernable from random fluctuations, noise and other statistical (non-)effects.

It can be real but unimportant. An extra drink every week might increase my liver cancer risk by 4%, but that's negligible if it's the difference between a 1 in 1 million chance and a 1 in 960,000 chance.

Reply on news.ycombinator.com