Ask HN: Tools to evaluate scientific claims? | Hacker News Reader

Ask HN: Tools to evaluate scientific claims?

6 pointspgt9 comments

What statistical knowledge and training do I need to evaluate the legitimacy of scientific papers, esp. in the medical field?

Whenever I read about any scientific claims, I ignore the press and go straight to the original paper cited (if there is one, often it is misquoted). I then read the abstract and the testing methodology. If I can spot any issues in the methodology I usually stop reading, e.g. small sample size, obvious confounding variable, blatant causation/correlation errors. But if all seems well, that still doesn't tell you if the study's claims match the test results or if the threshold parameters make sense.

Given a basic stats background, how can I obtain a deeper, intuitive understanding of things like p-values (which seem to be outdated anyway) and other sample sizes. Thanks, HN.

View on HN

Get a good book on statistics?

About p-values: they aren't exactly outdated, but they are the subject of a pretty fierce controversy. Many, if not most scientists still use them when doing statistical analyses because they are simple to apply and to understand and provide a quick metric for measuring the significance of a study's results.

Other scientists say that they are too simple and don't convey important information about the actual data (such as the spread). Apparently, there are more modern statistical procedures that do a better job than p-values do. (I'm not a statistician though, so don't ask me what these procedures are...) Also, p-values are all too often subjected to "p-hacking" - massaging the data until you get a statistically significant result (p <= 0.05). In fact, the very concept of significance is problematic. Originally, the p <= 0.05/0.01/0.005 significance limits were just approximate guidelines to help scientists interpret their data. Nowadays, they are often treated as definite boundaries of "truth". ("If my data gives a p-value of 0.049, the result is significant therefore my hypothesis must be true. If p=0.051, it is not significant, therefore my hypothesis must be wrong - or I must tweak my data until I get p=0.05.") This is obviously nonsense, yet a surprisingly common attitude (though not always as extreme as in my example).

As far as I personally am concerned, the real problem is not the actual p-values as such, but perhaps a lack of understanding of statistics by many scientists. (Coupled with the pressure exerted by journals that only want to publish "significant" results and so indirectly encourage p-hacking.)

Can you recommend a good stats book?

Sorry, I'm afraid I can't :-/

I got my own statistics knowledge from a lecture series, a book on R, an ecology textbook and various articles...

Check your local university library if you can or google around. I'm sure you'll find something.

Just curious, isn't this kind of basic legitimacy verification supposed to be done by the peer reviewing process? Are scientific journals really publishing papers so obviously illegitimate that laymen with a bit of statistical knowledge can spot obvious errors or problems?

Unfortunately, yes. Of course a lot of it depends on the journal, but there are enough out there that will publish just about anything. Even the more serious journals sometimes let "bad apples" slip through.

In the end, peer reviewers are only humans too, make mistakes, have a lot of other work to do, and don't even get paid for their review work. Recently we had an extensive discussion on HN about problems facing science as an institution (based on this article: http://www.vox.com/2016/7/14/12016710/science-challeges-rese...). Peer review was the fourth point on their list of problems.

Yup, that's https://news.ycombinator.com/item?id=12114551 --- "Big Problems Facing Science" (vox.com)

Things like a small sample size and confounding variable, doesn't in any way invalidate a paper. Especially in fields where collecting data is difficult (only a small number of people have a certain obscure disease for example, or it's practically impossible to isolate one variable from another) those are simply facts of life that you have to deal with. Sometimes you all you can do is say I only managed to find 4 cases of X, but they all had Y. Was not able to control for Z, still I wasn't expecting that and found it kind of interesting. As long as you aren't faking data (like you actually found 12 cases of X but threw out the ones that didn't fit your conclusion), then that is absolutely worth publishing.

It's always worth getting results out there even if they don't/can't live up to statistical certainty one might hope for. Every paper doesn't have to stand on its own and come to some undeniable ground breaking conclusion. Over time you'll hopefully collect enough published data to be able to do some more statistically powerful and useful meta-studies down the road.

Absolutely. Especially in the life sciences it can be very hard to get statistically solid data.

None of which means there isn't a lot of statistics crap floating around, though.

Hi @mbrock. Take, for example, the acetaminophen study[^1] currently on the front page claiming an association between prenatal use of the drug and behavioural and emotional problems in children. @fifteenforty works in this area and immediately pointed out[^2] the potential cause not mentioned in the abstract: why were these women taking painkillers in the first place and could that be the cause of the developmental problems?

Further down, in the Results & Conclusions, "association" implies "causal" when they mention "via an intrauterine mechanism?"

[^1]: https://news.ycombinator.com/item?id=12293675

[^2]: @fifteenforty comment, https://news.ycombinator.com/item?id=12295107

Link to original study: http://archpedi.jamanetwork.com/article.aspx?articleid=25432...

LA Times coverage: http://www.latimes.com/science/sciencenow/la-sci-sn-acetamin...

Reply on news.ycombinator.com