The study is terrible. All the reported effects include 0 well within the 95% confidence interval (e.g. ranges like -2.3 to +3.2), yet mysteriously the P values end up just the right side of "significant". This reeks of confirmation bias, or worse.
The study is terrible. All the reported effects include 0 well within the 95% confidence interval (e.g. ranges like -2.3 to +3.2), yet mysteriously the P values end up just the right side of "significant". This reeks of confirmation bias, or worse.
Other than that, this does look like a pretty poor study. They do that classic (but misguided) thing where they say that their study was conducted on a small sample and so the fact that they found anything at all is super-duper-extra telling. In reality, it just makes it more likely that any effect that is reported will be exaggerated – as was brilliantly pointed out by Andrew Gelman and John Carlin in http://www.stat.columbia.edu/~gelman/research/published/retr....
This gets into the flaws with hypothesis testing (and p-values) as a tool altogether, though. And while I would love to see the natural sciences eschew these rather rudimentary heuristics in favor of the more sophisticated modeling tools that we already have, I'm not holding my breath on that happening anytime soon.
My only concern is with cherry picking the chemicals. It wasn't clear if they were only reporting specific chemicals that had significant relationships, or if all the chemicals that would be expected to have a significant relationship did.