Hacker News Reader

Comment by rgrieselhuber | Hacker News Reader

Peer-reviewed research isn’t a great standard, imo, for many reasons. There are sources available, ultimately it seems to turn into an ideological battle. FAN [1] probably has the most links out to different sets of research. Really it comes down to making your own decision, which is why I don’t think it should be automatically added to the water supply. Fortunately there are options to remove it.

[1] https://fluoridealert.org/mobile-home/

View on HN

I would love to do an analysis of all comments online to see the correlation between negative sentiment - peer review and "make your own decision."

> Peer-reviewed research isn’t a great standard, imo, for many reasons.

What is your recommended alternative?

Depends on the topic. If you're trying to prove (or, more accurately, fail to falsify) a certain set of results or a discovery, open data plus reproducibility should be the gold standard.

If you're trying to make decisions about personal health, it pays to take a cautious approach (IMO) and let people decide for themselves. It is not inconceivable that commercial entities finance "peer-reviewed research" in order to support public health policies.

I'm confused. I thought that reproducing a study was pretty much doing a peer review?

That is the way it was sold but peer review fraud is a major problem.

Do you have a peer reviewed source on that? Major usually implies 50%+ . Are you saying that 50%+ of the peer reviewed scientific papers out there bogus and fraudulent?

I don’t think 50% or greater is a good standard for defining what we would consider to be a major problem with scientific fraud. I’m sure you’re well aware that “major” can refer to relative impact depending on the context of the issue being discussed. In the case of scientific publishing, I would argue that even 5% fraud rate would be grounds for labeling this as a major problem.

That said, this issue has received a lot of attention in recent years. Some articles try to put a positive spin on it but the stark truth of where we are today is that there are no standards enforced requiring data publishing (“available on request” doesn’t cut it) and reproducibility. This, combined with the amount of money in this industry and the impact that it has on public policy is an acceptable situation.

Some good coverage:

https://www.experimental-history.com/p/the-rise-and-fall-of-...

(Lots of links to follow on this above post)

https://retractionwatch.com/

https://www.science.org/content/article/what-massive-databas...

(This one puts a positive spin on the editorial process but the fact remains that there is a massive flood of fraudulent papers being published. Going back to standards of open data plus verified reproducibility would go a long way to mitigate this problem.

Peer review means someone knowledgeable about the subject read the paper and didn't spot anything wrong. They almost never attempt to reproduce anything.

A small but important reframing: I agree that (open data) + (reproducibility) + (review) is a better bar than any one in isolation.

(This applies to empirical sciences only. In contrast, for a mathematical paper, rigorous reviews of the formal logic are the gold-standard.)

I choose the phrase 'review' intentionally; it doesn't need to follow one of the typical [1] 'peer review' processes from a journal. There are likely alternative methods that are (at least as) or (more) effective. Some fields have seen amazing contributions from 'lay people' all across the world.

> If you're trying to make decisions about personal health, it pays to take a cautious approach (IMO) and let people decide for themselves.

For a person equipped with the time, skill, rationality, and motivation, I agree. But it is an empirical question to assess how many people have these skills. I also want to push back against what might be an individualistic bias [2] in the statement above. Even if one is an individualist, it isn't optimally efficient nor practical for an individual to 'take on' all this responsibility. It is rational and efficient to place some degree of trust in others.

[1] These processes vary and should be treated with a fair bit of skepticism. I want to to dig in more; e.g. to what degree have these processes been designed to account for human realities so as to maximize quality?

[2]: See also https://so2020.isosonline.org/conference/the-individualist-b... "Individualism claims that social entities (e.g. groups, institutions) are essentially nothing but suitably arranged aggregates of individuals. This is usually tried to be demonstrated through developing accounts of reduction or supervenience. Holism, on the other hand, claims that social entities are sui generis as something over and above individuals."

Replicated research. Peer review is when journals ask other researchers to look at a paper and determine whether it's suitable to be published. Attempting to replicate the findings is typically not part of the peer review process, peer review is a sniff test.

Reply on news.ycombinator.com