Hacker News Reader

Comment by peterbonney | Hacker News Reader

I think I say this every time I see a headline like this on HN, but I'll say it again: The past decade of social science research publication has proven that provocative-sounding results should be considered fraudulent until the underlying data has been published, and then (assuming the data passes muster) should be regarded with skepticism until independently replicated with a fresh data set.

View on HN

Sad but true, and extends to economics, psychology, lots of neurological and medical research as well.

Everyone needs pre-registration, and everyone needs open data.

One of the most eye-opening studies I saw was that social science papers with the most citations, had the highest likelihood of not being reproducible. Meaning that Social Sciences are actually pushing out misinformation more than not.

This isn't an issue of social science. This is an issue that has been found in a diverse number of fields, basically everywhere where someone has looked: https://www.frontiersin.org/articles/10.3389/fnhum.2018.0003...

its a little disingenuous to make it seem like all fields are equal. when that is clearly not the case

Is this somehow a flow on effect of more commonly cited papers having more reproduction attempts?

No, a moderately successful paper has 1-10 k citations, and usually 0-1 reproductions. Extremely successful papers have 10-40 k citations and 1-3 reproductions. They largely dont matter at all.

Its mostly because extraordinary claims receive more attention, and thus is found by the group for whom the result supported their prior beliefs. They will use it whenever they need to support such. This is also why such studies tend to keep getting cited as evidence long after a long line of failures to replicate have been published. As the result is non-replicable, there also wont be a newer, improved version of the publication, meaning all cites go to the original.

The theory is that there is a pressure in certain academic fields to reinforce existing popular narratives with new publications, motivating researchers to confirm them, and thus for other researchers to cite them. ...but since such efforts attract low-quality researchers and are based on an implicit initial bias, due to its self-referential nature, that the results are not reproducible by others in the future since teams that try to reproduce results are usually more neutral on the topics.

That doesn't technically mean misinformation although but it does imply major room for improvement in methodology. If I roll a d20 and get a 7 and you roll it and get 13 that doesn't make the prior study inaccurate. The sizes are too low or not properly defined.

But the combination of publish-or-perish and grant seeking pushing sensationalism isn't good for honesty let alone accuracy or precision.

"A good scientist, in other words, does not merely ignore conventional wisdom, but makes a special effort to break it. Scientists go looking for trouble. This should be the m.o. of any scholar, but scientists seem much more willing to look under rocks." -- Paul Graham.

> Meaning that...

No, that does not follow at all.

But it seems very appropriate that you quote one statistic and then make a claim about a completely different one. Good exercise/example for the reader.

If you're interested, the paper says the underlying data is freely available on request.

They all do though. Including the outright frauds.

Thanks I always forget

they have to sell

Just like epidemiology, right?

Does epidemiology have a well-publicized replication crisis with numerous examples of outright fraud from high-profile researchers? If so then yes. If not then why don’t you just say what you’re trying to say rather than imply it obliquely in the form of a question?

They need to imply it obliquely because their arguments tend to fall apart under scrutiny.

My “argument,” which is really more of an observation since we’re not debating anything, is clear. Epidemiology, as a social science, should not escape the same skepticism we approach the other social sciences with.

Perhaps my comment is slightly off-topic, but I just like to leave a friendly reminder since I don’t find epidemiological conclusions under enough scrutiny in public fora these days.

Too easy

https://www.science.org/content/article/two-elite-medical-jo...

https://covid19up.org/neil-ferguson-fear-driven-predictions/

The best you can come up with for controversies in epidemiology are from non-epidemiologists?

Reply on news.ycombinator.com