Data detectives spotted fake numbers in a widely cited paper

The datacolada post makes the very reasonable request that all data should be released, and scientists should make that a standard thing to do by doing it themselves and requesting others do it.

It feels like this could be applied retroactively too.

In this case the 2012 authors still had the data that they released in 2020 which is how the analysis got done that showed evidence of fraud. Might be worth just asking a whole bunch of people to release data they previously hadn't and collectively putting some time and effort into that.

One amusing point is that much of Gelman's post has an error itself (that someone pointed out in the comments a couple weeks ago): the NPR interview was in 2017, so "Ariely, as a coauthor of this article, had to have known for at least half a year before the NPR story that this finding didn’t replicate." is incorrect. Maybe Gelman should retract that part :).

Annoyingly, the NPR transcript at [1] only has a small note "(SOUNDBITE OF ARCHIVED NPR BROADCAST)" at the top with no indication of when (the audio doesn't seem to have a date either). The podcast show notes are apparently the only recordation of the date. [2]

[1] https://www.npr.org/transcripts/805808486

[2] https://pbs.twimg.com/media/E9UQv8LWEAkpoR0?format=jpg&name=...

And how it funnily relates to the Gell-Mann amnesia effect. Although named after physicis Murray Gell-Mann:

https://www.epsilontheory.com/gell-mann-amnesia/

There's a lot of value being abused in the term 'science'. Science is a highly valued concept but it's the result of following the scientific method, not the output of anyone with a postgrad.

Fraud like this is next-order dispicable because of the knock-on effects it has on knowledge and policy. I don't know that stating that is going to have an effect on people who engage in fraud as a matter of course or as passive parties to it, but here we are. The incentives here are what makes most research an unsuitable substitute or appeal to authority for pubic policy. It's a proxy for a petty elitism, and it means to "trust science," just becomes scientistic nihilism. After all, these days you don't need popular consent when you have scientists.

Reading reports like this, I can see why there has been so much criticism in recent years of the concept of the banality of evil, and why so many researchers affect concern for the environment, because more than anyone, they seem to understand what it means to be responsible for poisoning an ecosystem and they need to get out in front of those narratives. Sure, it's just a bit of fraud in a journal, just like it's just a bit of PCB or mercury in a lake, and only a minority of the population who will be impacted, but someone has to call it out as nihilism, or we're a party to it as well.

As much as I dislike blockchain ideas, it makes me think a blockchain DAG of metadata about the integrity of published papers for citations, reproducability, and evidence of certain types of fraud would rebalance the incentives a bit.

I think it's important to bear in mind that even with the system working perfectly well, and perfect ethics, we should expect to see a lot of papers published with false results.

Lets say there are 200 propositions we want to test and are candidates for publication, that 20 of them are true and that our error rate is 5%. That means when we test the 20 that are true 19 of them will be accurately shown to be correct and 1 will be erroneously found false.

However when the other 180 propositions are tested 5% of them will be erroneously found to be true, that's 9 propositions. This means we will end up with 28 'successful' studies that make it into prestigious journals, about 1/3 of which are false positives.

And as I said, that's if the system works perfectly with no fraud whatsoever. Throw in some human error and it's not surprising if a fair few studies start to look pretty dodgy. Add in some genuine fraud too and you've got a full-on replication crisis with all the trimmings.

I think it is important not to extrapolate from a replication crisis in one field (e.g. psychology) to all the other sciences, because the picture painted by this (to my knowledge) doesn't accurately describe the underlying practises.

> A lot of science today is basically parallel construction. You start with a sexy story that you know will get you a lot of press, like "promising you will be honest actually makes you behave in an honest way" and then you just make that paper happen, however you can.

Gigantic accusation, zero evidence.

If this was a paper, rather than an HN comment, I'd say there was every chance that it would be self-illustrating.

More than that, they should have their degree revoked too.

The spam in these journals puts Buzzfeed to shame.

If they did a good job hiding it then you wouldn't know about it. For every case like this where they made a obvious mistake, there are cases where they didn't, and nobody noticed. But you only tend to hear about the ones with obvious mistakes.

> Do scientists actually read what they cite?

I depends on ones probity, but yes the phenomenon is widespread. There are numerous reasons to cite a paper and just read it superficially: padding the references list, pleasing a reviewer by adding a paper he recommended, citing friends, etc. In my own lab they frequently cite a certain theory which if you actually read about it has nothing to do with what they are doing.

> Do scientists actually read what they cite?

No. What are you kidding me. There's like typically over 50 papers referenced in a typical publication, no way I have the time to read all of them carefully

> Do scientists actually read what they cite?

Yes, but few actually scrutinize the methodology of the studies. Statistics is really hard. It's easier to assume peer reviewers would have rejected the paper if it was bad.

Did YOU even check what you cite?

The AUTHORS of the original paper got a dataset from a company. They didn't assume the fraud from the start and published the paper based on it.

Later, when they tried to analyze the issue more in-depth, they couldn't replicate the results. THE ORIGINAL AUTHORS PUBLISHED a paper about a failure to replicate. It was just then that someone looked at the original data and found that it was faked.

The problem is often that reading it is insufficient. Unless you try to replicate, it may well look plausible, and often missing data or details creates barriers that makes you need to want to replicate really badly to put in the effort.

The pay is amazing.

Become a researcher and do peer review.