Hacker News Reader

You need to consider the historical context too. The referenced authors has been found performing statistical fraud before. P(Mistake) is probably lower.

No there is no reason to be skeptical of the author here because he based his evidence on statistics.

As the author describes, the p-values listed are effectively impossible to get from real data. Because of the magic of mathematics, we know exactly how p-values are meant to behave when you calculate them repeatedly - uniform random between 0 and 1. There are a battery of tests you can use to interrogate this very well understood distribution. Those tests, as the author states, prove without any doubt that the p-values in table 1 are effectively impossible. You can't make it to those p-values by mistake.

P values are, effectively, the probability of a result occurring because of a process. These p values are overwhelming evidence that the result did not occur because of the process described in the study.

That does not mean fraud (although, I stress, in this case I can see why it is probably fraud). For example, maybe the study is done in some sort of hospital with some sort of patient scheduling system and the front desk is playing a funny joke on the doctors in this study. It is quite common for experimenters to not be as much in control of their environment as a paper suggests - that would be an honest reason for why the data is weird.

In a more plausible example - I don't know how "matching" works, but maybe the patients were matched, the report writer didn't know that and the sentence slipped through proofreading. Unlikely, but certainly possible. That mistake would happen in the wild from time to time.

You don't need much imagination to find examples other than fraud. Although it does take a little imagination, which is concerning. But, frankly, there should almost always be an attempt at dialog with someone before accusing them publicly of being a fraud. Unless they're going to shoot you or something extreme.

To be frank the instances you listed sound very improbable, but could withstand the burden of truth needed to prove guilty

The author of this work addressed the notion that patients were matched and the methods were described incorrectly, even in that situation the distribution of p values is so highly unlikely that it’s most likely fraud.

Sloppy research methods leading to bogus results are can be construed as fraud, particularly if you don’t know the motivation for the lack of care.

A diligent researcher might take these results which were “incredible” and attempt to replicate them before publishing.

Sheldrick has just caused massive reputational damage, got thousands of strangers reading & discussing how Marik's work looks like fraud (despite few of us actually being qualified to assess its merits) & is posing a serious professional attack to Marik's wellbeing. It is unlikely but in the realm of possibility that someone would commit suicide or get some sort of trauma from this sort of attention.

He's doing that based on an interpretation of some study, which I suspect is approximately a 20 page document.

Now on the one hand, he is probably right. On the other hand, this is a very brave approach - the damage and the care taken aren't in good proportion here. This sort of thing is easy to get wrong. He should have talked to the paper's authors first.

> He should have talked to the paper's authors first.

Sheldrick did exactly that. It is the first line of the article.

"Below is an email I have sent to Sentara Norfolk General Hospital, the editor of CHEST Journal, and prof Paul E Marik"

People publishing a study that is so blatantly against everything that is statistically likely should be called out.

The author published an email he also - before publication - sent to the people involved.

Science happens in public for a reason. The reason is that others should be able to check the results being published. Authors should therefore be very aware of the fact, that their results might come under scrutiny.

If one were to publish claims that look so unlikely based on valid scientific methods they should at least address the unlikeliness of the results and provide a rationale why this is still valid, even if it looks unlikely.

That didn't happen - and so they get called out. The extreme unlikeliness is what validates the claim of this being fraudulent data. Now it is on the original study`s authors to react to that public answer to their publication.

I am guided by the "I have not requested access to the raw data or contacted the authors for explanation as the case is audacious no other explanation is possible." comment in the letter.

I am guided by the irrefutable mathematics which has more certainty than any of the actual prose that explains it.

> You can't make it to those p-values by mistake.

That's a dangerous statement. I'm pretty sure I got p-values of over 1 or under 0 by mistake. People will make every mistake you think is impossible. (Not that I think it's a mistake here)

No. Absolutely not. There is no non-fraudulent explanation here.

In the best case of "poorly described methods", we're talking about authors who blatantly lied about how they sampled their patients and deliberately omitted what would have been an extremely laborious process, involving many thousands of extremely non-consecutive patients, to produce an extremely well-matched treatment and control group.

You've got your choice as to the exact nature of the enormous lie or fabrication, but there's no possibility of innocent omission.

I think it's difficult to eliminate the possibility of a tremendous degree of incompetence, without outright fraud, based just on the p-values near 1.

As noted in other comments here, one possibility is that the authors failed to correctly describe the procedure they used. Sheldrick does consider this, pointing out that even if they actually did try to match subjects, unlike what they said in the paper, it would be very unlikely that they could match them well enough to obtain the reported p-values. But another possibility is that they were totally incompetent at using their statistical software - that the reported p-values are not actually the real p-values for the tests that they said they did. Maybe they accidently used the wrong dataset, for example. Or perhaps they copied numbers from the wrong column of the table the software produced.

In a wider sense, though, this would still essentially be fraud - it's dishonest to publish a paper that purports to be a competent study when really you are totally clueless about how to do research, or care so little about correctness that you put no effort into checking that you did things correctly. (And surely they would have to suspect that they don't really know what they're doing...)

This isn't about reported p-values. Sheldrick computed p-values from the reported treatment and control group samples. The latter are basic count data, not the output of statistical software. So here we are talking about either the raw data being wrong, or this tiny amount of raw data being incorrectly summarized.

Something like a column duplication would have resulted in perfectly identical data between treatment and control groups, which this data does not show.

The best case scenario is an errant copy-paste or Excel formula that somehow presented both groups of data in a way that actually only looked at one of the two groups, but also somehow introduced small differences along the way.

How exactly this could happen is beyond my imagination, but perhaps it introduces a sliver of doubt as to whether the paper is fraud by intention or merely by negligence.

I'd like to question how poorly the article was peer reviewed before publication