Hacker News Reader

Comment by jonnycomputer | Hacker News Reader

>The Chinese team is not rejecting the results obtained by the team at UoR—instead, they suggest it is possible that the nitrogen dopant present in their material was of insufficient quantity to produce the desired effect. They also note that in their sample, the dopant was unevenly distributed. They suggest further testing is needed to verify the results obtained by the group at UoR.

Replication can be hard. Failure to replicate shouldn't lead to immediate rejection.

View on HN

Replication can be hard. Failure to replicate shouldn't lead to immediate rejection.

My understanding, though, is that this researcher has a very problematic track record. It may not be appropriate to give him the benefit of the doubt here.

At the same time extraordinary claims require extraordinary proof. The refusal to allow anyone to use their samples, or to apparently provide sufficient information to allow someone else to create the material they claim to be a room temperature superconductor, leans to it reasonably being an incorrect claim. Others point to past behavior as indicating this is fraudulent, but I'm just going to assume it's accidental error.

Maybe we should just ignore all findings until another team has independently reproduced the result.

Even if what the original team concludes is true, a repro failure could indicate that it's under-documented. And if it's under-documented, then has science really moved forward?

Or we could come back to reality and listen to the actual professionals about why the reproduction might not have worked, because of known issues in the reproduction.

The only conclusion I can come to is that the comments here are exceptionally low value.

If something can’t be reproduced it definitely should be rejected.

'Cannot' being the operative word.

'Did not' does not necessarily mean 'cannot'.

If we want to go down the pedant route, the original group did not share their samples, therefore a true 'cannot reproduce' claim will remain out reach until they do so.

exactly.

Yes, but one team failing to reproduce it doesn't mean it can't be reproduced. Once there's two or three teams unable to reproduce it we can be confident to say "the described method doesn't work".

That usually means the initial team did not describe the experiment sufficiently accurately.

(Unless it is fraud.)

Or didn't control it well enough, and the important details are something completely different. Or that they made a mistake.

I see a juggler handling five balls. I succeed at three balls. I fail at four balls. Have I proven that it's impossible to juggle five balls at once?

I see a juggler handling an unspecified quantity of balls. They will not tell anyone how many balls are being handled. How do I know what they accomplished? How would I replicate it without knowing?

You've proven that the juggler's instructions alone are insufficient to juggle five balls. In this case, the juggler presumably has muscle memory you don't, which is equivalent to running an experiment on equipment that behaves differently. Or perhaps the juggler didn't convey the special trick you need for four or more balls.

Maybe the juggler was juggling 5 balls, but they made a mistake with serendipitous consequences and then told everyone they juggled 5 balls, when they really don’t know how they were juggling 5 balls. In fact one of them might have been a different sized ball or maybe it was a cube.

Worse still, they won’t let you check their original balls to see if they actually are capable of being juggled.

Lol ball juggling /s

In this case I mostly agree, but more generally there are so many more wrong ways to do something difficult that you could have 100 failed replications and 2 reproductions and it would just mean 2 people did it right.

Reply on news.ycombinator.com