Hacker News Reader

Comment by godelski | Hacker News Reader

> but the excitement and amount of replication on this paper was pretty fun to witness and experience.

I was really elated to see how people were so interested and getting to see what peer review in science actually looks like. How in the real world it is done outside of journals and conferences, which people frequently give the misnomer "peer review." I hope people will walk away from this experience with a better understanding of how science works and why replication is such a critical aspect of it. Because the truth is that our academic incentive structure has generally fallen out of alignment with the actual goals of science.

View on HN

Unfortunately I don't think we got "to see what peer review in science actually looks like" because this was such an unusual deal. The amount of interest and excitement gave us the ideal amount of peer review/reproduction. For the vast majority of things nobody even tries to reproduce it, and many of the publishers don't even provide the tools needed to do so.

Yeah, I agree that for the vast majority of things most people don't try. Or at least publicly demonstrate that they tried (key phrasing). But it is also silly to think that 3-5 people sitting at a desk reading a summary of work can validate said work. Really they can only invalidate or specify that it is indeterminate, but neither of these are validation. Which that's a key difference from the general public understanding of "peer review" (meaning journal publication).

But it might also be worth noting that often reproduction happens behind the scene. People point to big works like that which comes out of CERN, LIGO, or other massive projects and state that such works cannot be replicated. But actually those have high rates of replication, which is why there are hundreds of authors on the work.

For LK-99, people got to see a lot of what is typically done by grad students who never tell the public what they did (or even their community). That the communication between scientists is happening through preprints, email, twitter, and other methods that are not journal publications. Because science happens faster than the journal cycle. Most scientists are reading preprints, and letting the work dictate the signal of validity long before a journal can.

But what I was alluding to, which you might have picked up on, is that the reward system we have in place ("publish or perish", h-index, journals, etc) are misaligned as they do not reward this cornerstone of science -- replication -- (typically discourages is) unless there are credible claims of breakthroughs of the highest kind. Maybe we should rethink this system, and I hope that the timing of this along with the other discussions of academic fraud can help people to question the system and metrics that we use to evaluate, and ask if they are actually aligned with the original goals.

>I hope people will walk away from this experience with a better understanding of how science works

I don't think this was at all what this saga was about. People essentially turned a physics experiment into social media drama and science had nothing to do with it.

I also don't think the 'academic incentrive structure' has fallen out of alignment with the actual goals of science, despite the fact that people keep saying it, and in particular not in condensed matter physics.

If anything this whole thing showed two things: 1. Science works fine, 2. Please keep it to the actual scientists instead of turning it into yet another discipline dragged on Twitter. I know it's an unfashonable thing to say these days, but 99.9% of people have literally nothing to contribute to a debate about bleeding edge physics research, despite that apparenly everyone feels entitled to have an opinion on it.

> Because the truth is that our academic incentive structure has generally fallen out of alignment with the actual goals of science.

Did it though? Nobody published this, which is good, right? And then Max Planck Institute gave the most conclusive answer, and they're the most prestigious replicator-to-be mentioned, so that also sounds good right?. And now, Mr. L and Mr. K will not receive funding for this material, because it decisively failed to publish, which is also good?

I don't know. It sounds like the academic incentive structure worked really well here.

> Did it though?

Yes. Science is about more than producing novel ideas. A lot more.

> Nobody published this, which is good, right?

People did publish it, and many responses. They published on Arxiv. I think you are misunderstanding what papers are. Papers are simply a communication method between domain experts. The purpose of journals is to improve distribution and to provide a signal to help experts sift through the (possibly) large number of work. But you need question if they are accomplishing the goals. Do they provide more access than arxiv? Certainly no, arxiv is about as accessible as it comes. Do they provide better distribution than other modes? (colleagues, google scholar, semantic scholar, Twitter, etc) This is debatable and likely depends on your domain. Do they provide a useful signal to other experts? Also arguable and depends on your field. I'd say that the more papers/yr in your field, the weaker the signal. THEN you need to ask if these benefits outweigh the costs. That's both monetary costs from governments, corporations, and universities as well as the time costs to format the papers for the specified venue, deal with the back and forth with reviewers, and being a reviewer yourself. After you have considered the costs and benefits you can answer if these venues are good for science.

> Max Planck Institute gave the most conclusive answer,

Via "preprint"

> And now, Mr. L and Mr. K will not receive funding for this material, because it decisively failed to publish, which is also good?

Indeterminate. Science is noisy. You're wandering around in the dark. All we know is that they failed. But >90% of research results in failure. What is a better question is if their work advanced scientific knowledge. Which it looks like it did.

> I don't know. It sounds like the academic incentive structure worked really well here.

I'm what you think the academic goals are, which we need to know before you can determine if the incentives are aligned. For some added ethos, I'll reference Peter Higgs[0]

[0] https://www.theguardian.com/science/2013/dec/06/peter-higgs-...

> I was really elated to see how people were so interested and getting to see what peer review in science actually looks like. How in the real world it is done outside of journals and conferences, which people frequently give the misnomer "peer review." I hope people will walk away from this experience with a better understanding of how science works and why replication is such a critical aspect of it.

I saw people become enamored with a Russian anime cat girl on twitter.

This was vapid, consumptive entertainment. Which is perfectly fine, let's just not pretend it's better than the bachelor because science. Replace Chad had a date on love island with Anime cat girl did the science things, and that's about where we're at.

Reply on news.ycombinator.com