Hacker News Reader

Comment by tinco | Hacker News Reader

At this point, since papers like Nature and Cell are so important to scientists, could it be feasible for them to simply require any submitted paper to them would only qualify for publication if the results were independently replicated?

They could even smoothen the process by giving the draft a 'Nature seal of approval' that the authors could use to get other institutions to replicate their work, and add a small 'Replicated by XX' badge to each publication to reward any institution that replicated a study.

Funders of studies might improve the quality of the research they paid for by offering replication rewards. I.e. 5% of all funding goes towards institutions who replicated results from research they funded.

Of course there would still be some wrinkles to iron out, but surely we could come up with a nicely balanced solution?

View on HN

I think you vastly underestimate how much a lot of scientific studies cost. It would be ideal to be able to have studies replicated by a separate group, but there are many times labs that have specialized equipment and engineering that make replicating studies by some random group unfeasible.

In addition, if you spend too much time trying to replicate others' work, you have no time to work on the things that will actually continue to fund your own research.

The best thing is to have a healthy, skeptical, but collegial, competition in the field. That still requires more funding though!

> I think you vastly underestimate how much a lot of scientific studies cost.

On the other hand, the money currently being invested seems to produce mostly garbage. "Fake science", just to rustle some feathers.

> I think you vastly underestimate how much a lot of scientific studies cost

Do you have data for this claim? There's tons of extraordinarily expensive experiments through and through, but there's also stuff with incredibly high and time-consuming up-front design and exploration costs that is actually almost trivial to replicate on a per-unit basis.

If it only works on a Wizbang 200 and cannot be reproduced with the Doodad 3000 then it probably doesn't work or there are other variables missing from their results. Specialised equipment should not affect results (proving that equipment is broken has some value though).

Really, even without specialized equipment, science is expensive. And many scientists got into the field because they wanted to do their own work.

Locking a lab that can't afford to fund an independent replication study out, or who isn't prominent enough to be able to rouse someone's attention to do it, would be disastrous for a lot of early career researchers.

I published in nature. The results were based on a computation that was so large, no other entity in the world could have reproduced it at the time (I work for Google, we built a system that used 600+Kcores continuously for years). I don't think ,at the time, that anybody other than Amazon or Microsoft could have run the same calculation as us.

Should our publication have been prevented simply because nobody else had the resources to replicate?

> Should our publication have been prevented simply because nobody else had the resources to replicate?

Well... maybe, yes?

Just because you are the only ones able to do the computation doesn't mean your methods or your results are correct.

We share all the results from our research, so people may inspect them in detail. We also share our methods in enough detail that if somebody was sufficiently motivated, they could reproduce the work by spending time and money

however, If you follow your logic, you would prevent CERN from publishing the discovery of the Higgs Boson, because nobody else has a particle accelerator powerful enough to detect them. You would prevent the LIGO people from publishing because nobody else has a multi-mile-long interferometer that can detect gravity waves. There are many unique resources that would be prevented from publishing under your idea.

Contrary to what most non-scientists claim about science, reproducibility is not a requirement for discovery science. What is import is scrutiny- ensuring that when people discover something using a unique resource, other people can inspect the methods and the data and decide whether to believe the results.

Though I agree for the most part, isn't CERN using multiple teams on multiple detectors, even going so far as having multiple implementations of data analyses and such, just to make sure any result they find is never a product of one person or team making a mistake in their process?

CERN has multiple detectors (CMS and ATLAS are the two best-known) but they serve different purposes and are attached to the same system (LHC). Exacycle also did something similar (we ran multiple different protein folding codes on the same infrastructure).

You're addressing another issue: they built replicability into their scientific method (which is awesome) but it's still within a single logical entity (which happens to be distributed throughout the world).

LIGO went one better and injected fake events to make sure their analysis systems were working.

Well, maybe they could not run the full experiment, but maybe they could audit the source code of the computation you ran, maybe just a small subset of the computation on for example a synthetic smaller dataset, that could maybe be used to test whether your computation works like it should.

Did you think about the replicability of your research while you were working on it?

Gromacs (the code we used) is open source. MSM, the code we used to analyze the data, is open source. We ported MSM to an internal Google framework (Flume), so the resulting code could be open sourced, but wold be lacking the Flume framework to run inside.

As for thinking about replicability, sure I thought about it. That's why we released all the data we generated and described our methods in detail- so pharma and academics could use the data with confidence to design new drugs that target GPCRs. I also decided that rather than focus on replicability, I would focus on discovery and sharing, and I think that actually is a better use of my time as well as the time of other scientists.

How can there be any other answer than "yes"? The point of science was that it's reproducible; you are saying that what you did was not; therefore, it wasn't science.

What is the difference between your paper and an Uri Geller "experiment"? Both are extremely hard to duplicate for objective reasons so their results have to be accepted based on reputation. (Imagine someone trying to publish the same thing but using "Startup LLC" instead of "Google".)

Actually, reproducibility is not considered a necessity in science. It is factually false to say that a person who carries out an experiment in good faith and shares their methods and results is not scientific- they just aren't satisfying a criterion which is a good thing to have.

It's pretty clear what's different between my paper and a Uri Geller experiment. If you can't see the difference, you're either being obstinate or ignorant. We certainly aren't banking on our reputation. A well-funded startup with enough cash could duplicate what we did on AWS GPUs now. I would be thrilled to review their paper.

> Actually, reproducibility is not considered a necessity in science.

By whom? I think you're using as a premise the very matter that is being discussed.

I would say that most scientists, at the top of their game, who have years of trusted publications behind them, should be given some latitude to publish discovery science without the requirement of reproducibility, so long as they share their results in detail, and accept postdocs who then take the resulting research materials to their new labs.

I didn't always believe this, but after spending a bunch of time reading about the numerous times in history when one scientist discovered something, published the results, and were told they were wrong because other people couldn't reproduce their results, only to be shown correct when everybody else improved their methodology (McClintock, Cech, Boyle, Bissell etc).

This ultimately came to a head when some folks critized the lack of repeatability of Mina Bissel's experiments (for those who don't know, she almost singlehandledly created the modern understanding of the extracellular matrix's role in cancer).

She wrote this response, which I originally threw on the floor. http://www.nature.com/news/reproducibility-the-risks-of-the-... After rereading it a few times, and thinking back on my experience, I changed my mind. In fact, Dr Bissel's opinion is shared by nearly every top scientist I worked with at UCSC, UCSF, and UC Berkeley. The reason her opinion has value is that she's proved time again that she can run an experiment in her lab that nobody else in the world is capable of doing, and she takes external postdocs, teaches them how to replicate, and sends them off to the world. In other cases, she debugged other lab's problems for them (a time consuming effort) until they could properly reproduce.

I believe reproducibility is an aspirational goal, but not really a requirement, for really good scientists, in fields where reproduction is extremely hard.

For those interested in learning more about excellent scientists whose work could not be reproduced:

Tom Cech and his grad students discovered that RNA can have enzymatic activity. They proved this (with excellent control experiments eliminating alternative hypotheses) and yet, the community completely denied this for years and reported failed replication, when in fact, the deniers were messing up their experiments because working with RNA was hard. Cech eventually won the Nobel Prize.

Stanley Prusiner: discovered that some diseases are caused by self-replicating proteins. Spent several decades running heroic experiments that nobody could replicate (because they're heroic) before finally some other groups managed to scrape together enough skilled postdocs. He won the Nobel Prize, too.

Barbara McClintock- my personal favorite scientist of all time. She was soundly criticized and isolated for reporting the existence of jumping genes (along with telomeres) and it also took decades for other groups to replicate her work (most of them for lack of interest). Eventually, she was awarded the Nobel Prize, but she also stopped publishing and sharing her work due to the extremely negative response to her extraordinary discoveries.

Mina Bissel went through a similar passage, ultimately becoming the world leader in ECM/cancer studies. She will likely win a Nobel Prize at some point, and I think we should learn to read her papers with a certain level of trust at this point, and expect that her competitors might actually manage to level up enough to be able to replicate her experiments.

Thanks for the lisr. I think negative effects of labeling a paper irreproducible when in fact it is not is greater than the positive effect of correctly labeling a paper correct.

Because if a piece of work is important enough. People will find the truth. If it is not important enough.. then it does not matter.

Many times it requires years to build a laboratory which can even begin to replicate the work of an established operation.

And sometimes it is impossible to solve a difficult problem using the first lab one builds in pursuit of that concept.

In these cases the true type of laboratory needed only comes within focus after completing the first attempt by the book and having it prove inadequate.

Proving things is difficult not only in the laboratory, but in hard science it still usually has to be possible to fully support your findings, especially if you are overturning the status quo in some way. That still doesn't require findings to actually be reproduced in a separate lab if the original researchers can demonstrate satisfactory proof in their own unique lab. This may require outside materials or data sets to be satisfactory. Science is not easy.

I think all kinds of scientists should be supported in all kinds of ways.

Fundamentally, there are simply some scientists without peers, always have been.

For unique breakthroughs this might be a most likely source, so they should be leveraged by capitalists using terms without peer.

> Actually, reproducibility is not considered a necessity in science.

Incidentally, Scott Alexander just published an article [1] with a great quote:

"Peer review is a spam filter. Replication is science."

[1] https://slatestarcodex.com/2017/02/27/ssc-journal-club-analy...

There are two ways that things can be reproducible: reproducible in practice and reproducible in principle. You are stating that you have a computation that is reproducible in principle (just in practice, unfeasible).

I find this an easy and useful distinction and your publication should NOT be prevented from being published by this measure.

This would be a pretty big step - right now there is a big incentive to be first to publish. Often the researchers who would be doing the replication are competitors, who are racing to publish their results first.

Part of publishing is to put it out for discussion. It should be the best effort and thought to be true, but science is about the evolution of knowledge. Very few things are completely understood, especially in biological and micro/nano fields.

This would effectively lock certain fields out of publishing in those journals. For example, a great deal of population health research involves following people for huge spans of time.

Reply on news.ycombinator.com