Plan to replicate 50 high-impact cancer papers shrinks to just 18
sciencemag.org
sciencemag.org
This is truely a plague in research right now. I've came across quite a few instances where authors told me that their experiments weren't reproducible even for them! They do not note this in paper because ultimately everyone needs to show some result for the funding they received (i.e. "published paper"). They obviously never want to share any code, data files, hardware etc. On one instance, author wrote me back that they can't share code with me because they lost all the code because their hard drive crashed! Reproducibility is a fundamental tenant of doing scientific work and this is actively and completely ignored in current peer review system.
I think conference chairs needs to take stand on this. We get now 4X to 8X papers in tier 1 conferences. Reproducibility could be a great filter when area chairs are scrambling to find reasons to reject papers. Sure, there will be papers where very specialized hardware or internal infrastructure of 10,000 computers were used. But those papers would be great for Tier 2 conferences.
Seriously, why do you think the academic world doesn’t change? Is there too muc vested interest in the current system?
Passing peer review is something used as a substitute for reproducibility. Just like testing a null hypothesis is a substitute for testing the research hypothesis, and citation count is used as a substitute for making precise and accurate predictions.
The procedure (scientific method) that has lead to all the great stuff we have around us today, I'd even call it a pillar of civilization, has been undergoing a piecemeal replacement since approximately WWII.
Wow, pretty low bar for science. You can have peer review for the actual results and what they mean, as well as them being reproducible.
> The procedure (scientific method) that has lead to all the great stuff
I don't think you know what the scientific method is.
1) Explore and describe aspects of the world in detail, figure out what situations naturally arise or that can be devised that produce consistent, stable phenomenon.
2) Abduce (guess) an explanation for these consistent, stable phenomenon.
3) Explore the logical consequences of assuming your guess is correct. Figure out a few otherwise surprising (ie, inconsistent with other explanations people may have) predictions that can be deduced from it.
4) Collect data and compare it to the predictions generated in #3
5) Discard the guess or modify it to make it more consistent with the data produced in #4
6a) If the guess is modified, Go to 3.
6b) If the guess is discarded, Go to 2.
A lot of cancer research seems to be failing at step 1, ie they can't even get into the "testing otherwise surprising predictions" loop since there is no consistent, stable phenomenon to trust.
1. Define a question
2. Gather information and resources (observe)
3. Form an explanatory hypothesis
4. Test the hypothesis by performing an experiment and collecting data in a reproducible manner
5. Analyze the data
6. Interpret the data and draw conclusions that serve as a starting point for new hypothesis
7. Publish results
8. Retest (frequently done by other scientists)
Emphasis on 4) and the "reproducible manner".
Otherwise what's the point really? We would resort to trust or belief in one's sayings.
I wrote:
"figure out what situations naturally arise or that can be devised that produce consistent, stable phenomenon."
This is the same as figuring out the "reproducible manner".
You said:
> Passing peer review is something used as a substitute for reproducibility
And what I'm saying is "No, it's not". Reproducibility of the results is the most important aspect of the scientific method, otherwise, as I said above: "what's the point really? We would resort to trust or belief in one's sayings."
https://www.timeshighereducation.com/features/peer-review-no...
In other fields such as experimental physics or biology it might be more difficult to achieve the same effect though, as experiments are quite hard to repeat in general: For example, in my former field (experimental quantum computing), building the setup and fabricating the sample required for a given experiment could take years of effort, making it almost impossible to "just" reproduce someone's work for the sake of verification.
That said, in experimental quantum physics the exciting results tended to get replicated within a couple of years by different teams anyway, not because these teams wanted to verify the results but rather because they wanted to build their own experiments on top of them (I imagine this is similar in other fields). Another natural way of exchanging knowledge and improving reproducibility was via the exchange of PostDocs and PhD students: If you do good work in one group, another group will usually be very eager to give you a position so that you can help them to set up the experiments there as well. I'd even argue that this is one of the main mode of knowledge dissemination in experimental science today, as most research papers are just extremely hard to reproduce without the specific -and often not encoded- knowledge of the individuals that ran the experiments.
I'm not sure though if it is practical to document everything in a research paper in such a way so that any person can reproduce a given experiment, as many of the techniques are very specialized and a lot of the equipment in the labs (at least in physics) like sample holders, electronics, chip design templates and fabrication recipes is custom-built, so documenting down to the last detail would take years of effort.
That's why (IMHO) written PhD theses are so important, because that's kind of the only place where you can write 200-400 pages about your work, and where you can include minute details such as your chip fabrication recipe, a description of your custom-built measurement software and a detailed summary of the experimental techniques used in your work. In that sense, PhD theses are probably more important to reproducibility than short papers.
I used to be a "everything must be replicable by even the simplest of people" person, but I changed to "for progress to be made there must be <X> competent people who can reproduce challenging experiments and run new ones".
Maybe more people would be able to run the experiments if papers detailed how to do so. The knowledge would no longer be esoteric.
> Sure, there will be papers where very specialized hardware or internal infrastructure of 10,000 computers were used. But those papers would be great for Tier 2 conferences.
This is an accurate description of a _lot_ of Google papers when it comes to Computational Linguistics. And I don't know whether a Tier 1 conference that rejects the biggest players could remain a Tier 1 conference for long.
There are many papers in Comp. Ling. right now that claim state of the art by running neural networks for a couple weeks on very expensive hardware. If those papers are not allowed in Tier 1 conferences then we should call it "state-of-the-cheap-art", but accepting them undermines the whole point since only a select few can afford it.
I don't say it can't be done. But I do think that it's not trivial.
Having opaque methodology is not on the other end.
http://blogs.nature.com/ofschemesandmemes/2018/08/01/nature-...
A few political science journals also hire staff to make sure that the code reproduce the results in the article:
https://ajps.org/ajps-replication-policy/
A nice writeup of the AJPS's experiences:
https://www.insidehighered.com/blogs/rethinking-research/sho...
One major problem is that there's not much real incentive to make your work reproducible. Money granting organizations favor researchers breaking new and exciting ground, not those rehashing an already published method. Publishers don't require reproducible methods, and reviewers don't have the time, desire, nor expertise to do an in-depth methods review.
Wet lab experiments are 1-2 orders of magnitude more expensive and difficult to reproduce, that's true, but we're not even getting the basics right!
Think of a research lab as a company that gets paid per prototype and then has to market the concept for the next prototype in an infinite loop. If you can't package up what you're doing into a sequence of small prototypes then you're not getting paid.
You can't expect the results to be reproductible 20 years later otherways.
Speaking of, I haven't played with R - what are its standard methods for handling dependencies? I'm particularly enamored of the pip and npm way of doing it, where you create a version-controlled artifact (requirements.txt and packages.json, respectively) that defines your dependencies. Does R not have a similar system, or do people just not use it?
I'm a bit bitter about the whole "writing reproducible code in R", as I'm currently wasting a lot of time trying to get R code I wrote at the start of my PhD to run again now I'm writing up.
also, labs keep the data around, and sometimes use it again for novel posthoc analyses, even years later. in fact that's what I'm doing now!
OTOH, it would be awesome for me, as I've been doing consulting on exactly this for 15+ years :)
How do you do science if you can't repeat experiments?
"How do you do science if you can't repeat experiments?"
Well, the way we've been doing it for 300 years? It's not like the flawed processes of the past didn't yield anything. To put it in operations research terms: science doesn't work like hill climbing, it's more like a stochastic optimization process without a (well defined) halting condition. Again, I'm not saying we can't and shouldn't strive to improve, but the first year grad student level huff puffery in this thread is just completely detached from reality. 'Science' as you were explained the concept in high school is only a high-level abstraction of the concept, it's not how actual things are or get done in the real world.
personally I find the worriers overstating the problem of reproducibility. lots of results are reproduced all the time, but in the course of testing other hypotheses.
if a result depends on exact replication of methods it isn't that robust an effect, and just might be a trivial one.
I first started thinking about this problem after I heard this talk by Arfon Smith [1]. It contextualizes reproducibility tangentially in discussions about viewing modern research as a heterogeneous (nodes can represent papers, experiments, software, authors, etc.) dependency network.
[1] https://speakerdeck.com/arfon/academias-biggest-blind-spot
That's the tool, and that page cites the paper.
Who is paying for non-reproducible papers? What’s the point?
https://en.wikipedia.org/wiki/Reproducibility#Reproducible_r...
The term reproducible research refers to the idea that the ultimate product of academic research is the paper along with the laboratory notebooks [12] and full computational environment used to produce the results in the paper such as the code, data, etc. that can be used to reproduce the results and create new work based on the research.[13][14][15][16][17] Typical examples of reproducible research comprise compendia of data, code and text files, often organised around an R Markdown source document[18] or a Jupyter notebook.[19]
> abstract numerical disciplines
Could you give an example of a non-numerical science?
If a paper is very novel and impactful but difficult to reproduce, it doesn't seem to matter much for citation counts as long as people just believe it to be true.
And yet there's no revolution among funding sources to send money away from those and instead towards researchers who spend more time on it.
"Could you give an example of a non-numerical science?"
This is the point you're probably going to argue about what is or is not 'science', so let me use 'scholarship' instead from the fields I work in: law, environmental science and geography. There is a lot of research that is not purely numerical in nature. Note that this doesn't mean that it doesn't use numbers; you misread what I wrote - I didn't say 'numerical disciplines', I said 'abstract numerical disciplines', by which I meant disciplines that require at least some judgement of qualitative properties or inexact measurements of things along the way, somewhere.
No, but I had spent a whole year at university trying to replicate experiments in physics lab. I was studying CS but they had too many physics teachers so we had many of the same courses :)
The equipment was abused by decades of students, you had 45 minutes to do the experiment, and the results usually weren't even the correct order of magnitude :) It was stuff like measuring speed of sound, atmospheric pressure, gravity constant, etc.
Still - at least I knew it's the problem with equipment or with me, not with some obscure details that weren't mentioned and that I can't replicate.
> My point is that there are a lot of aspects to most research that aren't easy (or even at all) to replicate
And my point is that it makes publishing everything that can be adjusted that much more important.
Another question - when there's economic and proffesional incentive to fudge the numbers, and no way to check if the numbers were fudged - how do you trust the results?
I wouldn't.
I’d assume people are paying for it who want high quality research.
This is why everything should have CI, and one should ask at least one person to set up the system from scratch using just the documentation.
There are some incentives to make work reproducible. If you publish a method implemented in software, then making that software easy to use, well documented, with example data and respond to questions/issues etc it greatly enhances the chance other people will use it and cite it. Things have been getting much better in the last 5 years.
Preprints also help - I saw a preprint the other day where people complained on twitter there was no methods section and no code. The authors responded. They realise that potential reviewers may be the ones making these comments or at least see the comments. This talks to your point about not treating a paper as a single point in time, but an ongoing process.
I think things are improving. I am now seeing papers publish Jupyter notebooks in python or R scripts to reproduce all the diagrams and analyses, along with curated data that just plugs in. In general this actually works (although sometimes crucial bits of data are missing, and unlike an error in the paper, these are not necessarily fixed).
Snakemake is a parallel build tool designed for data instead of software as its main use case.
You're right. I use laziness out of frustration, but in reality I know it's hard for me to make my own work fully documented and reproducible.
A large part of the situation is cultural. Software engineers are born into a world of version control, unit tests, documentation, managing complexity, reproducibility, debugability, etc (and we still struggle with it). That sort of culture and associated tools are missing from data science.
Big governments need to fund it and solve this problem once and for all. At the moment most grant procedures require something more concrete than just "solve the reproducibility problem", though.
I found a paper that deleted (one at a time) each gene in yeast, to see which ones were "absolutely required". They published the list. For each "novel discovery of a required gene", I was able to show the gene overlapped (yes, genes overlap) with an already known required gene, and so, much of the paper's "novel discovery" section came into doubt.
It's nice because genes have precise coordinates and range intersection is cheap to compute.
https://www.nature.com/news/1-500-scientists-lift-the-lid-on...
https://en.wikipedia.org/wiki/Replication_crisis
Papers have become a target for success, so scientist need publications for better status and remuneration:
http://www.sciencemag.org/news/2017/08/cash-bonuses-peer-rev...
A new scientific "mafia" is in place around the world:
https://www.technologyreview.com/s/608266/the-truth-about-ch... https://retractionwatch.com/2017/08/10/paid-publish-not-just...
So papers have become a good business, no the way to disseminate outstanding research results.
That's awfully cynical and over-broad, but I agree to a point. Greedy and unscrupulous publishers are part of the problem, but so are lax or unprincipled scientists eager for prestige and a career-making publication in a top tier journal. It's an unfortunate chicken-and-egg cycle now with no easy way to cut it. Perhaps more emphasis on replication post-publication? Perhaps a reputation system for unethical publishers or scientists?
That's just incredibly unfair. There are some fields and methodologies where p-hacking and cherry-picking have been a problem, but the primary reason that papers aren't reproducible is just noise and basic statistics.
As a scientist, you control for what you can think of, but there are often way too many variables to control completely, and it's probable that you miss some. Those variables come to light when someone else tries to work with your method and can't reproduce it locally. However, real scientists don't stop and accuse the original authors of being "unprincipled" -- nine-point-nine times out of ten, they work with the original authors to discover the discrepancies.
It isn't surprising at all to actual, working scientists that most papers are impossible to reproduce from a clean room, using only the original paper. It's the expected state of affairs when you're working with imperfect, noisy techniques, and trying tease out subtle phenomena.
There are some fields and methodologies where p-hacking
and cherry-picking have been a problem, but the primary
reason that papers aren't reproducible is just noise and
basic statistics.
Almost any result could in principle be attributable to noise; where are you planning to source all of the funding to run large enough studies to minimise that? And no matter how large your experiments or how many you run, you're still going to end up with some published results attributable to noise since, as GP says, that's the nature of statistics. By its nature, you cannot tell whether a result is noise. You only have odds.
I'm not saying there aren't problems with reproducability in many fields, but to suggest that you can eliminate it entirely is naive.
No, not naive - wrong.
Well, with a single paper the odds indeed are that it's noise. That's why we need reproduction. Now of course a paper needs to be published for it to be replicated later. But the paper (and/or supplemental material) should contain all possible things the research team can think of that are relevant to reproducing it - otherwise it's setting itself up to be unverifiable in practice. Papers that are unverifiable in practice should not be publishable at all, because a) they won't be reproduced and thus it'll be forever indistinguishable from noise, and b) there's no way to determine whether it's real research, or a cleverly crafted bullshit.
My issue is the flippant and silly claim that "[i]t's possible to imagine a version of academia where results that can be attributed to noise don't get published".
Take a sampling of a large number of papers, give them some sort of rating based on whether they provide enough information to reproduce, how clear their experimental and analytical methodology was, whether their primary data and scripts are available, etc, and then look at that rating versus their citations.
Hopefully, better papers get more attention and more citations.
(And yeah, "peer review" as it is done before a paper is published is not supposed to establish a paper as correct, it is supposed to validate it as interesting. Poor peer review ultimately makes a journal uninteresting, which means it might as well not exist.)
Sounds like a ridiculously low standard. If your paper is in principle unreplicable, then I only have your word for evidence of what you're claiming. This is not science. Even journalists are held to a higher standard.
I'm not a real scientist or even a pretend one, and I'd like to believe your 9.9/10 figure, but don't delude yourself there aren't those out there publishing papers for the sake of nothing more than retaining their position in a university. Or bumping their citation count or pushing an agenda or whatever.
We're in this 'reproducibility crisis' precisely because this game of science being played doesn't reward reproducibility and scientists are just as much participants as publishers are.
If you don't get a quantifiable amount of reproducibility, there is no point to using statistics at all and what you are doing is not science.
https://en.wikipedia.org/wiki/Predatory_open-access_publishi...
https://jefferson.libguides.com/predatorypublishing
https://beallslist.weebly.com/
And now predatory conferences:
https://en.wikipedia.org/wiki/Predatory_conference
https://libguides.caltech.edu/c.php?g=512665&p=3503029
However, the storm actually started with the failure of reproducible of research in high impact journals:
https://www.frontiersin.org/articles/10.3389/fnhum.2018.0003...
http://www.sciencemag.org/news/2016/02/if-you-fail-reproduce...
https://www.timeshighereducation.com/features/reproducing-re...
This is not an chicken-egg problem, it is an ethic problem, motivated by remuneration and funding.
Problem areas include detailed protocols and reagents used. If you don't what someone did exactly and what they used then replicating it is going to be very difficult.
[1] https://www.linkedin.com/pulse/scientific-reproducibility-re...
And everyone understands that its hard to know exactly what info someone else will be missing, so it wont be perfect at first. That is why there is supposed to be a back and forth the first time a new method is used.
Figuring out what needs to go into the methods is going to be an iterative process. As it is right now its a disaster though. Like they say in this article no one even knows the most basic info like cell density. The entire system is not set up to deal with replications at all because they haven't been doing them.
I don't agree. I think this stuff should be in supplemental notes (we call it the 'artefact' in my field, not sure if that's universal). A paper should be reasonably readable, not a huge list of processes in detail.
Why do journals accept these papers? If you can just make shit up, it kind of makes the whole value of a paid journal moot. What value do they add if not reviewing the damn paper?
For example Yoshihiro Sato was recently (2016-2017 time frame) caught fabricating results, turns out he had been doing it for 30 years. He had more than 20 papers retracted. Also note he wasn't caught by a fellow scientist in the field he was caught by a statistician bulk processing papers looking for anomalies. So yeah big penalties if you are caught, but the odds of getting caught are so low you have to be stunningly stupid to get caught.
You make a good point about colleagues in the same lab noticing unethical behavior. A shared research culture that values integrity and transparency is a good defense against malpractice and fraud. Building up and maintaining that culture—and the methodological skills to back it up—is a continual challenge.
Even medications made in high volume processes have rejected units (and sometimes, batches). Process drifted, machine failed, raw ingredients had some issue.
A scientific paper at the very early stage is meant to show that something is possible if the conditions are just right. It may be actually outside the capability of the lab to document (or measure, or even know) every variable that makes up these conditions. One of the reasons there is such a chasm between lab and practice is that as people dive in for deeper review and productization, they find the process is too finicky, the applicability too narrow, etc. But the scientific paper is the first step in this process of discovery.
That said, it's critical that authors document whatever they do know or have control over. Electronic publishing, tools like Github, etc, all make the barrier for disclosure much lower.
Confirmation is inherently impossible, but we should be able to weed out bad papers with higher accuracy.
a) if the null hypothesis is true, 5 times
b) if the null hypothesis is not true, anywhere between 0 and 100 times (depending on what is the true alternative)
This is quite different from your previous assertion that the expected number of successful replications would be 95 out of 100.
> Even medications made in high volume processes have rejected units (and sometimes, batches). Process drifted, machine failed, raw ingredients had some issue.
Sure, but here the process is known in details, so the manufacturer can just rerun it. A scientific paper (with supplemental material) should be exactly such a detailed process description, so that researchers could run it if they want.
Now, if the paper does not contain enough information to enable its reproduction, then there's really no way to tell whether it described real results, tampered results, or whether it was just made from whole cloth.
There's no shame in being wrong. But there is shame in making sure no one can check if you're wrong.
Ideally, every research team would have a remote collaborator that checks their results and gets credit for doing so (even if the answer is no!). But right now there is nothing to incentivize that kind of rigor.
that seems really cheap to me based on my brief experience with lab research. replicating 18 studies is a great achievement.
Source: https://xkcd.com/980/huge/#x=-1166&y=-2798&z=5
Add to that expenses for the experiment, and $1000 seems not far fetched.
While labs have huge economies of scale—-I worked somewhere with a quarter million mice—-the mice aren’t just random field mice. Some of the genetically engineered ones are hundreds of dollars a pair (and possibly more if custom, raised under unusual conditions, etc). Xenografts need skilled labor and very clean conditions for the immunocompromised mice. $1000 seems a little high, but not much, especially if the work isn’t being done by an underpaid grad student.
By the way, you can poke around some of the animal vendor's sites. They mostly have their pricing on their websites along with a bunch of other information. Four really big ones are Taconic, Jackson Labs, Envigo, and Charles River.
While you are doing the experiment, everyone knows theres a 10% probability of eventually being selected, so healthy pressure to make sure everything is properly documented, and everyone looks out to detect fraud by collaborators.
I don't see the problem, unless its the 10% price hike... if 10% is too much, just do 90% of the usual number of projects. I'd prefer 10% less projects if it ensures much higher reproducibility rates...
big deal, 11% more expensive science, but 1 out of 10 results get reproduced, stoichiocratically, so you don't know if it will be reproduced until after publication...
Also: your comment about reproducibility details being scattered over the previous work of the original authors... as I said, in a world where we use my system, you are incentivized to put all details for reproduction within the paper, since you wouldn't want to risk possible reproduction by others to fail simply because they didn't read your previous papers...
There have been several news items on Hacker News recently about academic publishing, reproducibility in general and pre-print servers and I would love it if eLife got some more attention, for better or worse, for the excellent work they are doing and helping to fund.
To those biology researchers in the comments, please submit your work! even failed results! Even if it's software heavy. Editors are listening closely to your feedback.
edit: disclosure, I work for eLife
Alert options explained here: https://elifesciences.org/alerts
early versions: https://elifesciences.org/rss/ahead.xml final published versions: https://elifesciences.org/rss/recent.xml
Outside of biology, I have seen many "academic" papers published on computer-related topics that refer to software programs developed by the papers' authors that are crucial to the research but not publicly available. Is there any similar unwritten rule to that in biology where another researcher reading these papers can request a copy of these programs from the authors?
Obviously, in many cases other researchers cannot replicate and verify findings without access to the same research tools used in the published papers.
Then the study already isn't replicable by definition, so why waste time asking the original authors? Just mark it down as 'not replicable' and move on.
Nobody ever reproduces papers exactly, because you can’t. There are too many variables, and even though you try to control as many as possible, you can still be blindsided by the random variate that you didn’t anticipate.
Scientific results that are robust to random variation are the important ones. The ones that can only be reproduced exactly as specified are most likely to be “meaningless”.
March 2012: https://www.nature.com/articles/483531a
June 2015: http://www.sciencemag.org/content/348/6242/1411
Dec 2015: https://www.nature.com/news/cancer-reproducibility-project-s...
Jan 2017: https://www.nature.com/news/cancer-reproducibility-project-r...
Not a chance. I don’t think these guys have any idea what they’re doing. I feel sorry for the funders.
I worked there for three years :)
If all there doing is running a few western blots, because that's all they can do on this budget, well then ok. But I'll venture to guess that that's a far cry from reproducing the most important results in these papers.
Most science isn’t.
Most of it is a set of nearly fanatical beliefs about reality only vaguely related to facts agreed through consensus by authority. Look at how strongly doctors fought against germ theory. They saw no reason to wash their hands for delivering a new born after having spent their morning cutting open cadavers. Their response was anything but scientific even in the face of easily tested claims and results.
If research can’t be reproduced it’s just a story with interesting data. I doubt most scientists using statistics would be able to provide the alpha / p value / confidence interval etc for their hypothesis.
When the bar is set so low should we be surprised by low quality?
Rant complete.
Initiate beer.
The handwashing protocol is there to stop people getting sick, circumventing it because you're a doctor and you know better is not going to help.
It’s amazing how real something becomes when beliefs intersect with consequences, especially those involving self-preservation.
Understand the over the top and common sense angles here, but the "unmonitored" drop was so dramatic as to imply that this was less a matter of reverting to a more sensible approach and more one of dropping down to outright carelessness. That may not actually be the case though.
The official rules are slowly changing, and the funding agencies tend to require that scientists make raw data available, and put effort into making their experiments reproducible. But the reality is changing much more slowly than the rules.
"In fact, many of the initial 50 papers have been confirmed by other groups, as some of the RP:CB’s critics have pointed out."
This article is staying assiduously neutral, but one perfectly valid interpretation is that the original initiative was flawed in a way that was predicted by the initiative's critics. Science is routinely reproduced -- just not by labs working in isolation, using publications as clean-room instruction books. This is the sort of thing that programmers believe about science, not something that scientists believe themselves.
There are a great many serious, legitimate scientists who believe that this "reproducibility crisis" is verging on irrational, and it's important to consider their arguments. It's particularly scary to me that so many comments here are dovetailing with the sort of nonsense you encounter on anti-vax and global warming denial forums. We're literally gaslighting the process that has done more to advance society than any other in human history:
http://www.pnas.org/content/115/20/5042
"The discovery that an experiment does not replicate is not a lack of success but an opportunity. Many of the current concerns about reproducibility overlook the dynamic, iterative nature of the process of discovery where discordant results are essential to producing more integrated accounts and (eventually) translation. A failure to reproduce is only the first step in scientific inquiry. In many ways, how science responds to these failures is what determines whether it succeeds."
"In fact, many of the initial 50 papers have been confirmed by other groups, as some of the RP:CB’s critics have pointed out."
I don't know that I would be willing to take a statement like that at face value. I once read of a scientist who claimed that there was absolutely no need to try and independently reproduce his work, since it was being routinely reproduced in college labs everywhere as part of other experiments - or something like that. To me that meant that it should then be trivial to reproduce these results when attempted by trained professionals who were specifically trying to do that very thing. So why the reluctance to allow that to happen?
Nowhere is it written that you should be able to "easily and routinely" reproduce a scientific result. It's damned hard for skilled scientists to reproduce most scientific results. And yes, it's utterly impossible for amateurs. I'm sorry if that makes you uncomfortable, but it's the truth.
There are many things in life that you routinely accept on authority: When you fly in a plane, that it isn't going to crash into the earth. When you turn on the tap, that clean, safe water comes out. When you go to your doctor, that she is prescribing you medication that will help you, not hurt you. When you vaccinate your children, that you are protecting them from horrible diseases. I'm sorry that it bothers you, but "science" is just one more thing in the world that you're going to have to accept, because nobody -- scientist or otherwise -- can independently verify all of human knowledge.
If it makes you feel better, you can look around, realize that we're no starving from famine, or dying from minor cuts, or poisoning ourselves with lead, or dying of Polio or Smallpox, and you can try to convince yourself that in the long run, the method works. Because that's the argument you're missing. You don't have to take my word for it, or anyone else's. It's a system. The system works. You just don't fully understand why it works, and you're not willing to read when a group of scientists write a long article that tries to explain it to you. Because you'd rather believe that explanation is "dogma".
"I don't know that I would be willing to take a statement like that at face value."
Nobody is stopping you from digging into it. If you're really so bothered by it, I encourage you to follow up.
"So why the reluctance to allow that to happen?"
There is no reluctance, whatsoever. Nobody is trying to stop these people. They're just beginning to realize that their approach is a lot damned harder than they originally expected, and those of us who knew it would be are pointing it out.
Don't you have any more recent examples? Perhaps people used to be doing reproducible science but this got lost somewhere along the way, that's why all your examples are decades old.
And I don't blindly trust that "the system works" really at all to the extent that's so it often claimed to. An awful lot of what gets published these days tends to just not hold up to close scrutiny, and often what appears to be groundbreaking research may just kind of up and disappear with no further trace soon enough, without ever directly or even indirectly leading to a practical new product or a new drug or whatever. I understand that this kind of thing is going to happen, and it may happen quite a bit if you're doing really groundbreaking stuff, but in general this should probably be more the exception than the rule.
But yes, I do understand how the method works, and how the system that's supposedly based on that method works (when it actually does work), and also when it doesn't work and why, and what the motivations might be to try and pretend that it works even in situations when it actually doesn't. And rather than blindly defend it, maybe you try a bit harder to understand how it actually "works", too.
Something is wrong about this. How did these "other groups" get the protocols while this reproducibility project couldn't? Are they being stonewalled? Did the other groups not actually get the real protocol but instead fiddled around until they got the same result? Unfortunately that claim doesnt have any citations.
Its that at least two groups try to measure the same thing under the same conditions (as similar as you can get them).
you don't. You either add a 3rd party (4th... 5th...6th... Xth)or accept the conclusion. If you add a third party how do you know they are trust worthy: infinite regression.
But of course the more independent, even adversarial, the two groups are the better. Like in this story it mentions the "critics" of this project claim other labs (their friends?) have already replicated these studies, apparently using the super secret protocol they couldn't explain to this group.
I'd compare it to using sms 2FA. Is it perfect, no. But it is far, far better than no 2FA at all. And there's nothing stopping you from putting your banking app on the same phone you use for 2FA.
Now the leaders of this project are basically saying - we can't do difficult experiments, and can't replicate studies which have already been replicated by others.
What have we learned exactly, except that experimental biology is difficult?
These papers underlie a lot of modern research and treatments. If they aren't reproducible, the papers' results, and other papers that use their results come into question. This is a serious, documented problem [0].
> What have we learned exactly, except that experimental biology is difficult?
The problem wasn't that the project couldn't do difficult experiments, it was that not enough information was provided by the original papers. That's what we learned here.
Cancer research is not an edifice constructed on foundational results. It is not like physics in this regard, where for example, accurate calculation of the gravitational constant is vitally important, and over time repeat measurements are generally more precise but not wildly different, or do not dispute the importance of the gravitational constant.
Take one of the papers they reproduced - Transcriptional amplification in tumor cells with elevated c-Myc. This is an interesting result conducted in highly contrived experimental conditions. Nobody is going to base their career, a drug development program, treat a patient or even start a PhD based on this result alone. I say this is a translational cancer scientist and medical oncologist. The contribution of this paper is to our knowledge of the biology of Myc, which has a multitude of actions which are context dependent. Myc is studied in a variety of different ways using many different methods. If this paper were being repeated today, the technologies and techniques would be quite different. The result of this paper is not plugged into the central dogma of cancer biology, setting us down an erroneous path for the next thousand years. So to return to my original question, what have we learned in trying to replicate this study, that we didn't already know? The money would have been better spent on orthogonal validation/extension of the result using modern techniques - another name for this is 'science'. The replication crisis suggests that this routine extension/validation process is somehow less important than going back and repeating the original experiment, which is I think a complete misunderstanding. You also seem to be saying that we should ask researchers to document in excruciating detail all experimental conditions such that a pastry chef or meteorologist could walk into a lab and successfully reproduce the experiment - this is an impossibly high bar to set for scientists who are already working under very difficult conditions, and is not the solution.
It sounds like you think it doesn't matter if that result was published vs "Transcriptional amplification in tumor cells with depressed c-Myc".
If I misread that title replace it with whatever is the opposite result in this case. Anyway, like I said elsewhere if it isn't worth trying to replicate, then the original study should have never been funded. How many of these studies just exist as a "jobs program"?
It sounds like you think it is most of them, in which case great. We can then easily cut out 90% plus of funding current going towards jobs program stuff and devote it to the <10% that is worthwhile...
I'm not sure what you are trying to achieve by making outlandish comments about cutting funding or jobs programs (the idea that cancer research is a jobs program is utterly hilarious!!), but it doesn't really seem you read my comment.
If no one cares whether the authors got it all wrong and "Transcriptional amplification in tumor cells with elevated c-Myc under conditions xyz" should actually be "Transcriptional amplification in tumor cells with depressed c-Myc under conditions xyz", then why was this funded?
If someone does care then it should be replicated.
It is tested in other forms, but isn't generally replicated in the sense you seem to think is paramount. Nor should it be. I'll give you a silly example - would you support a project to go back and replicate electromagnetism experiments performed at the start of the century? Say we do repeat Millikan's oil drop experiment and get a different result (which is actually what happened) - does this mean there is a reproducibility crisis in physics? If we don't repeat the exact experiment, does that mean that Millikan shouldn't have received funding? Why is it that replicating the result the way Millikan did it more useful than doing other related experiments with more sophisticated or different apparatus? The latter is actually MORE useful.
Yes, of course! That is a great idea. Everyone should be doing this experiment in high school or undergrad science class by now. In fact that seems to be a thing:
https://hepweb.ucsd.edu/2dl/pasco/Millikans%20Oil%20Drop%20M...
If no one cares whether the authors got it all wrong and "Transcriptional amplification in tumor cells with elevated c-Myc under conditions xyz" should actually be "Transcriptional suppression in tumor cells with elevated c-Myc under conditions xyz", then why was this funded?
I disagree. Consider computational biology papers – you should be able to show me code that takes the raw data and turns it into your results. Peer review is a sort of global, public code review in this case. There is a lot of value (education, validation, propagation, sharing, etc), and it's completely practical. Sadly, we (as a field) are nowhere near that, IMO.
My argument is that a 'reproducibility project' of the kind described above is pointless and impractical. I do not see evidence that this project has taught us anything.