Cancer reproducibility effort faces backlash

Hopefully the people trying to reproduce results don't follow the same strategy.

Also see https://en.wikipedia.org/wiki/Data_dredging .

I remember one professor studied metabolic reaction networks modeled with differential equations. The networks themselves were oversimplifications and relied on ~5N parameters (N being the number of compounds in the network). The problem was that while all the examples in publication converged on a nice steady state (yay homeostasis is a consequence of the model!) it was trivial to randomize the parameters within their bounds of experimental measurement and create chaotic systems. Did this mean the model wasn't so great? No, it just meant those couldn't be the real-life configuration of those parameters . . . sigh. And now I'm a data engineer and no one asks me to get data from an API that doesn't actually provide it and I'm much happier.

In the vast majority of labs, this is nothing nefarious, it's the way science is done. If something fails, you try to figure out why it failed, try to fix the issue, and then repeat the experiment. Only once everything works correctly can you get valid data to then try and interpret. And if you get an interesting result, you still need to repeat the experiment 2-3 times to be sure.

Repeating failed experiments isn't an issue - and has nothing to do with alchemy. It's just basic troubleshooting.

You could re-run your experiment of course if you thought there was some experimental methodological error, but as you disclose somewhere your first, your second, and your third dataset all showing the same data with more or less the same methodology, you would have to show increasing confidence in your fourth or fifth dataset (the one you would otherwise publish alone), because it has to explain all of the earlier datasets as statistical flukes only: you would no longer have the incentive to run the fourth experiment alone and publish it without reference to earlier trials.

To give an example, suppose anyone is rewarded by publicity if they show that flipping U.S. coins favors heads by more than 2%. This is temptingly easy to do if you don't publish experiments that don't statistically prove that: you just keep re-running the experiment until you get the result you want at the p-value you want, so if the reward for the result is more than the number of experiments you need to do to get it times the cost of each experiment (which can be truly tiny), the experiment presents an attractive ethical hazard.

But if you are ethically forced to publish (or even summarize) your earlier complete experiment and dataset really in any old place, then if there is no actual difference it becomes vanishingly unlikely that you can suddenly prove your theory and explain all of the earlier experiments as statistical outliers. You would stop exploring after your second or third experiment. (Which you would still quickly summarize.) This does however present an added burden to researchers, especially if they quickly test something before really refining the methodology to do so. So these quick disclosures could still be considered quite dirty and not very meaningful. However their disclosure would give a good indication regarding how strong a result really is. (i.e. by glancing at how much dirty data precedes the actual experiment being published.)

-

† I'd like to recall the words "moral hazard" because it does indicate that people are tempted toward the bad behavior. But economically I think that term is too specific (meaning risk-taking where someone else has to clean up in case of failure) -https://en.wikipedia.org/wiki/Moral_hazard

Many fields (macroeconomics, IO, etc.) write models that end up with some type of calibration which is implemented in thousands of lines of code. Those lines are written by RAs with almost no programming experience, so what happens is this:

While (result!=expectation) Ask assistant to look for bugs and repeat simulation

The end result is that you don't stop when there are no more bugs, you stop when you've got the coefficient signs you want, and then get published.

Only if your paper has a high impact, has the code and number open, AND was coded in a very easy language, people discover the flaws:

"the paper ... was, and is, surely the most influential economic analysis of recent years [...] First, they omitted some data; second, they used unusual and highly ques ionable statistical procedures; and finally, yes, they made an Excel coding error"

http://www.nytimes.com/2013/04/19/opinion/krugman-the-excel-...

There are known and tested protocols that can fail. Not every step can be accurately recorded. It's very common that an experiment will not work well the first time it's performed (even when supervised by someone experienced). Over time, researchers improve their skills and achieve better results following the same exact protocol. Does that mean that the science behind the experiment is bad?

The pharmacy of the future will be entirely curative to non-invading diseases, will repair the DNA that's been mutated, will express or inhibit the proteins that need to be expressed or inhibited. And small molecules will be the payload of these fancy protein-based nano-machines. But this hunt to bring down the cost of those small-molecule targets at the cost of the reputation of the science itself might be foolhardy when the cost is near-infinite in the first place because they're looking from the wrong perspective.

tldr; Pharmaceutical companies' hammer that worked so well against the nails of bacterial infection is in no way suited to the plumbing of cancer. And now they're 'investigating the plumbers' to figure out why their fancy new 50-billion-dollar "water-hammers" don't work so well to unclog pipes.

I don't know enough to know if that's true or not. But it seems at least possible that in the rush to reproduce much, the project is cutting costs by using less-skilled contract labs.

There may well be other labs which tried, and failed, to reproduce the original author's findings. In any event, it is unlikely we would know, as failure is unfortunately rarely publicized in academia (especially so when such failure is commonly dismissed out of hand for reasons such as a "lack of expertise").

I personally think the work this replication work is incredibly important and should absolutely be a key priority for funding institutions.

Rick Young is an MIT Professor, has been in biology research for 3 decades, is in the National Academies of Sciences and has published hundreds of high-impact papers across a huge number of disciplines. I know this doesn't mean he is God but to say that he doesn't understand "sampling errors" or "heterogeneity between labs" is an absurd insult.

https://biology.mit.edu/people/richard_young

> 30 emails, wow!

It can take up to half a day to reply to an email containing even one fairly technical question. Obviously the time invested depends on the content of those 30 emails, but it's not at all difficult to imagine this eating a week worth of a PI's time.

> 2 weeks of a graduate student's time -- these are the people who are the least paid right? Below minimum wage even?

The typical attitude is that this is a justification for not making them spend their time managing these sorts of tasks. You don't get to say "we get to pay you crap because your job is exciting and rewarding" and then load that same person down with a bunch of grunt work (on top of all the normal grunt work). That's how you bleed students and kill the lab's productivity.

Also, grad students are not well paid but they are typically no cheaper than post docs; PIs pay tuition.

> The demands on their time seem so low, yet the complaints are so high

So let's imagine a world where people calling for massive improvements to reproducability get their way. Let's say it is a month worth of time for the lab. For each paper. Multiple papers a year. That's a pretty massive time investment. If you're a top lab, that could become a full-time position. And believe me, you're not going to be able to fill that position with a grad student. That person will have to be well-paid, because their job is going to suck.

So it's reasonable that scientists are peeved when they invest all this time and don't perceive their collaborators as acting in good faith, or feel like their collaborators are trying to cut corners to pinch pennies.

That you used brand-X tubes should generally not be important to the science (conceptually, though sometimes they are physically), and so should be left out of the paper's protocol precisely so that another lab does not use that tube during a replication study. It's not bad science that using brand-X tube yields different results than brand-Y tube, though it can be frustrating to learn that is the crux of the difference you've been seeing. It is precisely that kind of variable that must be ferreted out, and can often only be ferreted out when labs attack the same problem from a different angle.

The Reproducibility Project is run by people in the business of replicating experimental design for profit. They argue that a proportion of public research money should be given to businesses like theirs for this process. This will take a lot of resources that would otherwise be spent examining biological knowledge from different angles rather than the same angle multiple times. In the end, we will know less.

If the reproducers don't quite know what they are doing if left to their own devices, they will invariably be influenced by the original researchers into being blind to exactly the same mistakes and making the same wrong inferences.

Another software analogy: it's like QA people needing detailed hand-holding from the original developer in testing some program, instead of independently looking for ways to break it. That is then no better than the original functional testing by the developer.

I feel it's like a piece of closed source software releasing "the source code" and yet no one can make it compile. You don't have to have a clean codebase, it can be a bit tricky to build, but it has to be able to be compiled before anyone can have any confidence in your claims. If you don't provide that, it's much less useful.

The separate issues of for profit labs and what support they should expect in this context are tough issues, but we shouldn't undermine the science to thwart them.

Can you elaborate on why this new approach isn't even worth trying?

How do you "focus on the veracity" of the claim if you cannot reproduce it?

This is a phenomenon I've seen many times in my work as a consultant translating academic discoveries to commercial code. This work invovles digging through both published papers and actual code to find that some grad student has implemented a clever heuristic that is not described in the paper because it is "inessential" that is what the claimed performance actually relies on.

The article says, "It's unrealistic to think contract labs or university core facilities can get the same results as a highly specialized team of academic researchers, they say."

I can understand this sentiment: science is hard, and at the cutting edge it's really hard. I've certainly got results that I think it would be hard for others to achieve and I've succeed with experiments where others have failed simply because I have good hands.

I once took over an experiment that had been worked on by a heavy smoker with shaky hands. He had developed a complicated apparatus for streching ultra-thin plastic foils over a frame. I just used my fingers. If I had published the way I did it and someone like him had tried to reproduce it, they would have failed.

However... the kind of reproducibilty being talked about here is not (primarily) technique, it is results. If a result can't be reproduced because the cell line was so very special or whatever, then it likely isn't robust enough to be very interesting in practical biology. It certainly isn't of much interest in drug discovery. So while the science may be OK, results that are incredibly hard to reproduce are of very low value for biotech, and that knowledge is worth something in and of itself.

> none of them were able to make a dish as good

Defining "good" is the problem for both cancer research and cooking. You can have two experts saying this is "good" or this is "bad" and not really be able to prove who is right. Fine, maybe for cancer research you can ultimately prove who is right but its not realistic in most cases given the resource constraints of even top-flight labs.