Hacker News Reader

Comment by ISL | Hacker News Reader

As I tell my students all the time -- properly-executed, an experiment that doesn't work is not a failure. If you don't internalize this idea, you will find experimental physics to be psychologically untenable.

If you build an experiment (at any scale) to test out an idea that you think is going to work, and it doesn't work, you made progress. Now you know something really essential -- that thing that seemed like a well-motivated idea has some kind of unknown flaw. With that in hand, you can proceed to try something better, whether it is a refined approach or something entirely new.

Without doing the experiment, that well-motivated idea would still look like something worth trying.

If a large team of people were willing to spend years and lots of resources on an Alzheimer's clinical trial, it was likely to be because they thought it had a real chance of working. If it didn't work, we all benefit from their effort.

View on HN

I've worked in both experimental physics and experimental biology (specifically in drug discovery, an early phase that can lead to clinical trials).

Failed experiments in biology don't make much progress compared to failed experiments in physics. beyond the inability to control for wide range of variables that are normally managed in physics, most people in biology aren't good at running high quality, reproducible experiments that falsify hypothesis. I have seen many coworkers run the same exact gel over and over until they get one that "looks like what they were expecting". I don't see the same level of experimental inadequacy in physics.

The best approaches in biology now seem to throw away the idea that we can construct a rational narrative to explain the biology that causes a complex disease, and more on collecting very large amounts of high quality data to produce sophisticated statistical models that deal with the underlying complexity in a useful way.

ON the other hand, those methods of collecting large amounts of high quality data might only seem promising because they're effectively just ways to run thousands or millions of gels at once before picking out the ones that "look right." The fundamental incentives to not publish null results and to publish or perish haven't disappeared. It's not that people aren't capable of running high quality, reproducible experiments, or that the tools aren't there, it's just that people need to publish to advance in their careers. These new high throughput screens get your quick publications and let you overfit models to the data you collect, and it might seem like a transformation now, but if people weren't incentivized to spend a couple hundred bucks to replicate a gel in the past, do you think they'll actually be incentivized to spend thousands of dollars and computation time to try and replicate screens and the models derived from them?

I gave a talk on basic bioinformatics algorithms to my pathologist colleagues once, explaining that it all amounts to collections of good guesses and you really need to know how this was done before you believe it. They just sat there, waited for the lecture to end, and went back to their jobs, accepting whatever the molecular labs were telling them that week. They simply have no mechanism to push back except to play gatekeeper, and they can't even be an informed gatekeeper.

The issue here is that for the past 25 years, multiple teams have spent years and lots of resources on the same idea despite it failing to work every previous time.

Negative results are valuable. The same negative result over and over starts to feel a lot more like sunk-cost fallacy.

On the other hand, if you test the same hypothesis again and again and again, and don't finish up a trial and use interim data readouts to inform decisions, then after enough time, by pure chance, you'll apparently be able to pull an effect size out of the air and convince wall street and apparently even maybe regulators that your drug now works!

I really hope the FDA tells biogen to take a hike. The last thing people need is false hope, and the last thing we need as a society is to be spending billions of dollars on an alzheimers drug that doesn't work. For something that important (and that profitable), the least biogen could have done is run that trial through the end.

From a previous post by the blogger on a separate drug washing out:

And I think I’ll add this every time we have a big drug trial wipe out: keep in mind, that under a safety-only regulatory regime, that people would have been taking this drug (and paying for it, one way or another) for the last few years now, and it would have done them no good at all. When they could have been actually taking something that might help them. We do not know enough about investigational drugs to approve them without efficacy data.

And the great thing about safety-and-efficacy regimes is that until you get a drug to work, the market can't be filled. If the FDA makes exceptions for biogen, then the alzheimers market for new drugs shrinks, disincentivizing research into drugs that will actually work. Why invest billions to make drugs that work if you can make that same money by pumping trash into the market?

There's a post for that one, too :) https://blogs.sciencemag.org/pipeline/archives/2018/05/25/fe... and https://blogs.sciencemag.org/pipeline/archives/2018/06/21/we..., which concludes:

the next wave will not be just people who have seized on this plan opportunistically, but who have targeted it right from the start. Who will pick out terrible diseases for which there are no therapies at all, the better to insure a supply of completely desperate patients and families, and provide them with utterly useless therapies at stiff, stiff prices. Utterly harmless therapies, though – don’t want to get sued while you’re raking in the bucks. I’m talking polysaccharide therapy for pancreatic cancer, antioxidant phytonutrients for Alzheimer’s: in other words, corn starch and grape juice. What the hell. The customers are going to die, anyway. Why should they die with their money as well?

It's unfortunate that in this legal situation the most it seems regulators can do is avoid giving snake oil regulatory legitimacy and hope that people are dissuaded by "these statements have not been evaluated by the fda, and XYS is not approved to diagnose or treat any disease." At the very least though, it would be nice if the FDA didn't then turn around and give legitimacy to fancier snake oil - hopefully they do the right thing with biogen haha.

You can thank orrin hatch for the total unregulation of the suppliments racket. Or curse him, which is probably the better course of action.

https://blogs.sciencemag.org/pipeline/archives/2019/02/12/th...

for actual drugs, the FDA can and does show up at your plant unannounced with guns for two weeks for an unannounced audit more or less whenever they want.

I await the day they can drop the hammer on the woo-aisle at whole foods with very poorly concealed anticipatory shadenfraude. Right now the best they can do is to sue POM Wonderful and the like : https://www.forbes.com/sites/chloesorvino/2016/05/02/the-ver...

I wonder if going after Purell is a way to start testing the extent of their power?

https://www.nytimes.com/2020/01/28/health/purell-fda-ebola-v...

Often failed experiments, in their failure, give an indication of an entirely new direction. Many groundbreaking discoveries are made that way.

For example, the discovery of synthetic dyes came about from trying to synthesize quinine.

https://en.wikipedia.org/wiki/William_Henry_Perkin

The discovery of the vacuum tube was an accident resulting from Edison playing round with light bulbs trying to figure out why the filaments were breaking.

https://en.wikipedia.org/wiki/Thermionic_emission

Ah, but if you "build an experiment (at any scale) to test out an idea that you think is going to work", you've already gone wrong.

This is the Tao of the null hypothesis: if you build an experiment to test out an idea you think won't work, things can only go one of two ways—either you'll be proven right, or you'll discover something about the universe. Either way, you get to brag!

I always thought that was sort of the point of the business side of "Fail Fast".

Try it now, if it doesn't work, now you know something valuable and you can try something else.

As someone who suffers from a lot of analysis paralysis, distraction ... sometimes I find just making myself keep typing and trying things in order to fail quickly and learn from it leads to much faster / sooner successes rather than burying myself up in all the minuta and every aspect of something and... not doing it.

There was an old TV show called Connections where they tracked various inventions and progress over history, and IIRC it is telling that almost every positive step they note involve someone failing to do something else they were trying, but still finding out something valuable that adds up to the next step for something else and so on.

https://en.wikipedia.org/wiki/Connections_(TV_series)

Unfortunately, even though these amyloid drugs keep on failing, the "try something else" part of the "if you fail, then you've learned something and can try something else" equation doesn't seem to really register with the industry for whatever reason. We keep pouring money into amyloid trials when it's pretty clear that there's still a lot of basic research to be done to find new leads to pursue.

It's also probably not encouraging that the Connections TV show spans ... often hundreds of years, or more.

A lot of us are just falling for someone else.

This is a teaching that really stuck with me from one of my university professors as well. I really wish this was something that was taught more to the general public, I think especially in schools. At least for me, it's something I'd never heard said in a succinct way until I was in university.

I wish this idea would sink in for scientists! Academia (my experience is in biological sciences) has a long tradition of hiding negative results and only valuing the rare positive results in a sea of messy data and unexplainable failure.

I doubt many scientists don't know this. However the current situation is largely if not all motivated by how funding is provided. Instead of a culture issue, I'd say it's a money issue.

Out of interest, roughly what proportion of experiments in physics "don't work"? Does data that fails to support the experimental hypothesis get published prominently in physics journals?

99% of experiments in fundamental physics "don't work" in this sense. All of them get published. If you got funded to build an apparatus, people are going to want to know what you saw.

You can check for yourself, by looking at the experimental papers on the arXiv [0]. Every paper here that mentions data, but doesn't have the word "discovery" in the title, is a negative result. There are thousands of such papers.

0: https://arxiv.org/list/hep-ex/current

I suppose you could think of it as an unexplored continent that has very specific requirements to survey each region. You mount an expedition, and you probably just find more trees and brush. The important thing, though, is that now that area is on a map, and the map keeps getting bigger.

I'm having no luck finding any URLs, but if you look at histories of many physical constants, you see that new measurements tend to lie within error ranges of the most recent previous measurements.

But occasionally, there's a jump to a substantially different value.

The explanation is that scientists debug their setups and equipment until they get a result that's plausible, relative to prior published measurements. Because otherwise, people would be driven to figure out what they did wrong. And it could be embarrassing.

But once they do, they tend to stop tweaking, and publish.

So it's only scientists who don't manage to get plausible results, and yet are confident that they've debugged well enough, who publish results that end up becoming new standards.

Reminds me how some rejections from HR felt like a motivator. When I could finally talk a bit about what was needed I could isolate the flaws and redesign the "theory".

I wonder if it's passion that makes researcher pain-free regarding failure. Or if they also dreaded it but learned how to dodge or tame it.

Reply on news.ycombinator.com