Escaping Science's Paradox

This so much reminds me of my company's approach to innovation. First, let's create an "innovation group" whose job it is to innovate. They'll have their own budget. See investors? We're innovating!

Make sure the innovation group isn't comprised of anyone with any experience or expertise in the industry vertical, or any other vertical for that matter. After all, we don't want them tainted with experience! That may interfere with their innovation! They can't be constrained by existing knowledge!

Create a plan for how the innovation group is going to innovate because, you know, we need to show the investors what's being bought with their money. So, let's look at the set of problems that have been bedeviling us and everyone else in our industry for years and lay them out on a roadmap! We're going to solve all these problems!

What you end up with is a group comprised mostly of recent college grads having next to no experience and a group of older grifters who've decided to make a career out of "innovation." Those poor college grads have no clue they're being conned and that the grifters are blowing smoke up everyone's ass until 3-4 years later when little to nothing has been "innovated" they're all let go. The grifters know when to bail, those poor college grads don't. The grifters move on to con somebody else, the college grads meanwhile struggle to find another job.

I've seen this play out several times at several different companies. You'd think people would catch on and it would stop but no one wants to admit they made a mistake. So the grifters con on.

This is all to say you can't plan innovation, just like you can't plan inspiration. You need to have a company culture where people can scratch their own itch and have time and resources with which to innovate if they so desire. It's the same with science - you can't plan discoveries and breakthroughs. You don't know a priori where all the blind alleys are. Failure is useful - it communicates to others where not to look, but no one wants to admit the failures so they don't tend to get published. Therefore we only want to put our money on what we're reasonably sure will succeed, which is risk-averse and here we are. More and more money going in and fewer results coming out.

I feel like the 21st century is all about going through the motions and not caring about the actual results.

This is interesting.

I have a couple of points. It says:

"To be sure, the problem seems much less acute in harder sciences – e.g., physics, chemistry, cosmology – that have an established tradition of skepticism, replication, or even blinding researchers to their own conclusions."

Of is this because there are very few people who are trying to replicate the studies? The equipment is a barrier. Its not clear...

My second point regards innovation, is that surely replicability is essential. If studies aren't reliably replicated, what exactly can the innovation be? So I see replication as a core issue.

Replicability is also only one of the problems science faces. The other issue that isn't well discussed, is that funding comes from 3 sources by and large - the government, the military and corporations. We, as individuals, cannot assume that this method of funding will serve our interests. Why would studies on 'good food' receive funding over a new type of medication. When the funding is tied to profit, or vested interests, there is only so much inclination to 'rock the boat'.

But in general, I'm certainly glad this very major issue is being discussed. I'm not holding my breath for any solutions. And I only wish there was greater awareness of how much trust, or faith, goes into un-skeptical acceptance of scientific pronouncements.

When I was working in research, we had a journal group which each week critiqued the statistics done in scientific journals - most often contemporary articles from Science or Nature.

As I recall, every single one of the articles we looked at would have failed our "Red Team" standards.

I don't (only) mean that they failed by making trivial errors of statistics - I mean a small team of researchers could find fundamental errors that potentially or likely undermined or contradicted the whole article. It's not surprising that something like 50% of the conclusions are not reproducible.

"I do not think there’s a contradiction between reproducibility and innovation. Contrary to common belief, we can improve both at once – by incentivizing failed results, and by funding “Red Teams” that would aim to refute existing dogma or would be entirely outside it."

I don't know - the mechanism that creates the problem with reproducibility is that all teams are "Red Teams". What is published is what is unexpected, what deviates from a common understanding, but nothing confirming what we would expect. Most of what seems new and exciting comes out to be just random accidents - that is natural, but it is a problem only when we put too much weight to them.

This is a mixed bag; While the author never really articulates the titular "paradox" some of the background is good, but the "solutions" part is not well argued and perhaps not well thought out.

For example:

> we need to specifically empower some people to be an antagonist, with the explicit role of trying to refute, attack, and discredit other scientists and their theories

This is not a new role; it's been a career approach for scientists for basically the history of science. Presenting this as a new idea rather than basically a question of incentives draws focus away from the more fundamental questions I think.

Another one:

> After all, by the time of a Phase III clinical trial submitted to the FDA,...

Bringing up Phase III trials was a great opportunity to engage with the issue of cost vs. confidence which would have been interesting - instead is offered only as data on the publication rates of null results (again an issue mostly of incentives I believe, although that does related to the cost issue of course) Phase III trials are typically eye-wateringly expensive and partially for that reason imply the existence of a funnel of Phase 0, I, II experiments before them to make sense. The structural, financial, and opportunity costs of doing this are very much in scope for this articles topic but aren't engaged with.

Ironically, the entire suggestions section reads to me at least a bit dogmatically.

“It doesn't matter how beautiful your theory is, it doesn't matter how smart you are. If it doesn't agree with experiment, it's wrong.” - Richard Feynman

Science progresses when new paradigms are discovered. From that new mathematical models can be derived and tested.

For example, there is an experiment that breaks with quantum mechanics. It does not break it completely, but it replaces it with a different model that is so simple that it is the best according to Occam's Razor.

Like: Null-hypothesis: quantum mechanics shows that there are no photons https://saidit.net/s/Physics/comments/18rz/nullhypothesis_qu...

With the experiments we can see that photons are not only complicated, but also breaking with the observations. This breaks the paradigm of the particle-model. Instead we have something else, which we can test now with simple experiments.

Experiments are on: http://thresholdmodel.com

And I don't know if the conclusions are correct, but I am excited to see a testable breakthrough in the stalemate of quantum interpretations.

Personally, I am not a fan of an adversary approach.

I think, reproducing (or failing to reproduce) scientific results simply should be given more value/money without incentives for disproving the result.

The ability to reproduce a result drives also research as currently often the authors are not motivated to make it easy. And you don't get payed for simply redoing what was already published.

The culture in academia is afraid of failure, that's why grant money is allocated very conservatively. It sounds like that didn't use to be the case. I suspect that at some point funding was cut or was threatened to be cut by Congress and the powers that be decided that never again would they be in that situation and thus decided to play it safe.

That trauma is seen in the pressures to publish only positive results, which leads to embellishment or fabrications and thus a reproducibility crisis.

Playing it safe means that few risks are taken, and only grant proposals that abide by the current models and ways of thinking are funded. I assume that some discoveries may have been swept away as outliers or manipulated to fit into current models to ensure a smooth path towards publication.

I don't think the current system will fix itself, there's too many barriers, too many bureaucratic stops, too much politics.

Here's a zany idea: what if research proposals were funded by some kind of ranked-choice voting, and then as a separate step, research teams were assigned to the research proposals, based on their track record of whether they were able to carry out past research according to the agreed-up on plan (but not whether the results were the expected ones)? Suppose that there is also a sizeable reward for winning proposals -- enough to make it worth-while to write good ones.

There are a number of benefits: (1) prevents scientists from being too attached to their own hypotheses (2) demands that the research plan be described well enough up front that any reasonably capable team should be able to understand it (3) rewards scientists for executing well on the proposals, even if the hypothesis turned out to be wrong.

"it turns out that when famous scientists die, the subfield in which they worked sees a flowering of new scholars and publications compared to other subfields"

The only logical solution to this situation is to find ways to shorten the life span of famous scientists.

Being a red team member would be my absolute dream job.

The preregistration of studies seems to essentially be a panacea for publication bias. This would require no changes except for simple audits to ensure timelines are as described.

If the problem of scientific innovation is merely one of policy and incentives (rather than more fundamental limitations) should we not be able to point to specific policies that were changed or added that resulted in the slowdown of innovation?

If so, then I would think simply reverting back to a previous set of policies would be sufficient.

On reproducibility and innovation in science.

> Many have argued that policies seeking to avoid reproducibility issues will create a constrictive atmosphere that inhibits innovation and discovery.

Aka we can't be encumbered by the truth that it does or doesn't work, we'll just say it does until it doesn't.

You can be serious...

Reproducibility seems like a social problem of people publishing shitty science for citations. As for the rate of innovation, discovery is certainly an NP-Hard problem. How can we guarantee any sort of expected rate of innovation?