We don't know how to fix science (2021)
worksinprogress.co
worksinprogress.co
> Ideally, we’d like to measure the benefit provided by a study to society. We might ask: had this piece of research not been funded, would a given invention have been delayed? If so, for how long? And what was the impact of that on societal welfare? We could also try to estimate intermediate metrics of research usefulness, like whether a given basic research paper will end up being used as input to an application. It is an argument for humility, not epistemic nihilism.
The whole promise of basic science research is that it may be valuable in ways that are immeasurable, or certainly uncapitalizable. One major issue with science is that we're always hoping that it will say more than it should. I would take a more mindfulness-y approach to science. You do science because it is a good end, and if you do it for that reason, good things will come. If you chase "good science" you get the problems we have.
The public got invested because it had a lot of useful outcomes in the 20th century.
How would you assess things like that?
To your suggestion, I am not arguing that we should not fund things, just because we don't have an initial understanding of how it will be useful.
Applied sciences where the end goal needs to be something that actually works and not just getting published are in much better shape and arguably innately of more value to begin with.
This is an incentive problem, and more funding does not fix that issue
The other alternative of "funding lotteries" is more suspect. As the article summarizes, the argument for lotteries is:
> Advocates of lotteries make two key critiques: a) the current system forces researchers to spend a lot of time preparing grants; and b) peer reviewers cannot reliably identify “good” grant applications. They claim that a lottery system would reduce the time spent on review (because reviewers would mostly skim the proposals to check for minimal scientific robustness) as well as the time spent on preparing proposals (because there would be less of an incentive to meticulously craft proposals, given that no matter how detailed and well written they are, they are going to be chosen at random).
Of course (a) is true, but as the article suggests, the evidence for (b) is a bit shaky. Moreover, we should have strong priors against (b). To be sure, reviewing is not perfect, far from it! But, scientists must at some level be able to judge the future prospects of work, or else they wouldn't be able to make any fruitful decisions about what research to conduct. So the primary plausible way I see that (b) could perhaps be true for a given pool of applications is if all of the ones in the pool pass some thresh-hold bar for quality that makes it hard to further distinguish between them. It's not clear to me that that quality level would be maintained if we move to a lottery.
My prior is that peer review is better than random chance. This seems sensible; after all, a human being should be better at determining scientific validity versus random chance. Under that assumption, it makes sense to have peer review to narrow the pool down to a smaller number of grant proposals.
It doesn't seem unreasonable that peer review could get the "best" proposal into a pool of 10-15 applicants around 90% of the time. I think that would reduce the pressure on proposals by a decent amount.
It comes down to whether the large amount of time spent is from people polishing and re-submitting stuff that's good but not "great", or whether it comes from the initial stages of getting something to that good quality to start. In my experience, it's the latter. And if that's typical, then I don't see how the lottery helps.
It should do - if you're only aiming for the top 50% of proposals rather than aiming to be the best, then you can submit a proposal that's "good enough" rather than putting in a lot of time squeezing out the final few percent of marginal improvement.
Please, someone, correct me if I'm wrong, but I think this is essentially how the European Commission allocates scientific funding--professors basically get big block grants to fund their labs to do whatever. Then there are some big grants that go to institutions, with those like ETH Zurich soaking up a lot of the funds, and then those institutions can distribute to the smaller ones as they see fit. Rubbing up to high-profile professors at ETH or GeoForschungsZentrum feels a lot like petitioning King Frederic of Prussia, rather than anything merit-based.
It's really not clear to me that this is a better system; at least in my field (geosciences) I don't think that the European labs are more innovative or productive than their American counterparts, which is what the 'fund people, not projects' idea is supposed to enable. Furthermore, it completely blocks nonacademic research funding, or much project-specific funding. For example, I work for an applied science nonprofit based in Italy, and we have a lot of ideas that would (in principle) be big hits at NSF, but there is no real avenue to fund ~€100-300k projects on an ad-hoc/project-specific basis.
Long time ago I had a colleague who lamented that his projects were never selected by the R&D funding committee of our employer. I suggested him to multiply by ten his proposed budget and then at the next round his proposal was accepted! This was at a time were there where much more money in the Telecom sector than now.
So please do not hesitate to multiply by 10 your proposed budget. It does not have to be more ambitious on your side, all is needed is to find additional partners who could bring value to your proposal, for example by implementing it.
(I was both work package and project leader for EU FP7 projects)
At least in FP7, the EU wanted implementations and new markets or businesses, because they didn't want to pay for something they didn't understand.
Trade metrics are easy to understand and less objectionable than funding someone with an innovative idea on a domain where there are few experts. Th EU commission themselves has also to report about their actions to EU parliament and EU council.
> You are just advocating to game the EU bureaucracy better.
You may not know but it happens that EU commission, or national authorities ask for reimbursement of funds if they thinks they have been gamed. Tricking the "EU" is not advisable.
Perhaps but that's not my impression. My impression is (as in this thread) there is no good intrinsic metrics for research, except if it shows an impact on society.
So yes, bureaucracy. Because instead of actually doing the job that needs to be done, the EU is more interested in covering their own asses so that it looks as if they were doing the job. But the EU is not special in any way here, except that it is a very big example of a bureaucracy, of course.
I think that's the EU mindset (at least in FP7), it wants their funds having the greatest impact as possible, certainly a small project can't have much impact.
> Ioannidis et al. (2014) note there are 15M scientists that published anything in the 1996-2011 period, but only 1% that has published every single year in this period. This smaller 150k-strong group accounts for 40% of all papers and 87% of all papers with >1000 citations.
One percent seems like a good size for a major niche; those who do the bulk of the publishing.
This is important, but if that's the only niche you're looking at you're bound to miss 99 percent of what you want to find.
[0] https://nintil.com/newton-hypothesis
Edit: Maybe the people that need to be funded most are secretaries for some of the other 99 percent.
Or those 1% are real professional that know their stuff while the other are publishing because they are doing some internship, or publishing low quality articles on an obscure domain is a way to evade criticism (how many ALS experts in academia?) or other short commitment to a topic.
I read a lot of literature on ALS (Lou Gehrig's disease) and clearly 99% of publications have no value at all, even when their university claims of "a breakthrough".
As far I know there were breakthroughs only every ~5 years (TDP-43 in 2006, C9orf72 in 2011, FUS 2010) and no similar breakthrough since. Big companies like Biogen, which bet much on various genetic (ASO) therapies, are now removing drugs from their pipeline. We clearly do not understand the field, despite publishing 15,000 articles per year!
In particular, any system that identifies superstars would probably have identified Lesné, the guy who seems to have been forging Alzheimer's research. He had already obtained huge and lavish grants for his lab on the back of this work. Winner-takes-all is a system that would throw accelerants on the fire of fraud, as presumably once a scientist has managed to get a blank cheque cut, nobody is incentivized to scrutinize how that money is being spent too closely. They're a superstar, after all.
I also think politics in science would become even more of a problem. all of our money towards science would go toward funding research that backs their worldview.
I also think we'd see even more social media like driven science. "I am going to research this because I can write on social media/youtube on this and get lots of click/likes-> money"
That will cost more but that’s a solution.
Hypothesis: TL;DR Goodhart's law: 'the measure became the target and thus is no longer a good measure' -- Publish or die / secure funding or die has lead to risk averse research, a lack of research into new fields, and a lack of reproduction / verification of existing research (checking the known is not 'sexy' / profitable).
Experiment: Proposal: Create a 200 year funding guaranteed tenure tracking inflation funded public domain / commons expansion researcher reward that ensures a life long appointment (and retirement) based on some other excellence requirement. Maybe revisit former initiatives like 'project paperclip' (WWII) for criteria suggestions.
Data Collection / Reports: Conduct the real experiment, for the first of at least 200 years OR until all enrolled applicants retire / expire. Issue interim reports every couple years (4, 10, something) to see if clear results contrary to the typical environment persist.
[https://emilkirkegaard.dk/en/wp-content/uploads/The-Basic-Wr...]
"Science used to be valued as a means of getting to know the world; now, owing to the triumph of technique, it is conceived as showing how to change the world."
That it has.
Russell? British aristocracy. Von Neumann? Heir to an Hungarian and newly titled magnate. Wittgenstein? Wealthiest family in Austro-Hungaria. De Broglie? French aristocracy. They, of course, were insanely smart, but putting that intelligence to such a wonderful use was enabled by their background.
This is a sad state of affairs for researchers today because, by default of having the pre-existing money to explore, they must become subservient to it: grants and fundings.
That said, almost all of them come from families that did not have to struggle to put food on the table or quickly eacaped this environment with external funding. Let finding is that you need "free" time to actually do science.
Not having to worry about how to get bread on the table opens up the door for all sorts of interesting creative labour!
Newton? Grew up with a reverend as a step father.
Kelvin? Son of a teacher.
Faraday? Son of a blacksmith.
Plank? University professors.
Our obsession with credentials is preventing more people from being scientists today than the classism of the 19th century did.
Science requires action in the face of uncertainty.
To your point about inaction, from the article:
"""
This does not mean we should wait decades before implementing any change. Waiting for strong, crystal-clear evidence to act would be engaging in the same flawed thinking that led to the claim that “there is no evidence that masks work” which we heard last year—the costs of delays or inaction would be high. Demands for open access, “red teaming” science, or running multiple-lab reproducibility studies (be it in the social or life sciences, or elsewhere) shouldn’t get stalled by the lack of RCTs. Where there are strong theoretical considerations, indirect evidence, and broad agreement that a proposal will improve science, without a serious cost if we’re wrong, we should just go ahead in at least some cases, and assess the benefits afterwards.
"""
The last paragraph sums up the thesis:
"""
Those interested in meta-science may disagree about what the best way to reform science is, but all of us can agree that we need more evidence about the proposals being made. We have many interesting, reasonable ideas ready to be tried. It is a glaring irony that the very same institutions that enable practitioners of the scientific method to do their work don’t apply that same method to themselves. It is time to change that.
"""
We are getting a real time test to the question of if mask mandates work in the wild: https://www.abc.net.au/news/2022-07-22/nz-jacinda-ardern-cov...
Australia has no mask mandates, NZ does. NZ has a higher death and hospitalization rate than Australia. The answer is that it's health theater on par with shoe removal at airports.
Of course we have the typical bait and switch of pretending that masks as used in an operation theater are in any way the same as the crumpled cloth mask from your pocket.
NZ looks to me to have a lower death rate than Australia. See [0].
I haven't done a deep dive of the literature but from what I understand wearing face masks is pretty well established to correlate with lower infection rates. A Google search gives [1] which seems pretty reputable to me.
It sounds like you're picking a fight about your personal cause celebre? The original article is talking about trying to figure out how to build evidence. The underlying assumption is that once you have evidence, you use it to inform decisions. You're pushing a false narrative by not only ignoring evidence but also misinterpreting or lying about the evidence that's available.
[0] https://aatishb.com/covidtrends/?data=deaths&location=Austra...
[1] https://www.publichealthontario.ca/-/media/Documents/nCoV/CO...
By the way, lots of people believe masks don't work. You can line up places that differ only by mask mandate status and observe that the case curves are the same. There are many situations like that, but in theory you only need one to disprove the claim that mask mandates (always) work. The case against masks is made by Ian Miller in this book:
https://www.amazon.com/Unmasked-Global-Failure-COVID-Mandate...
In the past month Australia has not had a mask mandate, NZ has. NZ has had more than twice the per-capita deaths and hospitalization of Australia. If masks worked it should have been the reverse.
The same link but with Australia and New Zealand in every chart [0] just blatantly disproves your narrative.
I'm just not that familiar with the mask mandates of NZ and AUS. AUS looks to have lifted their mask mandate on June 17th 2022? Looking at the link you sent, to my eyes, there's a clear upward trend of deaths in AUS nearly three weeks after June 17th (June 10th 2022).
The numbers for both are so low, relative to other countries like the US, so it's kind of hard to really read too much into them but NZ looks at worst to be 20% more per capita than AUS.
[0] https://ourworldindata.org/coronavirus/country/new-zealand?c...
However, the only gold standard study done on cloth masks to my knowledge found no statistically significant benefit. If we are talking about cloth masks, I think we would have to accept that any benefit from them is very minimal at best.
Other types of masks like surgical or of course n95 do prove beneficial.
Currently NZ has the equivalent of 2000 US deaths a day, Australia 1300 US deaths a day and the US has 500.
So, basically, during the last week or so, NZ had roughly 25 deaths per day whereas AUS had 50-70 per day in that week.
My apologies for not interpreting the data correctly.
The data still disproves your narrative. That one graph does have AUS at 2x NZ in a few key dates but overall no.
I'm going to stop arguing about this. I think the utility of further discussion has already diminished to the point of minimal reward.
No proposal
No committee
No eminence
No hesitation
Just do
But I need money to just do. Approximately a million dollars.
Who is going to give me a million dollars without a proposal as to my plan on how to spend that?
Is one person going to be the decision maker on who gets money? That seems like it would be susceptible to strong biases. A committee could make decisions with less bias.
And how will they discern between proposals? Money isn’t unlimited, and a million dollars is a lot of money — you don’t want to just give that away to anybody. Some prior track record of success on the part of the applicant would help demonstrate their ability to accomplish the proposed research. Maybe they would demonstrate that prowess somehow, through written means perhaps, like a document, published for all to read.
So from first principles we’ve arrived back at the current situation. I don’t think things are perfect, but throwing out the whole system without understanding why it’s there will just lead to more dysfunction.
Yes no one likes writing grants, and yes publishing can be gamed. But until you solve the fact that research costs money, and a few people have control over most of the available money, then we’re (scientists) going to have to spend a lot of time convincing them to give us some of what they have.
>research costs money, and a few people have control over most of the available money, then we’re (scientists) going to have to spend a lot of time convincing them to give us some of what they have.
Definitely, and I think it pays to be very adaptable whether you are operating with an academic approach or not.
Institutions that have developed over sometimes hundreds of years can be hard to beat. I wouldn't want to stop that kind of progress. But I think there should be alternatives, and I do think a lottery has its place.
>I need money to just do. Approximately a million dollars.
I'm a lot worse off than that.
It would take $10 million to build the kind of lab I need, to deploy the full one percent of my findings.
So I would have to accept the situation, being likely to settle for less than one percent and focus on that.
And to potentially scale up from there I think I'd have much better luck by personal meetings & communication with money people compared to what results I could expect from issuing scientific papers. Where I have had some good luck when reaching a commercially viable milestone.
But that's seriously a lot of money so I would have to really be ready for some high-touch sales.
>Who is going to give me a million dollars without a proposal as to my plan on how to spend that?
I know what you mean.
And I'm at the other end of the spectrum, without a PhD. Making outright grants even more elusive today. I would have to be very persuasive about how I would return their money, and in multiples rather than just margins. So that would be a lot of explaining to do.
Which is why I built my first lab 30 years ago entirely from recycled materials.
There wasn't $10 million forthcoming back then either.
I'm glad I didn't wait.
Imagine what software development would look like if we applied the principles of science:
We wouldn't start from scratch with a new software program. Instead, we would refer to code from the 1960s that has been proven to be a development-theoretic consensus and accepted by developers.
We would not speak in software life cycles, but would either "revolutionize" a piece of software or extend the existing standard.
We would allow that we trust code from developers because some other developers say it will fit. However, we don't know if the "other developers" would have the same financial interests.
And we would allow interested parties with enough resources to influence this process at will by publishing permutations of similar software and trusting each other.
We would have our own marketing agencies that are adept at injecting arbitrary constructs of their own into the "developer consensus".
And every developer who refuses to do this, who develop their own frameworks and pursues their own software ideas in a detached manner, would be branded as a liar and their reputation be destroyed with the help of the "developer consensus" spoke persons under the payroll of corporate interests.
I guess Poe's law strikes again.
Yes, both disciplines have their own way of handling that subject. And each discipline is struggeling how to find a way to gain evolution.
For me the key performance indicator is that in software development people controlled open source now takes a crucial part in the basis of most software projects whereas in science everything is much more corporate infected.
In science knowledge is created where money is. And money comes from corporate interests.
In software corporate interests tried to control the market but in the end had to capitulate. Most servers are run based on open source, most software is developed with tools being open source. Of course, there is no standard. You can write an application in Vue, React, Angular, JSP or whatever.
Yes, in the end software industry is insance. They have managed to get accepted that every five year the old software is scrapped and basically the same software is developed again.
But letting old software die and creating a new version with new ideas in mind is much closer to the way nature is doing things.
And to be honest - the only one who really can deal with complexity is nature hands down.
Let the scientists simply do science or teaching (which is actually a method to educate the next generation of scientists) instead of having them handle an insane amount of academic bureaucracy in their work time.
The only question then is around equipment, which though is not trivial is at least smaller than salaries and equipment.
despite popular beliefs to the contrary, private citizen donors and industry does indeed understand the value of basic research, and may even be better positioned to value it properly as a whole, despite the private interests of them on an individual basis.
But I come from a background in small-lab experimental physics, and my spouse from synthetic chemistry. In both cases, a model of planning, followed by execution, followed by analysis, doesn't work, for a number of reasons. Often, an experiment fails, over and over again, until the design and operating conditions are refined to the point where the data begin to make sense. In my experiment, the equipment didn't even exist until I built it. And in experiments such as mine and my spouse's, the researcher (grad students) are also developing and refining their own abilities as they progress. I was my own electrical, mechanical, and software engineer. Sometimes, preliminary results change the direction of the project.
An additional issue is that experiments are rarely documented well enough to hand the data off to another team to analyze, without significant back-and-forth.
What little I understand about the "agile" model, seems more applicable to this kind of science.
It seems to me that our model of "collect the data and analyze it yourself" is a sort of "ten engineer startup" scale process. Now that many fields of science have four or five digits of PhDs collaborating between countries, there's an increasing need for specialization, specifically in creating reusable data. It'll make us less efficient on a small scale, but creating any artifact at all that can be reliably used by ten thousand people is much higher leverage than a fully packaged data+conclusion that's even odds to be nothing but noise.
- Making public funded research publicly available (since it was paid with public funds): papers, etc. for anybody to access, legally, for free.
- Lowering the cost of textbooks.
With more people having access to the materials we might realize there are a lot of latent geniuses out there in need of a chance. The easier to get access to those papers and materials the easier it is to double check the results, too, and to cross-pollinate between fields.
---
(PubMed, arXiv and biorXiv help a bit with that, but it's still not enough.)
If you are one of the researchers who is in the experimental group with lifetime capped funding (or at least a decade of capped funding, until the study is up), your career is going to be really impacted, isn't it?
I can't believe the OP suggested this with a straight face, it's obvious why it wouldn't fly, and why NIH was considering only doing it universally, to at least treat all researchers equally, right?
But yes, I understand the argument for how an experimental approach would help us determine the best funding models... although it's not really clear to me how this experiment works. Even assuming that we know how to measure which group did "better" after 10 years... wait, what are you expecting to see in the comparison between the research-capped group vs not in determining if it worked? I don't actually underestand this experiment.
Anyway, regardless, it's not a mystery why it's not going to happen, right? Experimenting with researchers careers.
If you believe, as I do, that a functional "science" is the most important force in making the future better than the past, would not this be a worthy sacrifice? In terms of its impact on all of humanity, now and forevermore. I would also think that true scientists would at least be the most likely to be sympathetic to this rationale.
Why don't you ask them if they would be willing to accept a cap on their salary, to be excluded from the most prestigious and well-paying institutions, and to do less research than they otherwise would, for a period of ten years, because they will be participating in an experiment which, if it was designed and executed well, might result in findings that might help improve research funding.
Perhaps very few actually practicing scientists are "true scientists".
To encourage cross-disciplinary research, I would award the top 1% of projects non-randomly but only if the PI has not been previously funded by the granting agency.
The problem is not enough “gatekeeping”. Accepting a publication should be more stringent.
Trial pre-registration and open science are good first steps though.
This reminds me of the welfare debate. On one hand, we could erect a giant bureaucracy to take a fine tooth comb over every application to make sure only the “worthy” get aid, which creates holes through which needy people fall. Or we can accept that driving fraud to 0 is impossible and get rid of the bureaucracy, then put the savings toward more aid. Sure some unworthy people will get it, but others who are worthy (and otherwise wouldn't have) will, and that’s more important.
This is not like welfare. We are trying to enact a stringent gatekeeper so studies are completely reliable. The analogy doesn’t work here.
I don't think the typical person who deeply loves science becomes happy in finance of big business. On the other hand, the people who go to the latter don't have the deep love for research that is necessary to bear an academic career.
Also, in my experience (I know some examples), the typical hiring processes of finance and big business weed out the bright people who have too much of a love for science.
So, I don't think what you mentioned is a problem.
While expertise in science is a valuable perspective to have when discussing the philosophy of science and how to improve the public's trust in science, it's not strictly necessary. We're reimagining how to fund science to achieve multiple objectives, not just scientific objectives. For instance, there's a misleading impression that scientists are only after grant money and that biases them. I say "misleading" and not "wrong", because I think you know the skewed incentives in this process all too well, and this has resulted in some kinds of bias.
Some of the proposals discussed here, like grant lotteries, would entirely eliminate that argument against scientists while also solving a few other problems with the grant process itself. I expect many scientists haven't even considered the public view of their incentives when evaluating lottery grants though, and instead focus on whether it's effective in funding good science. Public trust in science is a serious issue though, so if a slightly suboptimal funding process can make scientist's motives unassailable, that could be a justifiable trade-off.
The system is working as intended in many ways. People just aren't aware who the meaningful stakeholders are. It's not the PhD, it's not the taxpayer, and it certainly isn't "science."
There is just so much we don't know, every time someone is profiting real nicely.
And so much we do know every time someone is profiting real nicely.
The author's bio says:
José Luis Ricón is a book reviewer and blogger on various topics including longevity and a roadmap for the future of science at Nintil. You can follow him on Twitter here.
This is certainly ad hominem but is there a reason to think Mr. Ricón has any insights, experience, or publications that make him worth listening to?
I'd argue that to make science what it used to be, one would even have to cut funding.
Scientists used to be aristocrats interested in truth. These days they're just interested in grants.
I'm not sure if that's actually what you're advocating, but personally I'm very much against science only being the domain of rich people who can afford to do it as a hobby.
I also believe a significant amount of science should be democratically driven.
Generally it's the kids of rich parents.
Ladies and gentlemen, to add insult to injury, the greedy scientist myth strikes again.
There is not a better feeling in the world than to work for free for several years, six days a week (plus Sundays of course), with the weak promise of receiving a "turtle grant" someday. Maybe. (Tomorrow perhaps, the turtle is blocked somewhere. You will be paid in two years) and then being called "greedy".
Yes, there are people getting rich with the money for science, It just happens that they aren't the scientists. They are the politicians that keep the grants as their own personal loans. Science can't be fixed because is the guarantor of politicians that only need to go creative and put impossible requirements year after year to be allowed to keep most of the money for themselves.
In 2010 under the president Zapatero, the spanish government de-funded first and then keep for themselves the 25% of the money compromised for science. In 2015 under president Rajoy the government keep the 48% of the funds allocated for science and in 2016 the 62% of the promised money never reached any researcher. The money was used instead for, who knows... a new swimming pool for each minister maybe, or fixing accounting holes in other sectors... I wouldn't discard cocaine parties neither.
https://elpais.com/diario/2011/04/02/sociedad/1301695203_850...
https://elpais.com/elpais/2017/10/04/ciencia/1507133529_8680...
Just because there exists valid criticism of a thing, does not mean all criticism of it are valid: and in my experience, I mainly roll my eyes in exasperation when people criticize science for unfair reasons.
What the public thinks is broken in science is very often not what experts think is broken in science.
But in this case the valid criticism is just being fully acknowledged, rather than falsely extending the validity to all criticism.
In which case exactly? The comment I replied to was referring to unspecified instances.
Do you know this podcast decoding the guru, it is funny at times in breaking down internet persona like Jordan Peterson and Eric Weinstein but what annoys me about it though is this "researchers" talking about fringe internet phenomena. Sure you can find your cooks on the internet but maybe they should point their sharp criticism at their own sometimes and draw conclusions from it, as in stop funding it.
What do you call people that disagree to agree on something ... a research community. For all the good science I know I can equally find you the most asinine navel gazing triviality repackaged and sold as groundbreaking progress. Please don't get me started on what experts in innovation management ISO 56002:2019 can do to help valorise all this knowledge as they piss on us from from the top floor of their ivory tower.
I never suggested that it did.
You sound very frustrated, my friend.