Biden proposes $250B investment in research
sciencemag.org
sciencemag.org
I've read a lot of research papers over the past year related to COVID. Coming from the computer science literature, it was a shock. Standards are systematically poor to non-existent. I've seen papers doing things so bad that my eyes popped out of their sockets. For example, papers that give citations for claims which turn out to not actually support the cited claim, or which specifically state the opposite. Numbers being cited as if they are empirically determined facts which turn out to be model outputs, or even worse, arbitrarily chosen model inputs! (i.e. pulled out of thin air). Circular logic is rampant. The most influential COVID model was written in C and riven with basic memory corruption bugs, typos in RNG constants and thread safety bugs. Its prediction for the number of UK deaths varied by 80,000 people with the same PRNG seeds simply due to coding errors, all blown off by scientists with absurd justifications like "we use a PRNG and average at the end so these bugs don't matter" or "if you held us to professional coding standards science would stop because we can't afford it".
The root cause of these problems is that science is flooded with government money and the government is an undiscerning buyer. The only mechanism science has to stop a race to the bottom is peer review, but peer review appears to be nearly useless:
1. It's far too slow. Average time for reviewing COVID papers = ~2 months, a major improvement on the 4 months a normal paper takes but still totally miscalibrated to what's needed. Almost every week universities announce a new scientific finding that on close inspection turns out to be obsolete on the day of publication, but it's extremely difficult to notice this because the paper doesn't state when the research was actually done.
2. Peer reviewers appear loathe to reject a paper completely even when the premise of the paper is nonsense, presumably as that wouldn't seem like friendly constructive feedback. They are only willing to nit-pick, which has value when the paper is basically 80% good and just needs some polishing. It's not enough when the study should never have been done to begin with. Alternatively, they'll point out 6 major problems, 3 will be fixed, and they'll sign off on the paper because now their "concerns have been addressed".
3. Peer reviewers are often engaged in the same bad practices and don't seem to see anything wrong with it. Peer review cannot stop a whole field circling the drain, as is clearly happening with a lot of public health research.
Fundamentally, academics are incentivised (by governments) to make very interesting claims with absolute confidence. They aren't incentivised to make true claims carefully. There are of course many sterling examples of scientists who rise above their incentives and do their work with honour, and these problems are not uniform across fields. Part of why bogus citations surprised me so much is that in 10 years of reading computer science research papers I've virtually never encountered those; in COVID papers practically every single paper I see has at least one.
A terrifying amount of human knowledge appears to be built on totally unsound foundations. An excellent essay was published in 2020 discussing the scale and extent of the problem of bad research. It's about social science rather than COVID research but if you're interested in the topic then there's half an hour of worthwhile reading to be had here:
http://fantasticanachronism.com/2020/09/11/whats-wrong-with-...
The author goes into how bogus citations seem to have become so frequent (scientists don't seem to actually read the papers they cite) and provides evidence derived from the propagation of typos in author names.
This is a fundamental problem of course - how do you evaluate what's really useful and worth it? That's what markets are for, but research is an area where governments feel justified in intervening. Whereas for e.g. food production they understand they don't need to and shouldn't.
My sense is that computer science is less afflicted because it's closer to industrial application, and there's a lot more movement between universities and industry. It's not worth trashing your reputation in industry by publishing obviously deceptive research because that could harm your future salary prospects, whereas if you're in a field with no commercial interest, publishing bad papers is fine because the ultimate consumers i.e. policy makers, journalists, voters, won't ever notice. They aren't reading the papers.
Academia and thus research definitely needs to be overhauled. You could probably get enough resource scarcity to push for optimal outcomes with just allowing former students to default on student loans if they can’t find work in a reasonable time.