I've read a lot of public health papers in the past year, probably more than 50. Here are some of the problems that cropped up in papers that were used by governments to support policy decisions. Many of those policy decisions later turned out to be incorrect, like lockdowns and mask mandates, both of which easily count as drastic and neither were textbook. In fact pre-COVID pandemic control plans by the WHO explicitly recommended
against lockdowns.
By the way, be careful to separate the question of mask mandates from masks themselves. Mask mandates don't work: just look at case curves when mandates were introduced or removed and observe the lack of inflection points. If they worked people would have hundreds of examples by now of case curves which obviously inflected right after a mask mandate was changed, but no such graphs are ever cited because those inflections don't reliably happen. Texas provides a recent example (mandate removed, curve continues prior trend) but this problem was obvious from within a week of the first mandates being introduced. Look at [1] for some case graphs with mandate change dates drawn on them to see the problem.
Anyway, errors seen in public health papers:
1. Circular reasoning.
2. Invalid citations.
3. Programming errors in models.
4. Use of extremely out of date numbers.
5. Absurd or obviously invalid assumptions and results in models being ignored.
That's not a comprehensive list. Unfortunately these aren't rare problems. Virtually every public health paper I've read has had at least one of these issues, often multiple. Circular logic in particular is mind-numbingly common, to an extent I've never seen before. For example, a common "validation" technique for models is to compare them to other models and declare their outputs to be similar (e.g. [2] or [3]). It's almost unheard of to compare model outputs to actual observed data, probably because doing validation right would invalidate virtually all public health models (this problem was admitted in a 2012 paper [4]).
For a specific example of these problems see the paper by Flaxman et al from Imperial College London [5]. This paper argued that lockdowns work using a statistical model, but they actually don't work, so to get this result required a combination of:
1. Circular logic: the model took as a starting assumption that case curves could only be changed by government intervention. In other words the paper encoded its own conclusions in its assumptions. The assumption epidemics can only be affected by lockdowns has no rational basis given the long history of epidemics starting and ending naturally.
2. The paper included Sweden in its data set, which attracted attention because Sweden appears to prove that the models generating the counterfactual were wrong. It managed to conclude lockdowns worked despite this because it concluded Sweden was a freak coincidence with an only 1 in 2000 chance of existing at all; in the graph that showed the different per-country fudge factors the model was allowed to calculate, Sweden was simply hidden to obscure what had happened. The truth was discovered later by people who studied the tables of prior probabilities uploaded to GitHub.
3. The paper admitted half way through that its scenario was "illustrative only" and that "in reality" the results would be different.
There were other problems too, but none of them stopped the authors telling the press that lockdowns had "saved millions of lives" (i.e. the drastic policy pushed for by that very same research team). Nor did it stop international press agencies citing this paper in "fact checks".
After reading so many papers with really basic and blatant problems, it's hard not to conclude that open access is going to seriously damage academia's credibility. Being able to just download and read the output of academic scientists is a very new thing, and one of the few highlights of the time I've spent reading COVID research is that open access is real now: I've hardly ever hit paywalls. Only for old papers. Unfortunately open access is a double edged sword. Now we can all read what we're paying for and observe the dangerously low quality. The outcome will probably be a large expansion of the battle between "science believers" and "science skeptics". Up until now that has been mostly restricted to debates on climatology, but now I think it will widen considerably. There's just no way to read the literature and retain your confidence in academic science when so much of it is entirely un-scientific.
[1] https://rationalground.com/mask-charts/
[2] https://github.com/ptti/ptti/blob/master/README.md (see the paragraph starting with "Formalism-agnostic").
[3] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3001435/ ("There is agreement in the literature that comparing the results of different models provides important
evidence of validity and increases model credibility")
[4] ibid; ("few models in healthcare could ever be validated for predictive use. This, however, does not disqualify such models from being used as aids to decision making")
[5] https://nicholaslewis.org/did-lockdowns-really-save-3-millio...