How Should We Critique Research?
gwern.net
gwern.net
It's so absurd that it has a certain Monty Python flavour twist of comedy.
Yes, the reanalysis is terrible and it follows from the fact that the base methodology is unsound.
In your particular case, I will not contest that the reanalysis was terrible. My point was that reanalysis of a paper after an issue is raised is not automatically also crap. It may indeed be correcting the issue.
In this particular case, there were multiple outcomes tested (read: tens of them) and alpha was .05
However, in a wider view the issue is far bigger than that: for a clinician nowadays it's become basically impossible to both be a really honest researcher while acquiring sufficient publication velocity to rise above. In a word, our system selects for bad actors.
"Which car company do you work for?"
"...a major one."
A clear problem I've seen when meta-reviewing peer reviews is that the goal seems to be demonstrating collegial constructivity, in the sense that things that are obviously totally wrong and yield a useless paper will be highlighted as sub-optimal or skipped over, in favour of smaller things that are fixable. That's understandable from the perspective of keeping friends and building influence with peers, all of which are critical to academic success, but it's pretty fatal for wider trust in science.
Another problem is that peer reviews are extremely slow. Several months on average. If someone takes 3 months to review a paper, and then addressing the comments takes another month, there is a lot of pressure in the system to go straight to publication at that point - in a strict code review environment the reviewer might observe that some of the problems still weren't fixed and send bad work back four or five times, often easily more. But that's because code review has a fast turnaround time. You can't do that with the speed of scientific peer reviews.
On the plus side the reviews do tend to be pretty comprehensive and seem to be taken seriously within their own standards. Part of why they take so long is they're often pretty long. The issue isn't scientists half-assing it, the problems are deeper and more psychological.
I have the exact opposite experience/ problem. Whenever I'm asked to review, there s an artificially short deadline involved (typically a couple of weeks).
Which, assuming you actually work for a living, effectively equates to having a couple of days at the weekend to review a paper.
From the outside perspective, it appears that a lot of people do "open-ended" experiments, write up some findings and pretty much no one tries to confirm the results with the notable exception of actual industrial applications (e.g., pharmacy or aircraft design), where the validation happens out of necessity.
It is not true that no one tries to confirm the results. The labs do further studies internally to follow up. Also those working closely in the field in other monitor these "findings" and try to align them in their own research whether directly comparing them or indirectly using those ideas to support or reject their own observation.
It is never one study but years of past research that lays the foundation for a scientific discovery that finally creates a paradigm shift in our understanding of the field.
Reproducibility and other forms of validity are not currently factored in very well.
Almost no scientific discipline requires it, sadly. There's usually no career prospects if you do replication studies.
Replicability doesn’t mean the study was right. If a study doesn’t replicate then it’s almost certainly nonsense. If it does replicate that just means that it’s self consistent—but if it was garbage in, then it will invariably be garbage out.
For some fields, especially natural sciences, this statement makes sense. However, for sciences of the artificial (to use Herb Simon's term) I would respectfully disagree.
For example, sometimes the underlying context has changed. I remember a luminary in my field (HCI) once argued that we should consider re-doing many studies every decade, because each cohort uses a different set of tools and has a different set of experiences, and because the underlying demographics of those cohorts change.
Conversely, if your method fails to drive the desired outcome, your hypothesis could still be correct, just incomplete. Maybe your perturbation of the chemical reaction didn't quite reach the activation energy. Or maybe other essential components in the mechanism were overlooked, given the set of conditions you happened to explore.
Complex black box systems like the brain are notoriously perverse in reproducibly giving up their secrets, even when your hypothesis is correct and your method robust.
Disagree. A significant finding is expected to occur due to chance in direct proportion to the surface area for such possibilities. More studies, more forking paths within those studies, more models, all increase the frequency of spurious findings. So one can do everything perfectly and still get “garbage out”. It is certain.
However, when I said “doesn’t replicate” I didn’t have a single attempt with a 5% chance of failure in mind. I had a field’s aggregate attempt to confirm the result in mind, which would include multiple attempts and checking for statistical bullshit and all that.
Under those conditions the chances are vanishingly small of a whole field getting massively unlucky when trying to replicate a well-done study that theoretically should replicate.
That’s what I had in mind, and I still think it’s right.
—
Rereading what you wrote, a different interpretation of what you said is that the original investigators might have done everything perfectly, and nevertheless found a significant result that was spurious just because that stuff can happen by chance. If that’s what you meant, I don’t understand the disagreement, except maybe semantically. I would call a perfectly done study that shows a spurious result “nonsense,” and I would expect replication attempts to show the result is nonsense, even if the process that generated the nonsense was perfect. Maybe you’re just saying you wouldn’t call a perfectly done study “nonsense,” regardless of the outcome?
One could argue that it's a failure of not sufficiently detailed descriptions of the experiment, but it is how it is in different disciplines. A relevant example for that is a bunch of earlier machine learning research (current methods subjectively seem more robust in this regard) some of which were very difficult to replicate from scratch because it was very finicky and relied on many tiny details that simply can't all fit in the couple pages of a standard paper, but it was definitely not nonsense, because it could be replicated if you knew all the best practices from previous experience. I mean, providing code for a ML paper doesn't change whether the research results there are nonsense or not, but many people might have a very hard time replicating that research without the code.
In other terms of critique, Gelman talks about two components (which may appear to be in tension), one referencing the craft of research, and the other in terms of apparent novelty of effect. My interpretation is that a critique of research should factor in both the character of the bet and an assessment of the evaluation. I don't think big is better, I think incisive is better, but that broader impact (news worthiness) is conditioned on credible surprise.
[0] https://statmodeling.stat.columbia.edu/2014/08/01/scientific...
However, with something correct, there's very little to say about it.
Instead we get something that combines both so it is hard to tell where researchers beliefs come into play.
We're in an environment where people routinely discover flat out academic fraud, as in, scientists just made whole tables of data up out of thin air. Someone notices, informs the journal, and with 95% likelihood nothing happens. Or maybe the researcher is allowed to 'correct' their made up data. We're in an environment where you literally cannot take any number in a COVID-related paper at face value, even when there are multiple citations for it, because on checking citations routinely turn out to be fraudulent e.g. the cited paper doesn't actually contain the claimed data anywhere in it, or explicitly states the opposite of what's being claimed, or the number turns out to have been a hypothetical scenario but is being cited as a "widely believed" empirical fact, etc. We're in an environment where literature reviews argue that whilst very few public health models can ever be validated against reality, that's not a big deal and they should be used anyway.
Gwern criticises people who learn about logical fallacies and then go around over-criticising people for engaging in them. Yeah, sure, if you claim someone is taking bribes and they actually are then it's technically an ad hominem but still correct to say. Granted. But we are not suffering from an over-abundance of nitpicky fallacy-criticizers. Where are these people when you need them? COVID related research frequently contains or is completely built on circular logic! That's a pretty basic fallacy yet papers that engage in it manage to be written by teams of 20, sail through multiple peer reviews and appear in Nature or the BMJ. As far as I can tell the scientific institutions cannot reliably detect logical fallacies even when papers are dealing with things that should be entirely logical like data, maths, code, study design.
The notion that scientific criticism should be refined to focus on what really matters sounds completely reasonable in the abstract, and is probably an important discussion to have in some very specific fields (maybe genetics is one). But I'd worry that if people are arguing about p-hacking or lack of negative results, that means they're not arguing about the apparently legion researchers making "mistakes" that cannot plausibly be actual mistakes. Stuff that should be actually criminal needs to be fixed first, before worrying about mere low standards.
So, no, in fact I think we are suffering from an abundance of nitpickers. Science desperately needs more reviewers who can see the bigger pictures.
That said, is peer review really the place to dunk on a study because the whole goal is pointless? The work is done by that point, it's too late. It's the granting bodies that should be getting peer reviewed in that regard, but one of the root causes of the malaise in research is that the granting bodies appear to be entirely blind buyers. They care about dispersing money, not what they get out of the spending. If they didn't spend the money they'd be fired, so that's understandable. The core problem IMHO is the huge level of state funding of research. The buck stops at politicians but they are in no position to evaluate the quality of academic studies.
And it's not even politicians, it's grant committees too. For the usual reasons, they'd rather fund study #13 from a researcher than anything that is the slightest bit uncomfortable to them.