We have an epidemic of deeply flawed meta-analyses, says John Ioannidis
retractionwatch.com
retractionwatch.com
[0] http://slatestarcodex.com/2014/04/28/the-control-group-is-ou...
Are you critising the man in the article ( John Ioannidis ) for egotism because he somehow is disparaging of Further ( Meta ) analysis of his results ?
Or are you saying that Ioannidis has a point and that studies of studies are poor ?
I read SSC, and like it, but an editor would improve him. I think his superfans are very entertained by his lame jokes, and like his particular voice, so extra length is more of what they want, even if it doesn't contain much new information. For me, its OK, but if the essays could be 20% shorter with the about the same content, it would bump him up in my list.
I haven't read much SSC, but something tells me I'd like his lame jokes.
Common features of these conceptual papers include: - coming from a high profile lab, in a ivy league school (i.e. high schmoozing factor); - said lab is typically the best equipped to actually tackle that concept; - not supported by any new data, but piggy backing on others previous data from others; - some form of deductive reasoning; - reads more like a novel; - claims for openness and future collaboration (but remember the 2nd feature about being in the lab best equipped to tackle the concept).
At best these papers can be used to study informal vs formal logic. In reality, I find these papers being used as a form of first-to-publish or lobbying material to influence NIH funding. I think they work well as a good PR factor, and this not surprising since since the appearance of doing something has more value nowadays than being the one who actually moves the needle. So my conclusion would be the same as Prof Ionnadis, that these serve primarily as self-promoting and marketing tools. They lack the rigor and quality that we should expect from scientific journals, and should not be labeled as such.
And if this is the case, then what's wrong with "some form of deductive reasoning", as you put it? It would be great if more people applied the formal rules of deduction to their conceptual papers, and the work produced this way would not be less objective, less rigorous, or of lesser quality, than work that gathers some data and crunches the numbers over it, then calls it a day- quite the contrary.
If that is not the done thing then the problem is not with the conceptual papers per se, but rather with the way they are produced and one should advocate for better methodology, not different methods.
Sometimes it's not what you say but how you say it.
If I say that "I believe there is an invisible world all around us" that's sort of the same as saying that most of the mass of the universe doesn't relfect light and so cannot be seen, as in Dark Matter- except a) I'm not a physicist and b) I haven't got a clue.
Or, say, you can get to the right figure with the wrong calculations. Say, 5 + 5 = 8 / 2 = 4 * 3 = 10. Whut?
So maybe you had the right idea but couldn't explain why it was the right idea and people downvoted you? Because if you can't explain why something that sounds far fetched isn't, people will tend to think you're talking nonsense. And rightly so, and so should they, as should you too.
Vox Day (no endorsement) makes the distinction between scientistry and scientody, which I think is an increasingly useful one.
There's also some flaws in his analysis: https://www.ncbi.nlm.nih.gov/pubmed/27620683#cm27620683_2695...
Replication is a rigorous form of independent verification.
Should we replicate all that work? Or perhaps just obtain the raw, patient-level data from said company and analyze the data ourselves?
I am just curious which you view as more efficacious?
If you want to be a statistician of increasingly obvious limited social utility, you rerun the statistics.
Lets take a typical cancer drug. It goes through safty, dosing, and efficiacy clinical trials. In lucky cases, it will show efficacy in a multi-year clinical trial across many geographic locations with hundreds of patients.
If the trial is successful, the FDA will eventually approve the drug, the drug can now be prescribed. The FDA mandates that follow up study is done continuously to learn more about the drug; better indications for use, contraindications for when it won't work, etc.
Where does "replication" come into any of this? Why in the world would somebody replicate a dosing study and generate the same data? That would be unethical, dangerous, and counterproductive. At best, one would throw out bad data that was improperly collected. At worst, one would just abandon the drug and move on to a different candidate drug.
When it comes to human studies that come at real cost to human life, not using all the best available information is unscientific, unethical, should result in civil penalties, and should probably result in criminal penalties as well.
This is the situation that the parent was asking about; by making short blanket statements about what a scientist is and is not, without considering the real issues at hand, makes it seem like you're not engaging the issue.
"Scientists replicate data" is a simple thing to say if you're looking at stars or running a particle collider or working on a new synthetic compound; taking that simple minded attitude is not appropriate for much of the most expensive research out there.
The question of what to replicate and when is a difficult one; it's the tradeoff between new discovery and making sure you're on the right path. If you can make a new discovery that simultaneously proves or disproves that you're on the right path, that's a smarter move, but it's not "replication."
Not to mention the countless Institutional, Human Subject, and ethics review boards that must be satisfied before we can even begin to think about laying hands on a human to conduct a study of any sort - let alone one with an investigational new drug.
They are only the gold-standard for systematic reviews of therapeutic interventions.
Data can be faked, manipulated, unless you can trace it back to its origin and ensure it was recorded properly. There are multiple cases (even recent ones) where investigators modified the results of their trials in order to make them look better than they were.
Only replication can alleviate that concern.
I agree that replication will always be better and should ideally be mandatory. But meta-review is still way better than nothing.
And sometimes you just can't replicate for various reasons.
The purpose of meta-analysis goes beyond pooling. Indeed, one of the first steps in meta-analysis is "Is pooling even a good idea?"
It most definitely is not. Meta-analysis performs aggregation of existing research data in an attempt to validate specific research results for a larger group than initially tested. As such, it is primarily an inductive proof. That it can also validate or invalidate prior research is more of an artifact than a goal.
A rigorous independent verification can be performed deductively. No inference required.
Both are methods to reveal "Something is wrong here..."
No. The increase is because it is cheaper to do meta-analysis that it is to design and conduct experiments. They also carry less reputational risk.
1. They make excellent student projects. Part of this is cheapness, sure, but part of it is that meta-analysis can be done relatively quickly. Some observational studies will take years to complete - in the meantime, your Masters student needs something to do.
2. They are often "Step 1" of a number of study designs. For example, if one is eliciting priors for a Bayesian analysis, or in my case trying to parameterize a theoretical model, "Is there a meta-analysis on this, and if not, can we do one?" is one of the first questions asked.
3. It allows participation in a field. For example, I have thoughts about some aspects of clinical medicine. I am unlikely to ever run a clinical trial, what with not having a position in a medical school. I can however perform a meta-analysis of trial data as well (or possibly better) than the people performing the studies. Running a study and conducting a meta-analysis are not necessarily the same skill set.
Cost is a nefarious reason?