Hacker News Reader

Comment by gwerbret | Hacker News Reader

I'm going to make what is probably a controversial statement, which is that, in my experience, most meta-analyses are trash.

A meta-analysis is usually grounded on the assumption that an aggregation of multiple, well-conducted studies can provide significant information that may not be evident by taking each study individually. Essential to this assumption is the premise that those studies included in the meta-analysis be scientifically rigorous in themselves. Unfortunately, this is almost never the case. The great utility of the meta-analysis in the clinical literature is such that there have emerged a series of standards for how to conduct such analyses, such as controlling for bias, controlling for variability, etc. These seldom include actual assessment of the scientific quality of the studies, as this is harder to standardize. Consequently, there is a preponderance of meta-analyzes that come to conclusions that are simply not justified, because the studies themselves were not scientifically rigorous.

I believe this applies to the situation of ivermectin, and it just so happens that this is the thrust of the point raised in the Nature article cited here.

View on HN

It's best to understand that the vast majority of literature is just faulty. Either it's technically wrong (experimental mistakes), or missing information (left out a step in the methods), or the conclusions are not supported the (valid) experiment.

Learning how to filter literature is non-trivial and I don't think very many people do it successfully. I've found the best thing is to take a group of aggressive smart grad students and have them tear every paper apart until I'm convinced there are no basic technical errors, and the conclusion is supported by the evidence. Then and only then, would I care that the methods were incomplete because I'd be ready to run a replication experiment, and many of my replications failed because the methods section was not sufficient (missing steps, misleading instructions, challenging step).

> I've found the best thing is to take a group of aggressive smart grad students and have them tear every paper apart

That's a pretty passable simplification of meta-analysis studies.

Meta-analysis have a good place. But they also have known flaws/assumptions.

Often it is impractical to perform large studies. Partially by logistics and often by funding. But if there are lots of smaller studies you can aggregate the data to check for overall significant results. But mainly they are helpful to give a "survey" of the current research instead of having to link to 10 different studies and hope someone else sorts out what the data says.

But as you point out, there are two major flaws in the assumptions. First, that the scientific procedure is sound. Secondly, that the data is handled properly, and thus you can take the summary and back out the underlying data.

Unfortunately trying to fix the first is really, really hard.

The second is somewhat mitigatable. As the Nature article suggests, you could publish the underlying data (anonymized of course). This would help in two ways. First, the meta-analysis could check for confounding variables to control across all of the data. The second major one is it would help people spot fraudulent data.

However, as anyone who handles datasets knows, publishing and wrangling data into a useable state from multiple sources is a serious pain in the neck. Plus a lot of concerns about how de-anonymized the data would be. As we've known, with enough metadata it can be used to identify individuals. And publicly publishing dais data would definitely allow for some serious sleuthing work to be done.

The problem I saw with the Ivermectin metastudies (which I did a fair amount of research on given that people close to me had decided to employ it as a prophylactic against Covid) was that the people who did the metanalyses were trained to read the results rather than grapple with the methodology and statistics.

None of the study results were robust, but many of the study outcomes were positive. So they were counting up positive outcomes and saying that more often than not, it was evidence that Ivermectin was effective.

This is a good means of generating a hypothesis, but a bad means of generating a conclusion.

Most of these studies were not in any way controlled or comparable in their conditions, and some had a very low number of observations. It didn't seem plausible that the data could be aggregated together.

It will be very interesting to see the results of the larger RCTs underway. Regardless of the outcome there _has_ to be a lesson for at least some people who arrived at fervent conclusions about what a miracle/scam this particular drug is. Unfortunately it seems unlikely that it will be a lesson that endures.

In my mind, there is also underlying all this the idea that if a drug actually works, then there will be many doctors going on CNN to say that it totally works; that they're super excited by their patients recovering. In other words, the effect should be profound. If the effect is marginal, why bother, even if it is a real measurable effect in well conducted trials? I mean : that doctors observe that some therapy or drug works well is step 1, then step 2 is to conduct trials to prove conclusively that step 1 wasn't a statistical fluke. In this case we seem to have it backwards : we first observed that there is an effect in vitro, then ran trials which proved nothing much, then extracted noise from that data to prove whatever we wanted.

Determining effectiveness through the number or media reach of doctors talking about it is the opposite of science.

> ...then there will be many doctors going on CNN to say that it totally works...

I don't think we can trust "doctors" anymore.

There were a bunch of doctors (aka America's Frontline Doctors) that were trying to go around and say Ivermectin works. Same with HCQ. The "needle rape" doctor in Idaho said the vaccine is dangerous, but Ivermectin saves lives.

In Idaho, I assure you, doctor's are prescribing patients Ivermectin in the ICU. If it really did work Boise would be the groundswell of Ivermectin miracles, where people are walking out of the ICU, ending the crisis.

Unfortunately, that's not happening. ICU's are primarily being freed up from death now it seems like. ICU's are in the high 90's capacity.

https://idahocapitalsun.com/2021/09/22/today-in-idaho-hospit...

This might be true except for the fact that news outlets follow a script to push a narrative (and CNN is not exempted from this):

https://www.youtube.com/watch?v=ksb3KD6DfSI

I've published (and still am publishing) multiple meta analyses. IMO,meta analysis on aggregate data is an h-index gaming machine that can never increase the value of its constituents. The amount of missing info in aggregate data is such that it's _always_ impossible to know if the constituent studies really draw from the same population, which is the cornerstone of meta analysis. And that's very practical, because most authors (myself included) mainly use meta analysis as a means to build a career, and not at all for doing science. IMO, current clinical research has very little science left in it. That's a huge shame, and it's a real priority to make that change!

The federal government has specific guidance on how to meet the legal criteria for de-identifying clinical data. Once researchers meet that standard they're generally free to publish raw data, unless there's a stricter standard imposed by state law or some sort of institutional review board.

https://www.hhs.gov/hipaa/for-professionals/privacy/special-...

What do you think about the ability for researches to pick & choose which papers to include in a meta-analysis?

Double-blind is important for a reason, researchers are exceedingly good at proving what they want to prove even if subconscious, can meta-analysis studies ever overcome this?

Do researchers ever pre-register their studies when they do meta analyses? Pre-registration makes sense for the timing of doing actual studies, but not as much in the case of meta analyses (which presumably happen on a much faster timeline).

But given what we've seen with p-hacking, it seems like it could be good if people pre-registered their meta analyses before crunching the numbers.

For systematic reviews there are one of two ways that I'm aware of: PROSPERO [1] and Cochrane[2]. Cochrane is slightly more rigorous for conducting a review of this type. Although you do declare what meta-analyses you intend to do, you can't be overly explicit as it is largely dependent on the number of studies that are returned in your search strategy. For example two studies is too small for a meta-analysis so you would do a narrative review (discuss what you found).

[1]https://www.crd.york.ac.uk/prospero/

[2]https://www.cochranelibrary.com

Sure some researchers do pre-register their meta analysis studies on clinicaltrials.gov. It's generally optional. Of course there's nothing stopping a dishonest researcher from doing a little secret p-hacking before pre-registering.

Good practices dictate that preregistration should always be done nowadays, including for meta-analysis. In practice though, this has very little impact. There is absolutely no guarantee that what's written in the paper is a faithful report of what really happened. There's nothing holding back dishonest (or, at that point I guess I should write 'standard') researcher from doing absolutely whatever they want.

how would you know when they actually started the analysis?

That's one of the reasons I was asking. Perhaps they could pre-register prior to being given the full data from other studies?

If the data is kind of public or going around in your circles you could easily hack this pre-registration constraint.

FiveThirtyEight weights its averages by pollster accuracy/methodology.

Next, we determine how much weight to give each poll in our average. First, polls conducted by pollsters with higher FiveThirtyEight pollster ratings — a letter grade measuring how accurate and methodologically sound pollsters are — are given more weight

https://fivethirtyeight.com/features/how-were-tracking-joe-b...

There is no equivalent feedback mechanism in medical research. Other than looking for retracted papers, we don't have a reliable way to quantify whether a particular author is "correct" more often than another.

The author’s Hindex could be an initial weighting factor

"Initial"? You haven't proposed any way to improve the weighting, so it would just be H index forever. We'd be baking in a cultural presumption that researchers who are more cited are more likely to be correct. A really dystopian prospect.

FiveThirtyEight forecasts are very questionable themselves. Been completely wrong on multiple counts.

Sure, but generally because the aggregated polls were systematically off. You may be thinking of other examples, but the 2016 US presidential election really wasn't that bad of a look for 538, they had Trump at reasonable odds going into election night. It's not like he was at 5%, 538 pegged him at 1/3 odds as polls opened if I remember correctly.

So is the weather forecast though. 10% chance of rain tomorrow means, yeah it could rain. Doesn't mean it isn't useful.

You can check rain forecast results thousands of times and estimate how often they are correct.

Assigning probabilities to events that happen infrequently (US pres. Elections, wars, end of the world...) is bullshit IMO.

It's on the aggregate of predictions that you can see the accuracy. If my weather app tells me hundred times there is 10% change of rain but it rains 50% of the time after 100 days, it's completely useless.

Same for FiveThirtyEight. They have been massively wrong on multiple counts.

538 is pretty explicit about what forecasts went wrong and how they will try and fix it in the future

FiveThirtyEight is literally owned by Disney - and should be treated as such. How is FTE determining their "pollster rating"?

> should be treated as such

> How is FTE determining their "pollster rating"

https://projects.fivethirtyeight.com/pollster-ratings/

https://fivethirtyeight.com/features/the-death-of-polling-is...

It's so easy to abuse meta-analysis, you simply modify the inclusion criteria until you get the results you're looking for.

So I'm someone who has published several meta-analyses of different forms, and written about meta-analysis as a topic.

I won't disagree with you that there are many poorly conducted meta-analyses. However, I think there's many well-done meta-analyses as well, and most importantly maybe meta-analyses aren't really different from anything else in life: some are good, some are bad, and many are in between.

One thing I've always argued is that meta-analyses have as a benefit a way of honing discussion around concrete specifics. The linked paper, for example, exists in part because there was a meta-analysis drawing attention to the literature at large. There's a decent chance that these studies would never be discussed if there wasn't a spotlight being pointed at the area.

With reviews, what happens is people pick and choose studies anyway, or don't, and then come to some subjective conclusion that's based on some unclear process. Meta-analysis makes all of this clear, and forces everyone to be absolutely explicit (or as explicit as can be) about how they're coming to their conclusions. If there's something wrong with it, then you can point to the specifics of that instead of going back and forth.

The problem with relying on definitive studies alone is that sometimes there will be more than one of them, or there won't be any definitive study, but many decently-done studies. Or the "definitive" study will have some controversial feature that doesn't clearly rule it out, but clouds the waters in a way that several smaller studies might draw attention to. Alternatively, there might be important heterogeneity across designs that illuminates moderating variables (like dose, or environmental context, or gender, or age, or whatever).

This paper is about meta-analysis of summary statistics, which to me is kind of bringing up a red herring. Statistically speaking if you can calculate the right summary statistics, the results should be the same as having the raw data. Issues about irregularities in results apply to raw as well as summary statistics; it also seems unrealistic to expect raw data in every case, and journals don't apply that standard either (that is, journals don't expect reviewers to reanalyze the data from scratch).

What's really needed is open data sharing, and scrutiny about studies that increases as the stakes of the results increase. I can speak to cases where I've been surprised at the state of the raw data, even in situations where the whole point of the study was to skeptically replicate a finding. Maybe for something like invermectin raw data analyses are appropriate. But it seems absurd to expect to throw out studies in the literature just because you don't have access to the raw data in every case.

Even meta-analysis conducted strictly according to PRISMA can be fraught with many issues. IMO, meta analysis is very vulnerable to bias in the base data and the honesty of multiple researchers, all while p-hacking and various other tricks to make papers sexier are already the norm. Forcing open data would certainly improve things but would not solve the problem of dishonesty in research, since authors would simply begin to withhold parts of their data that do not go their way.

Meta-analysis is a mirror of empirical estimates, really, in my opinion. It doesn't solve the underlying problems of research integrity or lack thereof, but doesn't create them either. It does allow you to model what is being reported though, to be explicit about your assumptions of estimates being reported in the field and how to go about aggregating them.

I agree about open data, for what it's worth, and even things like preregistration to some extent. There's a lot about dishonest research that has nothing to do with the statistics you use, or maybe even your design. It's sort of a never-ending arms race in that regard, and just as important to address the incentive structures that drive it in the first place.

I agree with your general point but in all fairness part of doing a meta-analysis is weighting the studies included in it based on many of the factors you mention.

More over, they have a bit of a raid 0 problem.

There more studies you include, the greater the chance you'll get a bad one. If you look at where these meta-analyses often fail, you'll often see just a few "Bad" studies that end up corrupting their results.

You'd think the statistical tools they bring to bear in these would be resistant to this, but it appears to not be the case in several recent cases.

I am also skeptical of meta-analyses.

I will re-post some thoughts I have previously shared from John P.A. Ioannidis who is a professor of medicine and thoughtful critic of medical research. He often raises good points about trends in research and research ethics. His view is that meta-analyses are mass produced, redundant, misleading, and conflicted [1]!

One criticism of meta-analyses in [1], using anti-depressants as a case study: "the results of several meta‐analytic evaluations that addressed the effectiveness of and/or tolerability for diverse antidepressants showed that their ranking of antidepressants was markedly different. These studies had been conducted by some of the best meta‐analysts in the world, all of them researchers with major contributions in the methods of meta‐analysis and extremely experienced in its conduct. However, among 12 considered drugs, paroxetine ranked anywhere from first to tenth best and sertraline ranked anywhere from second to tenth best."

I like this quote because it highlights the conflict of interest and misleading-ness(or at least reproducibility problems) with meta-analyses. Antidepressants have a huge amount of primary research dedicated to them. They also have the attention of researchers experienced in meta-analysis. Yet, meta-analyses do not agree with each other (and in fact they strongly disagree with each other).

[1] https://pubmed.ncbi.nlm.nih.gov/27620683/

> I will re-post some thoughts I have previously shared from John P.A. Ioannidis who is a professor of medicine and thoughtful critic of medical research

https://en.wikipedia.org/wiki/John_Ioannidis#COVID-19

> In an editorial on STAT published March 17, 2020, Ioannidis called the global response to the COVID-19 pandemic a "once-in-a-century evidence fiasco" and wrote that lockdowns were likely an overreaction to unreliable data.[14] He estimated that the coronavirus could cause 10,000 U.S. deaths if it infected 1% of the U.S. population, and argued that more data was needed to determine if the virus would spread more.[28][5][14] The virus in fact eventually infected far more people, and would cause more than 600,000 deaths in the U.S.[29][28][5] Marc Lipsitch, Director of the Center for Communicable Disease Dynamics at the Harvard T.H. Chan School of Public Health, objected to Ioannidis's characterization of the global response in a reply that was published on STAT the next day after Ioannidis's.[30]

> Ioannidis widely promoted a study of which he had been co-author, "COVID-19 Antibody Seroprevalence in Santa Clara County, California", released as a preprint on April 17, 2020. It asserted that Santa Clara County's number of infections was between 50 and 85 times higher than the official count, putting the virus's fatality rate as low as 0.1% to 0.2%.[n 1][32][29] Ioannidis concluded from the study that the coronavirus is "not the apocalyptic problem we thought".[33] The message found favor with right-wing media outlets, but the paper drew criticism from a number of epidemiologists who said its testing was inaccurate and its methods were sloppy.

Okay then.

Nothing like spending a career picking apart people's research and then generating absolutely garbage research outside your field of expertise, that is widely criticized by people who are actually the experts in that field...as being inaccurate and sloppy.

COVID hit, dude went all Don Quixote seeing conspiracies everywhere, and then generated a paper that suited his personal biases...

Your comment is the worst kind of ad hominem. You're simply dismissing one of the most-cited scientists in the world because he wrote something you disagree with.

He was right about the IFR for Covid-19, by the way. Subsequent research has upheld the finding. The primary factor that influences average IFR is the age of the population you're looking at:

https://onlinelibrary.wiley.com/doi/10.1111/eci.13554

> All systematic evaluations of seroprevalence data converge that SARS-CoV-2 infection is widely spread globally. Acknowledging residual uncertainties, the available evidence suggests average global IFR of ~0.15% and ~1.5-2.0 billion infections by February 2021 with substantial differences in IFR and in infection spread across continents, countries and locations.

https://link.springer.com/article/10.1007/s10654-020-00698-1

> The estimated age-specific IFR is very low for children and younger adults (e.g., 0.002% at age 10 and 0.01% at age 25) but increases progressively to 0.4% at age 55, 1.4% at age 65, 4.6% at age 75, and 15% at age 85. Moreover, our results indicate that about 90% of the variation in population IFR across geographical locations reflects differences in the age composition of the population and the extent to which relatively vulnerable age groups were exposed to the virus.

Great with some links to information about the actual numbers of fatalities and age brackets.

It is also interesting to note the effects of simple things like vitamin-d which reduce the death rate (and sickness impact) https://www.sciencedirect.com/science/article/pii/S156757692...

Diabetes and other lifestyle related illnesses also impact the death rate to a high degree.

If you are worried about ad hominems then you should also be worried about Ioannidis, who has taken to Twitter to criticize grad students with orthodox conclusions about COVID for their physical appearance and lack of publications relative to full professors.

I'll go with the CDC data, which pegs it at about 0.6%:

https://www.cdc.gov/coronavirus/2019-ncov/cases-updates/burd...

I was a fan of Ioannidis when he put out the estimates, but he has lost me since then. I'd rather go with the massive amount of researchers that have built their career studying these things, than the guy from a different field that feels he is better than everyone.

I mean, if your networks ops people are telling you the cause of a problem in your web application, and a manager comes in from another team and says he ran tcpdump and he thinks they are all wrong, and then the networks ops people show what he did wrong with tcpdump, but the manager sticks to his guns, who are you going to believe?

The man ran around screaming about how COVID wasn't a big deal and thus we needed to stop with the lockdowns and whatnot.

As our ERs, ICUs, and morgues were overwhelmed.

Which is still repeatedly happening, in regions with low vaccination and poor pandemic health regulations/orders. Except now it's lots of kids too young to get the vaccine or below age of being able to get the vaccine on their own without parental approval.

It would be exceedingly kind to say his "this isn't a big deal, not that many people percentage-wise are dying"...was at best incredibly heartless.

Ad hominem would be "he's a covid-denier, his research is worthless."

What I actually said was that he was an expert in research quality, and then promptly helped generate very shoddy research that was outside his area of expertise and ended up widely criticized by people actually in that field. Pointing that out is not ad hominem.

Extremely early-on in the pandemic he ran around shouting with no evidence that COVID was not very infectious and not deadly at all, and really, we needed to stop with these silly lockdowns and social distancing and masks and so on.

He then changed his tune to claim that actually covid was very infectious but this meant that the death rate was very low (and thus we needed to stop with these silly lockdowns and social distancing and masks and so on.)

Even his claim of very high infection / low mortality rate were true, it doesn't change the fact that ERs and ICUs have been repeatedly swamped, predominantly in countries, states and counties where political leaders are not employing standard pandemic control measures.

The man was literally shouting "THIS IS NO BIG DEAL STOP LIMITING OUR FREEDOM" while ERs and ICUs filled to the gills with dying people and in many places they literally could not burn the bodies fast enough.

His early claims were junk in part because COVID often was not mentioned or listed as cause of death, especially early in the pandemic.

PS:

> You're simply dismissing one of the most-cited scientists in the world

Appeal to authority. See? I can play that too. Maybe you can fool the average HNer into thinking citations imply credibility, but absolute junk research can end up highly cited because it was junk and many researchers sought to validate said research or pointed out its obvious flaws.

Next time, try arguing this stuff with someone who didn't work in a lab that was doing linguistic analysis of research, a project that existed because citation counts are completely worthless for identify novel research or evaluating its validity. And to head it off at the pass: yeeeeeees, even if you include journal impact factor.

Thanks for posting this. I did not know about about this darker side to Ioannidis.

I think lockdowns are/were a good thing. I won't necessarily fault Ioannidis for comments on lock downs on March 17, 2020 because that's pretty early on in. However, the April 17, 2020 pre-print story is pretty damning.

Reply on news.ycombinator.com