Hacker News Reader

Comment by gregwebs | Hacker News Reader

The original Remdesivir trial by the drug company was a high quality RCT (randomized control trial) - except that they stopped it early once they saw a statistically significant positive effect (even though the effect was somewhat underwhelming: just fewer days of recovery). To me this moving of the goal posts invalidates the quality of the RCT. This study from an independent body without a moving of the goal posts is a more trustworthy result.

With that being said, anti-virals often only work well when given very early (sometimes even as a preventative) and right now from looking at this study it doesn't seem like this was done. So this study may only be confirming what we already know about how anti-virals should be used (at a very early stage) but it will hopefully have the good effect of stopping improper usage.

View on HN

> The original Remdesivir trial by the drug company was a high quality RCT (randomized control trial) - except that they stopped it early once they saw a statistically significant positive effect (even though the effect was somewhat underwhelming: just a couple fewer days of illness). To me this moving of the goal posts invalidates the quality of the RCT.

Based on what I've seen discussed, ending trials early is a fairly common (though admittedly contentious) practice in medical RCTs when you're dealing with very serious illnesses, especially when there are no alternative effective treatments[1].

The issue is that there is a tradeoff between an abundance of caution to procure conclusive results and giving patients access to medicine which "clearly helps". In an ideal world you would run every RCT for several years to figure out any possible long term effects and accurately determine efficacy, but there are many circumstances where you could argue doing so (if it's clear the medicine works during the early stages of the trial) is more unethical than giving people medicine which shows some promise to work.

But it should be noted that such trials are still solid science (the trial was an RCT), it just means you cannot take them as evidence of long-term efficacy.

[1]: https://www.ahajournals.org/doi/full/10.1161/circheartfailur...

I disagree that ending trials early is solid science, or at least not _as_ solid as running them to completion. It makes it hard to know whether results are due to a treatment effect or just random fluctuations over time. Essentially, it allows for cherry-picking of significant results in a way that bypasses family-wise error correction.

You can quantify, perhaps by simulation, the amount that it's less solid.

What's balancing against that is continuing the trial means giving patients a placebo when you have evidence of a better treatment.

It's a really complex issue both scientifically and ethically.

The solution to this is changing the stopping criterion in the trial protocol description, so that the real confidence interval can be calculated for the trial _with_ stopping (which is very different from running the trial until the end).

Good point, I should have added prospectively to that.

But even then, more participants and more time for follow-ups is generally better all other things being equal (which they never are).

Things got extremely political around these experiments.

When 2 days ago Orban announced from nothing that they will provide Remdesivir to everyone, I knew that again they did something stupid...I want the experimental drug that Trump got, not some leftover drug with lots of side effects.

Now I know that it was not Orban who was screwed, but the whole EU.

As I said, ideally RCTs would run for as long as possible. But in meatspace we cannot ignore that if a treatment shows serious promise it is unethical to deprive control group patients of the treatment. If we ran RCTs for all CoVID-19 treatments for something like 3 years before using any treatment, it seems pretty apparent we'd be looking at a significantly higher death toll.

My point about it being solid science is not that it is a good (from a scientific perspective) to end trials early, merely that the early part of the trial was still an RCT and the data is just as solid as any other RCT which ran for the same period of time. From what I've seen, this is a contentious topic in academia because the ethics of this make it harder to argue that RCTs should always be run to completion (with no exceptions).

But any good RCT will have a stopping criteria established before the trial starts to eliminate ex-post-facto cherry-picking. I'm not familiar with the exact details of the Remdesivir trial and whether they had a reasonable stopping criteria defined beforehand -- I was speaking more broadly about the topic of ending RCTs early.

There's a well known pattern that studies conducted by a drug company are more likely to report positive results than studies conducted by a neutral party.

The study you links to uses death or heart failure as an endpoint. If there had been lower mortality or ICU admission halting the trial early would make sense. With the only improvement being quicker recovery the value proposition of the treatment isn't that great. From a practical standpoint remdesivir is expensive and had supply issues so halting the trial early doesn't mean every American is going to start getting treated with it.

I don't think the science is solid when a trial can pick from 2nd-tier endpoints and stop the trial once an improvement is found. It is quite possible that if the trial had gone on longer the one positive effect would be reduced or even go away entirely or that we would see a negative effect in a more important endpoint.

In urgent circumstances where the demand for hospital services are high, a twenty percent improvement in recovery times is the same as increased capacity.

Nit: It's not quite the same. If need for hospitalization is growing quickly, cycling out people 20% quicker might not affect overload dramatically.

Steady-state, sure, you can treat 25% more.

> With the only improvement being quicker recovery the value proposition of the treatment isn't that great.

It's $3k a patient. The five days difference in hospital stay that was implied costs a lot more than that, even if there is no mortality/morbidity benefit, and that's ignoring that we were worried about running out of hospital capacity.

> five days difference in hospital stay that was implied costs a lot more than that

In case of COVID, it's not just a cost issue, but also clearing the bed as soon as possible, to put a new patient in.

10% less hospital stay, means 10% more patients able to get help.

Read the whole comment. I said:

> ... and that's ignoring that we were worried about running out of hospital capacity.

Anyways--

> 10% less hospital stay, means 10% more patients able to get help.

This isn't true. It's only true at steady state, but if there's an exponential process you get a smaller benefit. (E.g. take the ridiculous case that 100% of patients show up on one day-- obviously a shorter stay doesn't improve the number you treat in that case).

We're talking about a real virus here, not a theoretical experiment in math. Currently, most of europe is having an exponential growth of new cases, so even if 100% of current cases show up on day one, we'll have more new cases tomorrow, and any way to speed up the process will save lives.

Yes, shortening the process saves lives. I don't disagree with that.

It's just worth noting that you get significantly less than a 1/.9 benefit in capacity when you shorten hospitalization times by 10%. It's a benefit, but it's only this ideal value when things are perfectly steady state. So sorry for the pedantry, but "10% less hospital stay, means 10% more patients able to get help."--- it's not the predicted 11% or even 10%.

That assumes the effect in the study is real and that there is no increase in mortality or ICU admission: unfortunately stopping the trial early casts doubt on that.

I'm speaking just to the specific point I replied to. That's why I couched it with "that was implied".

Secondary endpoints, decisions about when to end trials, etc, are all fraught with peril.

It's worth noting that neither SIMPLE nor the ACTT-1 Remdesivir studies moved from a mortality to a non-mortality endpoint. SIMPLE did have an endpoint change before any data was available, but it was a relatively minor, innocuous thing.

> The primary efficacy end point was clinical status assessed on day 14 on a 7-point ordinal scale consisting of the following categories: 1, death; 2, hospitalized, receiving invasive mechanical ventilation or ECMO; 3, hospitalized, receiving noninvasive ventilation or high-flow oxygen devices; 4, hospitalized, requiring low-flow supplemental oxygen; 5, hospitalized, not requiring supplemental oxygen but receiving ongoing medical care (related or not related to Covid-19); 6, hospitalized, requiring neither supplemental oxygen nor ongoing medical care (other than that specified in the protocol for remdesivir administration); and 7, not hospitalized (see Table S1 in the Supplementary Appendix, available at NEJM.org).

The fact remains that it is standard procedure to halt trials early when promising results appear, and people are dying waiting for a drug. This is not something that's controversial, and this is exactly what happened with the previous Remdesivir study.

Of course it's disappointing to see it may not be as effective as we previously thought. And of course analysis may yet reveal issues with either one or the other or both studies. That's how science works - facts, not heartfelt desires.

Nobody is dying. The only endpoint that improved was recovery days.

> Nobody is dying.

Perhaps we are talking about different things? I was addressing the current COVID-19 pandemic.

The trade off between giving access to medicine that clearly helps and testing its effect carefully isn't done very well if you stop early with only a minor effect size. As at that point you're more than likely to just be giving people access to medicine that is far from clearly helping and possibly not even vaguely effective.

What is the _reason_ for ending it early? Is it the cost? I imagine it must be high enough because then they're gonna have to fight with this very criticism later on: RCT ended prematurely, therefore it is not as reliable.

A question to you and grandparent post: what are the things that distinguish a high quality RCT from a poor one? The number of participants, the design study, etc.? Any links/resources that discuss this would be appreciated.

edit: ok, I found a paper on this: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3429837/#:~:tex....

I am super curious now about any discussion that reveals why the WHO study was less-than-stellar, and why the original RCT was considered high-quality.

The reasoning is that as you gain information, you also have a duty to the people in the control group to use the best available information to take care of their health. Once you gain "enough" information ("enough" being statistically defined) that the drug helps, each additional person you let languish in the control group (who is denied access to the drug) is a cost that must be weighed against the benefit of getting additional information. When the data is clear enough, the cost can exceed the benefit, and you stop early.

Typically you'd register a "stopping rule" before you start your trial: a good drug often will trigger the stopping rule, as it helped so much that we learned about its efficacy on a smaller N than originally planned.

There are many different stopping criteria, depending on the trial (in safety trials you'd typically stop because you've found evidence that the drug is unsafe and continuing would be unfair to the folks in the treatment group, whereas in efficacy trials after safety has been established, you'd stop because you've found evidence that the drug is effective and continuing would be unfair to the folks in the control group).

The reasoning is money, the unreasoning is ethical sophistry.

Its typically an ethical decision. In the event of a deadly disease ( cancer, covid, hiv etc ) its considered harmful to continue giving a less effective treatment once you have shown some superiority/inferiority.

It’s both ethical and statistical. If you observe a strong enough effect, you can reach significance with less data. When people are dying, sometimes you throw a Hail Mary.

if you don't think the quality of the trial is so bad as to outright impair approval (and right now the political pressure for a tool, any tool, to fight COVID is so intense that there's zero chance it wouldn't be) then why continue? It costs money to do trials and it represents lost revenue where you could be the sole seller of the only known drug to treat a deadly pandemic.

in fact there is a downside risk to continuing as well, further study might show that the early results were an outlier and the drug is not as effective as previously thought (or has some serious side effects in some situation), and then you lose all the money you've invested. But if you push it through, there is no risk of being sued since the US Government has waived any liability on the part of drug companies for these therapies.

in a game-theoretic sense there is absolutely no benefit to continuing the trials and potential downsides to doing so. Remember, for-profit companies don't care about maximizing public health, they care about maximizing revenue. Especially when you've explicitly removed all financial liability for being risky with public health.

To continue an RCT once it reveals strong evidence for benefit in the treatment group, you have to deliberately inflict suffering on the control group. You can't do that ethically.

Strong evidence for a benefit is often measured by death or something extremely severe.

When using other clinical endpoints cherry-picking is to be avoided. [1]

> For a trial to stop early for benefit, the observed effect on the composite end point ideally should be consistent in the individual components.

[1] https://www.ahajournals.org/doi/full/10.1161/circheartfailur...

That's true, but that's not the case here.

It showed no evidence for the strong benefit they were looking for (mortality), but a similar trial started in China before that trial, ended - and showed no change in mortality, but some shortening of the hospital stay (even though that wasn't its stated hypothesis). At that point, the western RCT remdesivir trial added the "hospital stay" target, and lo-and-behold, shortly afterwards, it turned out that it does result in no less mortality but shorter hospital stay as well.

While it does give some credence to the "shorter hospital stay" result -- having reached it in to Gilead initiated RCTs -- It was statistically significant but not dramatic;

And now another trial says even that's not so significant.

My statistics are a bit shaky but I understand they "pay" for taking an early look (interim analysis) by reducing the power of the trial by using something called alpha-spending function

I wish antivirals are trialed with a different end point: reduction in viral load. This can likely be done definitively and a lot faster as well. Deaths caused by ARDS may be triggered by a virus but it doesn't follow that clearing the virus would cure one of ARDS. We really need a pill form of the antiviral and give that to people much earlier in the disease course. A pill is possible in this case as a Remdesivir analog https://www.statnews.com/pharmalot/2020/08/24/nih-gilead-rem...

Lilly and Regeneron's monoclonal antibodies use exactly that as primary endpoint. Unfortunately, most of them are e.v., with exception of one formulation by Regeneron, which is subcutaneous.

Wow, that's a textbook example of post-hoc hypothesizing, about the biggest statistical no-no in the book.

Reply on news.ycombinator.com