> In the group that took the drug, 11% needed hospitalization or an extended ER stay, compared to 16% of those on dummy pills.
Is that really promising? Very close to margin of error if not within.
Is that really promising? Very close to margin of error if not within.
Or were you trying to make a statement about the statistical validity of the result, because the outcomes differ by 5%? Because that's not how statistics work. A well constructed study of sufficient size can provide detect arbitrarily small differences in outcomes--you just need a bigger study size, the smaller the difference you want to detect. You would learn about this any 101-level statistics course, and a lot more useful information, too.
I think a large issue within that hurdle is that a lot of people simply won't want to learn, or apply the knowledge after learning.
I see this kind of error pretty frequently, and it's usually not a failure of mathematics... Most often, the person has some pre-existing negative emotional reaction to the stated conclusion, and their brain searches for something to criticize, in order to soothe those negative emotions.
I could only speculate as to why the original poster doesn't like hearing that an existing drug shows promise in treating COVID. Maybe they're a big Ivermectin booster, and they don't like the idea of another drug threatening their favorite?
OP is asking other posters to do quite a bit of homework for them... As in, "Is this study big enough?" <<YES>> "OK, but how big does it need to be to prove X?" ... At a certain point, it seems clear that OP is not arguing in good faith, because they feel compelled to make the critical statements, first, without understanding the topix.
I would caution you to not extend sympathy to OP's cause, simply because you happen to share a critical eye for this study. There is a HUGE difference between being right for the right reason, vs the wrong reason... and that's how we get crap like Ivermectin for COVID.
One sample had 79-86 out of 750 hospitalized, and the other had 117-123 out of 750.
Does that seem sufficient to demonstrate statistical significance?
Edit: found the actual numbers in the study
79 [11%] of 741 vs 119 [16%] of 756
p value was 0.9972 and the power was 92.40% it's a significant result.
"Variant B’s conversion rate (11.47%) was 30.08% lower than variant A’s conversion rate (16.40%). You can be 99% confident that variant B will perform worse than variant A."
That seems problematic. Did they stop the study because they had achieved statistical significance?
Is science allowed to research only one substance for specific purpose at any time to sufficiently avoid p-hacking for you?
Also if I read correctly they got >0.60 not >0.05
To be more precise without being too formal:
If there truly were no difference between them, you could be 99% confident that you would not get a result like this.
One third is pretty big when looking at it from a group perspective, but not really that big from a personal perspective.
If someone told me that taking a pill would reduce my chances of hospitalization by 5%, I would certainly be looking closely at the side effects.
Larger Number of people in the sample is a good start. However, bias is also a human element. Therefore you need that study reproduced independently, multiple times.
Researchers don't like this because studies are expensive and hard, and their boss is telling them to publish and get funding. The system is broken.
A cheap, simple observational study like this is merely a first step. No reasonable expert is claiming that this study proves anything worth making medical policy changes over. This study's sole purpose is to establish whether it's worth investing in further studies, or if we can just shitcan the idea now. Subsequent studies will cost more money, and involve larger samples, better controls, and get a lot more scrutiny from peers.
Time and money are finite resources. You have to have some kind of system for deciding where to spend those resources.
This is how medical science works. It's got a ton of pitfalls, but we still do it this way because so far nobody has come up with a better alternative.
I have no position on this particular study, but in general you can make your coding as simple and objective as you like and it won't do much about experimental bias. Objective processing of bad data isn't better than subjective processing of bad data.
This guy wrote a very good essay about data quality: https://desystemize.substack.com/p/desystemize-1
The essay you linked is very good, but it's actually illustrating my point... The mouse study ran into bias problems because they had to go and directly measure something in nature, and then turn that into a number. That introduces several opportunities for error.
But this SSRI/COVID study isn't doing that. They're literally just looking at patient records, and counting hospitalization vs current SSRI usage. They're not picking up mice to count ticks... They're exporting records to a spreadsheet, and summing columns.
Now, these guys might have problems with the quality of the records they're relying on... Who knows whether the records are accurate or not. But that's a problem of data quality, not experimental bias.
You might say "Who cares? Either kind of error undermines the results, just the same!" But it definitely suggests that the other guy doesn't know what he's talking about... Because if he did understand stats, he would have used the right terminology.
You're kinda right about one thing, though... I am definitely talking down to OP, and a couple of other folks.
I'm frustrated. There are real problems in how science uses statistics, and it sounds like OP & co have heard about those problems. But the way they talk about this study, they're just throwing crap against the wall to have something to say. They don't really seem to understand the problems they're talking about, or how those problems apply to the study we're currently talking about.
I should change my attitude, and stop taking out my frustrations on these folks. If they're wrong, I'm probably not going to change their minds by insulting them.
> But this SSRI/COVID study isn't doing that. They're literally just looking at patient records, and counting hospitalization vs current SSRI usage. They're not picking up mice to count ticks... They're exporting records to a spreadsheet, and summing columns.
> Now, these guys might have problems with the quality of the records they're relying on... Who knows whether the records are accurate or not. But that's a problem of data quality, not experimental bias.
Well, no, I have to disagree with this analysis. The mouse study described in that essay didn't run into bias problems, and the reason it didn't is that it was counting the ticks itself. But other tick-counting studies did have bias problems (due to bad methodology), and the Lyme-disease studies based on the bad tick-counting studies had the same bias problems (because they were using the bad data). The bias doesn't go away when you wash it through two publications instead of one. It's not a different kind of problem -- or even a different instance of the same kind of problem! -- and there's no reason to give it a different name.
Just a nitpick: it was not an observational study, it was a true experiment, because they controlled the treatment: it was assigned by researchers at random. It is by definition not an observational study which have a distinctive feature that it doesn't control for a treatment.
I mean take Remdesivir - https://www.nejm.org/doi/full/10.1056/nejmoa2007764 - statistically significant yet some other studies found it had no benefits. At least I don't think it was a slam dunk or promising in real life as day the vaccines.
Different studies can show different results for a lot of reasons... Sometimes, it's just random chance. Other times, it's due to bad study design, or poor experimental controls, or deliberate P-hacking. There's no way to tell from the results of a single study.
But that's not what this study is for, anyway... This is a preliminary observational study to look for possible drug candidates that might impact COVID severity. It's only suggesting potential future directions of investigation, not advocating for immediately dosing everyone with SSRIs... Medical policy changes won't happen until a much larger body of evidence has been established.
It's the nature of science that not all experiments produce correct results. Sometimes, we get it wrong--but that's why we have to keep repeating experiments, and trying different variations, before we trust a conclusion.
In the meantime, though, medical decision makers will weigh the costs vs benefits of proceeding with an experimental treatment... If the risks are low, and the need is great, they'll probably let the door open to more experiments. Most people seem to think that's a pretty sensible way to do things.
I recommend you take some stats courses if this does not make sense.
The study reached its statistical significance goal, and thus the study was ended as it would be immoral beyond that point to continue to give people placebos when they knew the drug was working. I'd suggest reading up on the Tuskegee syphilis study for the ethics behind why studies are ended early once treatments have been statistically validated.
This... is not how it's supposed to work.
There was a single hypothesis being tested here.
The bit "waiting for the trial to conclude" actually means "determine if the drug actually has any effect or not".
No matter how much you are interested in saving people's lives, or any other moralistic fallacy that might pop up, you can only help them if you give them treatments that actually work. Cutting short a study is falling short of the most basic requirement to meet that goal.
So now you are faced with a conundrum. You made sure that it has an effect to the degree you wanted. Now the math says that you either might get even surer by continuing the trial, or save 30% people from hospitalisation.
Also in treatment group just one person died while in control it was 12.
So what's it going to be? Making little bit more sure or saving few lives?
With a small enough sample, math can show that a coin flip gives tails 4 out of 5 times, specially if it's stopped too soon.
Don't you agree that stopping an experiment once we get the result we were hoping for and are looking for from the start does raise doubts regarding the conclusion?
And for a group of 1500 people it tells you how much confidence can you have. And it was more confidence than they demanded before starting the trial.
> Don't you agree that stopping an experiment once we get the result we were hoping for and are looking for from the start does raise doubts regarding the conclusion?
If you have a novel disease that's 99% fatal. And then you test a treatment that results in 30% people surviving 99% fatal disease. And you gathered enough data that that math tells you is enough to confidently claim that it isn't just a fluke.
Does stopping the trial and giving the medicine to the half of patients that you were keeping as a control group makes people raise doubts about efficacy of the drug or does that just make you not be called a second doctor Mengele?
Did they stop fluvoxamine trial? Or do they just mention that the effect was so strong that they could?
> He also cautioned against “play the winner” designs that increase the number of patients being randomized to arms that show the best results in an interim analysis. “That is most often used in early phase trials, like a Phase I or early Phase II trial where you’re really trying to identify the best dose level; such trials tolerate bias pretty well,” Sietsema said. “But you wouldn’t use a play- the-winner model in a Phase III trial because the potential for bias could lead to a wrong decision and the FDA would object.”
That sums up my concerns pretty well. This is being presented as a phase III clinical trial demonstrating safety and efficacy. For the level of effect they are testing for, I don't see how an adaptive trial is appropriate here.
Though it's like... isn't the vaccine in the initial phases like "80+% of hospitalizations prevented"? I guess it's good to have all these options but would be nice to just get good vaccine coverage.
You’re right to be suspicious; data dredging 101.
Why isnt everyone having a cigarette each day to interfere with the ACE2 receptors?
I don't know what the point of my reply is other than to recommend to you to have some humility that you might not know stuff...