"Many people also dropped out of the study. Only 96 of the 498 people recruited took part in the last round of testing. The researchers lost touch with 13 people, while 57 people died and 332 declined to take part."
Do studies like this often reach the front page of HN?
I tend to ascribe some value to "It was published in the peer-reviewed British Medical Journal."
Additional thought: Especially because it is essentially supporting the null hypothesis.
Well, one can't criticize the quality of the study without looking how well other studies did. But even if all studies involve a massive drop-out rate, the drop-out rate still seems like something to look at in evaluating the accuracy of the study.
I mean my concern, looking from the outside, isn't relative accuracy, but absolute accuracy. Not "is it as good as other studies?" but "it is good enough that I can rely on it?"
BTW, it would also be an issue if the subjects were compensated to the point that they differed from the actual population at large.
Fundamentally having more respect for doing things right, the right way, over the /long/ term, including the STEM fields, would be a 'good thing' for any society, but particularly western ones (where often the bad/evil antagonist is based on science or a cold lack of emotions; but the protagonist is based mostly on emotion and feeling instead of logic).
Everyone keeps replying as if my comments imply a demand that "something be done about this". That's not my point. Maybe nothing can be done about it. The situation is still significant or at least appears significant.
It's called longitudinal analysis or survival analysis, depending on the question being asked.
We statistician got test and models for these.
You can rely on it if the people who are conducting the experiment are competent and know what they're doing.
You absolutely can.
Scientific studies should be judged on their potential to produce interesting results. This study has too small a sample size to provide results that merit making even the weakest claims about. Even the initial sample size was, frankly, just barely large enough to maybe provide suggestions for future research.
If a meaningful drop-out rate is expected, I would argue that the researchers running these sorts of studies are mostly wasting everyone's time and money for the sake of publishing a few papers.
a rate of drop-out like that would certainly make me doubtful of the results of the study. It seems reasonably likely that people dropped-out in a fashion that would be weighted in one way or another.
I'm reminded of a study using the Nintendo Wii to help keep elderly people mobile. 90+% of them stopped playing immediately, within days, of the researchers leaving.
So you're saying it's logical that a lot of people dropped out? Seems reasonable.
But that doesn't mean that you wouldn't have the problem of the high drop-out rate implying the final people remaining are biased sample. When you are filtering a large number of people down to a small number, making sure that small number is representative of the large number is a hard problem.
Maybe there's no solution or no easy solution. Doesn't change the problem.
If that's the case, then the test should say this in the instructions... (It's a classic example of setting expectations, introducing objections up front..)
> This test may not feel exciting. Some questions could seem childish. Other questions could make people feel silly for marking the incorrect answer. These extremes are required for the study.
A reference: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3227332/
Sometimes, you have to trust the quality of the journal and biostatistician if you aren't able to read everything yourself.
Edit: Yes, they did. See ojbyrne's comment.
This is one of the major downsides to longitudinal studies— people are unpredictable.
.. I don't get it, why would you ask that?
(I'm tech lead for this site)