Hacker News Reader

Comment by Gatsky | Hacker News Reader

These results from observational studies almost never replicate in randomized studies. It's because people who decide to take glucosamine (or not) also do lots of other confounding things, like visit the doctor more, or have higher salaries, or don't live with people who smoke etc. It's been shown many times you can't really adjust for these things. Epidemiologists love doing these studies, because they are comparatively easy. The hardest part is the data sharing agreement with whatever registry you are using. You don't even need to leave your desk to collect any data.

View on HN

The real question is: why is anything other than randomised studies allowed to be published in the first place?

Not questioning the need for RCTs, but they have their own issues. Like, let's say you do a RCT and it shows an effect, but then in a large epi observational study there isn't any? There would be some problems to work out.

RCTs are great but can have problems with generalizability to real-world conditions. Ideally you'd study both.

With a sufficiently robust RCT, I'd think that indicates an issue with the observation in the observational study -- i.e., there are some confounding factors in the real world -- rather than any sort of issue with whatever effect the RCT found.

Conversely, if we had a large observational study that was then contradicted by a robust RCT... well, I'd still be inclined to trust the RCT.

(At the same time, I think a good observational study is, by its nature, more applicable to one's own life. If an observational study suggests that $FOOBAR is good, then -- as long as there's also a solid RCT confirming that $FOOBAR is safe -- why not try out some $FOOBAR for yourself?)

I guess it's ok to publish uninteresting and probably false things. This could be a fairly accurate description of biomedical science to be honest. It's annoying when it ends up on HN for example, or in the health section of a tabloid newspaper.

In theory, these observational findings could suggest treatments that should be evaluated in a randomised trial. I am not actually sure this is true however, it is possible they are less useful than selecting interventions for further testing in some other way. Anecdotally, folk remedies may have a higher hit rate for selecting treatments for proper evaluation. This is often surprising - eg a precursor to aspirin is found in the Willow tree, which has been used as a medicine for thousands of years. Another example is the foxglove plant from which the heart drug digoxin is derived, which was used as a herbal medicine predating modern medicine.

The history of willow bark being used as a pain killer by ancient Europeans is... debatable.

https://en.m.wikipedia.org/wiki/History_of_aspirin#History_o...

https://theconversation.com/hippocrates-and-willow-bark-what...

https://www.galvnews.com/health/free/article_8c8b50e5-6712-5...

Observational studies are far cheaper and easier, and are actually very useful for a lot of situations.

Published, or publicized?

All science needs to be published. Pilot studies, observational studies, quasi-experimental studies. Otherwise, we don't have the information we need to create randomized control trials.

However, I'd strongly support keeping all that science from being publicized.

> However, I'd strongly support keeping all that science a secret from journalists.

What’s your plan for doing this?

Some kind of secret police force with balaclavas and chairs in dark rooms.

> The real question is: why is anything other than randomised studies allowed to be published in the first place?

Because things other than randomized studies produce the results that generate the funding for randomized studies.

One answer: anyone can start a journal, there are no laws or standards. Peer reviewers are busy and don't always have the background to criticize every aspect of a paper.

Another: not all research can or should have randomized studies (like it just wouldn't make sense if you were, say, determining the structure of a protein). So drawing the line as to what does and doesn't need a specific approach is blurry, and really, can only be enforced during the peer review process, or by the editor.

On the other hand, this kind of work can still serve as a hypothesis generator. Someone else will read this, think, "well they did this all wrong!" and perform a better study that clearly demonstrates whether it has an effect or not.

This is a good point. I was recently having a conversation with a non-scientist (I am not one either) about two different studies. One was an RCT that found no effect for a treatment, and the other was a "case control, test-negative" design study that found a beneficial effect. My friend said the latter was as good as an RCT because it also has a control arm.

But when I looked up what "case control, test-negative" means, I found that it is observational, and there is no intervention provided to any of the individuals. Some of the recent discussions of the design indicate that it is understood by scientists, but can mislead laypeople, who assume that its conclusions are much more robust than they actually are.

We should definitely continue to do research of many types, partly as a way to figure out where to spend the time/money to do RCTs (which are more expensive than other types of analysis). But we need people reporting on studies to be very clear up-front when they are not describing an RCT. They should say what the study's conclusion means, versus what it would mean if there were a similar 'finding' in an RCT. Otherwise people will not understand that they are being told a weak conclusion, not realizing there is a strong conclusion that has gone unmentioned.

Not everything can be done with a randomized controlled study. For some fields (quantum mechanics, volcanology, climate change, paleontology...) most studies can't be randomized, much less blind. This is also true of longitudinal studies: you can't fully control anyone's exercise and diet over months, much less decades.

Yet we want to do research in these areas. So we have to make do with what's possible.

True! But medicine just so happens to be one of those fields that lends itself well to RCT’s. Perhaps if the OP had reworded their statement to everything in medicine it might make more sense.

some medical trials. Surgeries, for example, are tough. And these longitudinal nutritional studies, as I mentioned, are another.

Because academics and primary consumers of these studies understand their uses and limitations. Not all science has to be groundbreaking new discoveries, simply trying to replicate an existing study or exact same methodology on different cohort is perfectly fine.

The problem is when it gets printed in mainstream press who sensationalize and then posted to reddit/hn where people go 'duh this is obvious, this is a dumb thing to study', 'sample too small', 'not controlling for income/confounding'.

You can't do randomized studies for things like nutrition if the question is "does eating red meat for a decade increase the risk of cancer". Nobody's going to comply with it.

With all the common sense replies explaining why, I guess this was not the real question at all.

Actually, you're wrong about that. According to this Cochrane systematic review, RCT's and observational studies are usually in close agreement: https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.MR...

Hmm, that would be interesting, except it seems they only considered 14 studies. They find in 11 of these observational studies match the results of RCTs.

Im sure this will be different in areas where there is no public awareness of a risk factor vs ones where there are. Eg healthy diets select for people who care about their health, so it's not so infomative, vs something where the link is not publicly known (so that the observational study is already somewhat blind/randomized).

Well no, I’m not wrong. That article you linked is a review of reviews comparing RCTs and observational studies for medical interventions, primarily surgical studies or pharmacological treatments with well characterised effects. Observational studies of supplements and nutritional interventions on the other hand have a very poor track record of RCT validation.

@Gatsky: ah I see, yes that sounds highly plausible, but can you supply any evidence of this, or is it just hearsay?

Such a review is necessarily limited to topics which have both an RCT and an observational study.

As such, the general bias against publishing non-significant results could mean the conditions for being able to even try such a review risk biasing the review towards exaggerating agreement.

(It also seems the Cochrane review here may itself only be considering other reviews - a metameta analysis of sorts – which might also be influenced by a selection bias, if there’s any chance at all that researchers choosing work, & then publication decisions, prefer to address questions where there’s more agreement, rather than less-attention-catching mixed results.)

The most convincing evidence here would be how strongly preregistered RCT results, whether ever published or not, tend to match earlier observational studies. It doesn’t appear to me that this Cochrane review is focused on that.

@gojomo: Isn't it normally the case that an RCT follows an observational study? There's not much point in doing an observational study if someone's already done an RCT, but there is the other way round, since RCT's are more trusted.

Wrt publication bias; an underestimation of the false positive rate of observational studies wrt RCTs could occur if non-significant RCTs are being held back at a higher rate than observational ones, but it seems more likely to me that it's the other way around since RCTs are generally more expensive and time consuming than observational studies, and more trusted, so it's a bigger loss if the result isn't published.

I don’t have a sense of the ‘normal’; it might vary by field.

For example, for COVID vaccines, we’re being inundated with observational data - often informal, sometimes in reviewed studies - long after initial approval RCTs.

Good advice in general but it’s worth reading to see what confounders they tried to correct for. There’s still information to be gleaned.

This part caught my eye:

> A stronger association between glucosamine use and decreased lung cancer risk was observed in participants with a family history of lung cancer when compared with those without a family history.

A family history of lung cancer could simply mean a family history of smoking, which was much more common a few generations ago than it is today.

This was done in Guangzhou where I lived for years until 2018. The incidence of smoking amongst males still higher - gut guess of 30-40% adult males are smokers in the city and province. So in their cohort one would expect to be seeing still living relatives of young adult smokers who are also smokers.

Interesting that it’s sold as supplement for joint and related tissue repair and yet first big science study with observable results is about cancer.

Other interesting point is that it’s study of cancer. And that is simply because China still has huge numbers of smokers and the health cost is stratospheric.

To me it is plausible that in a time of great social and economic change, the odds of behavioral confounds in observational studies might increase. Non-stationarity everywhere.

That said, a meaningful result here would be welcome.

Is it actually stratospheric though, when you take out QOL year adjustments? Other studies, such as this one in Finland[1], found that when you don’t take into account QOL adjustments for the smoker that smoking is actually a net benefit for society because the smokers die earlier and don’t draw on the very expensive late life healthcare.

1: https://bmjopen.bmj.com/content/2/6/e001678

You’re right that there is non-random allocation to the exposure (taking glucosamine), but this is something that can be addressed to an extent with a bit of statistics.

I was disappointed to see the authors didn’t take the additional step of doing propensity score matching or weighting to account for propensity to take glucosamine.

Here’s a good reference about propensity score matching: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3144483/

I wonder, does anyone know of a good guide for lay persons that explains how to understand scientific studies?

Even something that details the difference between a population study and randomized controlled trials.

Ben Goldacre's book 'Bad Pharma' has a good discussion around this, especially in terms of caveats and gotchas to watch out for with such studies.

It might be more 'lay person' than you are looking for, but it's an excellent read for myriad other reasons, so you wouldn't be wasting your time (IMO).

Thanks. I was hoping for something online, like a blog post/website, which is an easy way to spread the word to other people so they know how to contextualize these things.

I agree with that. You can see this in Ontario COVID statistics.

People that did not take the vaccine, despite all the pressure, tend to be healthier as population. They kind of self selected by being self confident in their health and not worrying about dying from COVID. The triple vaccinated in Ontario are now the highest risk of getting COVID. They too self selected, by tending to be older with pre existing health problems, and worried enough to get boosters.

https://covid-19.ontario.ca/data?fbclid=1

does that include the people who died after not taking the vaccine?

asking because you only linked to the homepage of a general data portal which itself reports its no longer being updated,

versus to a study supporting what you're saying

Last update was June 10. So it's still pretty recent. Ontario did not release data on death rates to the public. I don't know why. I briefly read on twitter that UK is showing startling differences now, with the vaccinated having a much higher all cause mortality rate. If true, I don't know if its this effect at play or if some of the conspiracy theories were right.

just reiterating in case you missed it, regardless of when it was last updated:

you actually only linked to the homepage of a general data portal (not even a specific data set),

versus to a study supporting what you're saying

I have to imagine this is a mistake in getting the specific URL you were actually looking at

You are right that people who take vaccines and who do not probably bifurcate along other lines as well. But you’re doing everything even more of a disservice by speculating the dimensions as if it’s established fact. Maybe the unvaccinated aren’t healthier they’re just liars? Also any self respecting study would control for the simple demographics like age so your hypothesis would be quite moot if the study was done with any rigor.

> The triple vaccinated in Ontario are now the highest risk of getting COVID

I don't understand how the source you shared supports this claim*. If anything, it shows the opposite - you're at far higher risk of death if you're unvaccinated. And that's after considering the fact that unvaccinated people are likely to be younger and fitter.

The unvaccinated have, at best, half as many deaths per 100k as those who are fully vaccinated + booster (in reality, the two lines mostly track each other). Which sounds good, until you realize that the unvaccinated make up only 9% of the population. 33% of the deaths coming from 9% of the population - healthier, they are not. Delusional, would be more correct.

* I'm assuming "getting COVID" here means "having serious consequences from an infection" as opposed to merely being infected. The vaccinated make up the vast majority of the population, so it's obvious that they are also at the highest risk of being infected by COVID. There are very few unvaccinated people left for the virus to infect.

The numbers have already been adjusted. So what you're seeing is the rate of covid cases per 100k people in each group. 3rd graph from the top. You're trying to adjust the numbers twice for the unvaccinated.

If you don't believe me, here is a second set of data from Walgreens. Look at the third page.

https://www.walgreens.com/businesssolutions/covid-19-index.j...

The Ontario page does not show the death rate.

The Ontario page shows a graph comparing death rates for various states of vaccination search for "Deaths involving COVID-19 by vaccination status".

> The numbers have already been adjusted

Thanks for the correction.

> The Ontario page does not show the death rate.

Doesn't it? I see a graph titled "Deaths involving COVID-19 by vaccination status". And that still shows higher deaths/100k in the "Not fully vaccinated" group for every given adult age range: 18-39, 40-59, and 60+ (look at the All time data).

> here is a second set of data from Walgreens. Look at the third page.

That shows the positivity rate. I'm not talking about the positivity rate. We're beyond that now. Everyone will get Covid someday. The vaccine protects you from dying or having serious health issues. There's no vaccine, for any disease, that can prevent you from contracting the disease. That would require vaccines to create force fields around your body, which are science fiction. In the case of the most efficacious vaccines, your immune system will be so well-prepared that your body will fight off the infection without you ever getting any symptoms.

I missed the "Deaths stats" as I did not see those before, the last time I looked.

So the last cited death rate is 0.01 vs 0.02 / per 100,000 people. So the vaccinated have 1 death per (100 * 100,000 = 10 million). Or 1 death per 10 million and the "not fully vaccinated" have 2 deaths per 10 million. To give context Ontario has a population of 14 million people. This seems like an easy relative win for the vaccine.

Except:

1) The numbers of deaths are so low, its kind of meaningless to extrapolate to the general population from them, because of sample bias effects.

One reason being that deaths could be coming from a very likely specific sub population. For example very old sick people already in hospital or nursing homes near death that contract the disease. Pretty much anything could kill them. It has no bearing on how a random person from the general population would react. You might have situation where for example there are people that chemo therapy failed, and they either refuse the vaccine (since they will die within weeks anyway) or might be so sick and too weak to take the vaccine and then contract covid as the last straw that breaks them.

So it would be very disingenuous to claim based on such small number that the vaccine lowered deaths in the general population. It would be like telling people in Hawaii to wear gloves to prevent frostbite, based on data collected from Canada showing that people that did not wear gloves had twice the rate of frostbite.

2) "Unvaccinated" and "Not Fully Vaccinated" are two different things according to their definition. They're clearly counting anyone that died within two weeks of getting a shot as being unvaccinated, even if it was their second shot. This is not a fair comparison.

3) Our prime minister caught covid twice in the last 4 months despite being vaccinated and boosted.

There have been studies that show that natural acquired immunity is longer lasting. Since the odds of dying are so low at this point, would you rather get covid once and feel a little bit sicker, unvaccinated, for longer lasting immunity. Or would you rather get vaccinated and boosted and catch it twice, feeling a little less sick each time. Which is what the data seems to be showing is happening. This seems like it should be personal preference decision.

Reply on news.ycombinator.com