Comment by KempyKolibri | Hacker News Reader

Comment by KempyKolibri | Hacker News Reader

Parent Full thread

Dismissing all observational study designs out of hand because they can be difficult and easy to perform poorly seems like quite the take.

I see this all the time in people’s interpretation of nutrition research, and they do exactly as this article suggests and fall back to the “intuitive option”, and go onto some woo diet that they eventually give up because they start feeling awful.

I would disagree that observational study designs should be thrown out the window or that it makes sense to, as this article seems to do, lump cross-sectional ecological data in with prospective cohort studies.

Things often “make intuitive sense” only because of these study designs. We used to get kids to smoke pipes to stave off chest infections because it made “intuitive sense” and it’s only because of observational studies that we now believe smoking causes lung cancer.

The direction of evidence from prospective cohort studies to RCTs in the field of nutrition science on intake vs intake shows a 92% agreement. If we take RCTs to be the “gold standard” of evidence that best tracks with reality, it seems a little odd that these deeply flawed observational studies that we should apparently disregard seem to do such a good job coming to the correct conclusions.

https://bmcmedicine.biomedcentral.com/articles/10.1186/s1291...

It's important to be thoughtful about research interpretation, but I'm kind of tired of kneejerk dismissal of observational studies for a couple of reasons.

First, experiments have their own varieties of horrors. Many are small N, with selective data reporting, and lack external validity — that is, the thing you really want to randomize is difficult or impossible to randomize, so researchers randomize something else as a proxy that's not at all the same. Other times there's complex effects that distort the interpretation of the casual pathway implied by the experiment.

Second, sometimes it's important to show that any association exists. There are cases where it's pretty clear an association is non-existent, based on observational data and covariate analysis. You just don't hear about those because people stop talking about them because of the null effects. So there's a kind of survivorship bias in the way results are discussed, especially in the popular literature.

It's easy to handwave about limitations of studies, it's much harder to create studies that provide evidence, for logical, practical, and ethical reasons. Why you'd want less information about an important phenomenon isn't clear to me.

> We used to get kids to smoke pipes to stave off chest infections because it made “intuitive sense” and it’s only because of observational studies that we now believe smoking causes lung cancer.

This is an interesting example, because I don’t know of any studies (although there probably are some, if only old ones) specifically about whether smoking pipes staves off lung infections, but the “intuitive sense” answer has changed because of adjacent evidence. And in this case, it’s not the lung cancer evidence that makes it intuitively unlikely that pipe smoking would be helpful, but a broader understanding of what causes lung infections, and what tobacco smoke contains and doesn’t contain.

I think that’s a fair critique! Probably would have been better to say that the intuitive position was that smoking was unrelated to lung cancer.

It is quite the take indeed; one that I posit resonates most strongly with people whose societal views tend to conflict with the available evidence.

Reply on news.ycombinator.com