Hacker News Reader

Comment by _null_ | Hacker News Reader

I have been reading for years about the numbers of studies being done (especially in the public health field) that can be whittled down to correlation without any strongly proven causality. Is this an issue with science, or with how the media reports on science? Is it really as bad as it seems?

I understand that it's difficult to control for various factors when doing large scale studies of health and behavior, but my perception as a layman is that there is some willful ignorance or absence of rigor. I realize this is perhaps too harsh a statement; I'm just trying get my point across.

View on HN

Observational studies can be incredibly hard to do in a way where you can infer causality. But most of the data you can get your hands on are observational.

Does cataract surgery reduce mortality in older women? It sure looked like it, until it was recognized that the study suffered from a concept called "immortal time bias" [1]. Now the reverse "seems" to be true. (Unhelpfully, it's easy to make an "explanation" for either case. Surgery lowers mortality? Of course! Better vision keeps you safe. Surgery increases mortality? Of course! Failing vision is associated with other failing organs, and surgery itself poses risks.)

In cardiology, we like to refer to immortal time bias as the "cheetos" effect. An example: Let's look at people who had an event (heart attack) and try to estimate the benefit of a procedure done afterward (literally eating a bag of cheetos within 60 days after the heart attack). You will find that eating a bag of cheetos confers increased survival compared to not eating a bag of cheetos. The reason for this is that in order to have eaten the bag of cheetos, you necessarily lived long enough to eat the cheetos. If you died first, you count toward the non-cheetos group. So stacking things up, it will look like cheetos keeps you alive longer. (The solution to this problem is not to count the time before eating cheetos towards your survival time. If you do this with literal cheetos, which don't have a survival effect, then you will correctly see no difference between groups.)

This is just one of the numerous biases that exist and can be difficult to account for. In my opinion, this stuff is just hard. If you are interested, Judea Pearl is developing tools for causal inference [2].

1 = https://retractionwatch.com/2018/08/28/a-high-profile-paper-...

2 = http://bayes.cs.ucla.edu/jp_home.html

Is 'immortal time bias' not 'survivorship bias'?

Looking at the notice of retraction[1], "immortal time bias" seems to be a particular subset survivorship bias, in which there is a distinct delay between two events, and by definition one facet of the study only looked at those who survived that time period. In this case, the time period between diagnosis and the actual surgery.

[1] - https://jamanetwork.com/journals/jamaophthalmology/fullartic...

Yes, these are the same concepts. As is often the case, slightly different terminology across fields. But no difference in the main idea.

> I understand that it's difficult to control for various factors when doing large scale studies of health and behavior, but my perception as a layman is that there is some willful ignorance or absence of rigor.

I perhaps understand the sentiment behind this, but if this is really what you think then probably you don't really 'understand' how difficult it is to do large scale studies of health and behavior. As a thought experiment, pretend you are the researcher studying a glass of wine a night and write the script you would pitch to potential research subjects. You have to try to convince them to be randomized to either completely abstain from alcohol for a period of 10-20 years OR to drink 1 glass of wine a night for that same period. Do you think you could get 10,000-50,000 people to participate and that those people would reliably stick to their randomized groups? How would you monitor compliance? How will you keep track of those people for 30-50 years to see differences in mortality or other outcomes?

The 'answer' may be here that it's simply best to acknowledge that we cannot realistically study some, and arguably many, things. I expect your response would be that at least trying to do the best with the resources we have is better than remaining blissfully ignorant. But the problem is that coming to wrong conclusions far more dangerous than simply acknowledging we do not yet have the resources to come to any conclusion. And science is built upon the shoulder of giants. But what happens when the shoulders you've built upon collapse? One mistake decades ago can end up destroying decades of further work that was built upon it.

Science is never 100% on practically anything. Even in physics there are keystone works that are mutually incompatible indicating errors in one or both. But there is a difference between models that are based on predictiveness, direct experimentation, and falsifiability - and models that are based on correlation. Newtonian mechanics is a good example. The equations that determine a body's position given a set of inputs and time could be shown to be 100% correct in every single scenario. Well at least until Mercury came along with it's 1/100degree per century orbital aberration. And that 1/100th aberration per century of a single body was enough to inform that something was wrong. With correlational studies, actual results don't really matter and entire views become practically impossible to falsify. And when they do provide predictions, rarely is it the case that the failure of that prediction to come true would falsify the study. That's just not science by any stretch of the imagination. It is the difference between astronomy and astrology.

I mostly agree with this. But part of it is scientific illiteracy on behalf of most people, who want these 100% answers, not grades of evidence and certainty. An observation study is definitely helpful, but should be taken as what it is, an observational study, not truth, which is often how it is reported and understood.

But I don't think that coming to wrong conclusions is necessarily 'far more dangerous'. People who drink 1 drink a night are not likely to have dramatically different outcomes than people who abstain. (perhaps unless it contributes to binge drinking, but science already says that is bad)

I guess what I mean is that the results of these types of studies are useful, but only in pointing where to look for a more direct mechanism. In other words, we found a correlation, now we can look more closely at the elements involved to find causality. Mostly it seems like the correlation is accepted and then untested theory is exposed as fact to explain it.

>correlation without any strongly proven causality

How would one really prove causality or does it even matter. When I think of science, I try to think of it like how I think of physics, where we create a model that best describes the evidence but which doesn't have any guarantee of being how things really work.

Take Newtonian gravity. It is a pretty good model that describes a lot of basic interactions. Given a state at a given moment in time, it lets us determine things going forward or backwards (though for more complex physics, backwards stops working because of assumptions and estimates in the model). But at the same time, it is wrong. More complex physics shows there is a model that fits even more experimental data which contradicts what we thought was happening in the Newtonian model. Mass doesn't attract mass. Mass bends space time and impacts objects traveling through it in such a way that it appears mass attracts mass (though even this may end up being wrong and something else entirely is at play).

So, is it really important to describe why a ball falls to the ground when I release it, or is it good enough to have a system that describes the interactions enough that I can apply it to problems? If I solve a problem, say what angle and what speed I need to throw a ball to get over fence, does it really matter if I think the ball falls back to earth because mass attracts mass or because mass bends spacetime? Or is it just important for my equations to be accurate enough that any error is less than what is innate in applying the solution to real life (measuring the exact height of the fence, throwing at an exact angle).

What does applying a similar mind set in something dealing with vastly more complicated systems, such as in medicine, looks like?

> So, is it really important to describe why a ball falls to the ground when I release it, or is it good enough to have a system that describes the interactions enough that I can apply it to problems?... does it really matter if I think the ball falls back to earth because mass attracts mass or because mass bends spacetime?

I think you answer this question in your own post. The answer depends entirely on your goals.

"All models are wrong, but some are useful".

If all we ever wanted to do was shoot cannon balls over/into walls accurately, then we probably would've never bothered inventing modern physics.

But we did want to do other things, so we built better models. It's worth noting that some of those things we wanted to do were more philosophical than others, e.g., engineering ("put satellites into orbit"), explaining empirical observations that seem important ("explain how electricity really works") and philosophical ("understand the nature of reality") are all answers to the question "what are you goals?" that have inspired progress toward better models in physics.

> What does applying a similar mind set in something dealing with vastly more complicated systems, such as in medicine, looks like?

It takes a certain amount of philosophical sophistication to realize that even the most perfect model is still a model and to then reason through what that entails -- epistemologically -- for less perfect models and for the entire scientific enterprise.

>what are you goals?

I'm personally a fan of the 'make a better model to have a better model and let someone else figure out the application' model, but that one is harder to get funding for.

It matters for decision-making. If A causes B then forcing A to happen artificially (without changing anything else) will make B happen too. If it's not a cause then it won't. The correlation might always happen in the original environment, but it doesn't survive intervention.

This does matter in physics too.

There's a big difference between knowing that A causes B and knowing how A causes B. For most purposes relating to health it's sufficient for a layperson to know just that A does cause B, no matter how. But if A is just correlated with B and doesn't cause it, that makes a world of difference.

But you can never really know if A causes B or not. You can only create models, some that have A causing B and others than only have them correlated, and get rid of models as you find contradicting data.

What actually causes a ball to drop to the ground when I release it? We don't know. The best model (that I know of) is the bending of space time, but that isn't the real answer and may one day be overturned just as the older idea of mass attracts mass.

I guess the question is, are we sure enough of our model to be able to trust the airplane isn't going to drop to the ground like the ball does, and how do we achieve equivalent certainty in the biological sciences.

I recommend reading Judea Pearl's _The Book Of Why_, which details why historically science has been restricted to correlational observations, and how causal calculus can shed light on causality.

The incentive structure in "science" is completely fucked, and it's a well-known dirty secret.

Of course there is great research being done. The problem is that the cognitive load of evaluating which research is useful has been shifted to the end "consumers" of that research.

I think it's little more than a sharp increase in bias. The physiological sciences are heavily driven by moneyed interests and the social sciences tend to be full of people who are extremely politically motivated, and homogeneous, in their 'research.' Bias is always bad, but at least when it's counter-balanced with bias in the opposite direction both sides can help bring each other closer towards the truth by remaining extremely critical of one another. But bias without any counter balance is particularly dangerous as it leads to a feedback resulting in even greater extremism and bias: social media in a nutshell, but it's not limited to just social media of course.

Compare science that's goal is to find a truth to science that's goal is to prove an assumed truth, and you'll see the standard of research sharply decline.

There are some systemic issues in play for sure. Publish or perish means researchers need to put out new publishable work, or find a new career. And that in turn also introduces a publication bias. Spend 5 months researching a hypothesis only to find that it's wrong? Well that's pretty much 5 months wasted because journals are not so fond of publishing negative results. Massage the numbers a bit to get that positive result though, and you're good to go. Pretty nasty incentive systems there. And now add into the fact far more people than ever before are pursuing postgraduate education. It's a mess.

I think it's an issue with studies funded by someone that has an interest in a certain result, and also a media that is happy to publish a story that is nearly written for them.

Judea Pearl literally wrote the book on causality.

Reply on news.ycombinator.com