Hacker News Reader

> It did not control for many factors that may be relevant (e.g. smoking).

From the abstract: "After additional adjustment for race/ethnicity, income, education, self- and physician-rated health status, body mass index, leisure exercise, smoking, and regular alcohol use"

> On the authors of the correlation study, who should never have studied this question without looking at extensive control variables or without more specifically studying causation?

How do you suggest studying causation in this setting? A randomized controlled trial where we deprive people of health insurance? Even that will likely not yield a true causal estimate, because randomization only helps for pre-randomization differences in the population, and behavior change from lacking health insurance will occur post randomization. The authors do extensively discuss their control variables, and important to remember is the fact that most papers only control for variables which ended up doing something, a subset of all variables that were tried. The NHANES data the study was pulled from includes a staggering number of covariates.

---

While not an ironclad study, I found the paper itself vastly more compelling than the politifact analysis of it, which boils down to "Well, observational studies might be wrong because reasons".

A randomized controlled trial where we deprive people of health insurance?

Yes.

Before instituting Obamacare/Romneycare/$POLICY, we should have run a pilot program based on random assignment with clear predefined success metrics. But that's politically dangerous - after all, what if the experiment shows that $POLICY doesn't work?

We did that, by accident, in Oregon (google Oregon Health Experiment). There were no statistically significant results beyond the placebo effect [1]. Strangely, none of our fact based politicians have proposed scrapping the medicaid expansion based on that.

[1] People with insurance perceived themselves to be healthier before actually consuming any medical care and became less depressed. But no statistically significant difference was observed in any of the objective metrics chosen before the study started.

Medical and public health researchers are bound to ethical guidelines that would prevent something like this, because the preponderance of evidence is that having health insurance is a net positive for someone's health - the only reason it made sense in Oregon is the fact that they needed a lottery anyway.

As for the Oregon study, the results of that study are still relatively new (the idea that any measure focused on preventative health will show results after two years is pretty suspect). The authors of the study discuss this for diabetes:

"Medicaid significantly increased the probability of being diagnosed with diabetes after the lottery (by 3.8 percentage points, relative to a base rate of 1.1) and use of diabetes medication (by 5.4 percentage points, relative to a base rate of 6.4). As discussed in the paper, based on clinical trial evidence on diabetes medication, we would expect this increase in the use of medication for diabetes to decrease the average glycated hemoglobin level in the study population by 0.05 percentage points, which is well within our 95% confidence interval for the impact of Medicaid on the level of glycated hemoglobin."

By this logic, it would be unethical for the FDA to demand a random trial for any drug that has some correlation studies showing it is effective.

As for the number you are cherrypicking, it is true that health insurance increased medical consumption (including ER visits, in spite of what ACA supports claimed) among people who received it. However, no measurable effect on health (besides depression) was observed.

You'll find that clinical trials are regularly halted when the treatment is found to be markedly superior or inferior to placebo or the control group.

The original study that showed aspirin's effect on heart attack prevention comes to mind for the former circumstance, and a number of HIV prevention studies for the latter.

Halting a clinical trial when the result is clear is very different from skipping the clinical trial on the basis of a correlation study or two. Your comparison is so nonsensical that either you are being deliberately disingenuous or you don't understand statistics at all.

Either way, no point in continuing this.

I was being charitable to your example, because generally speaking there are no observational studies done before a drug is put up for approval - that's not how the approval pipeline works. The closest I can come up to your example is the occasional off-label use of a drug for some other condition, but the reports from those are largely small n studies. That's entirely different from a series of studies based on NHANES.

Beyond that, in an intentional randomized trial, rather than the 'happy accident' like the Oregon study, the actual control is not 'Nothing' but the medically indicated standard of care. Studies are often required to provide medical care, education, etc. to their participants. I cannot imagine a study managing to get "We deny a bunch of folks health insurance" by an IRB unless it was an externally forced process, like the Oregon study.

Your insult about not understanding statistics, in addition to being off-base, is rather spurious. This isn't a statistical question, it's a public health ethics question. Statistics doesn't really come into whether or not "Keep a bunch of people from accessing healthcare" will get nailed by an approval board.

Also, the FDA often does take observational evidence into account, especially when expanding things like what age range a drug is medically indicated for.

While we're on the topic, Bayesianwitch.com's description:

"Your new homepage goes viral, but you aren't sure what copy is converting. Hook that copy up to BayesianWitch and only converting copy will be showing. No waiting days for the answers from an A/B test."

Seems to imply a reliance on "correlational" data. Is there some hidden randomization in there? Or are you giving your clients a lower standard of evidence?

We are using a lower standard of evidence than medical decisions. The goal is to get as many clicks/conversions as possible in aggregate, most of the time. I.e., if you have a call to action ("3 day sale", "spring sale", "march sale") that dies in 3 days, we'll do the best we can to increase your conversions in those 3 days.

Similarly, if you have 10,000 seo-optimized microsites, each with traffic too low for a per-site A/B test, we'll improve your conversion rate across the 10,000 microsites.

If you want to make a long term change (e.g., logo, button color, feature) for a high traffic site (your one and only landing page) you are better off using a traditional A/B test.

If you refer to Table 2 of the NEJM article 'The Oregon Experiment — Effects of Medicaid on Clinical Outcomes', while none of the results are statistically significant, most of the effect measures are headed in the right direction.

Someone who works for 'Bayesianwitch' should know better than to rely on p = 0.05 as the sole basis on which to evaluate something.

I also know better than to change the criteria after the study starts. A study was proposed. None of the critics of the study had anything negative to say about it until after the results were in - that's probably because they thought it would vindicate the 45,000 number.

Actually looking at the results isn't "changing the criteria", it's looking at the results in a more nuanced way than null hypothesis testing.

I would have said that looking at chronic health outcomes after just a few years was probably a losing proposition, and asked to see some power calculations, or a longer term analysis plan.

I've done so for other studies. Was actually grousing about one in a meeting...two weeks ago?

Sorry, you seem to be suggesting that having insurance (access to non-emergency medical treatment) has no effect on life expectancy.

Which might be the case. You argue that Oregon showed that. Doesn't that paint medicine as a huge fraud, regardless who is paying for it?

No, it doesn't paint medicine as a huge fraud. For example, suppose people without insurance already do have access to non-emergency medical treatment. Then giving them insurance will not make them healthier - it will only make them wealthier.

So that's part of what happened - if you look at the data, both the control and treatment group did consume medicine. You don't need insurance to get treated. But medical consumption increased in the treatment group - it just didn't improve health. That suggests medicine has a point of diminishing returns, and people without insurance already consume enough to reach that point.

(Also, a caveat: the Oregon Experiment was too short to measure an effect on life expectancy. They measured several other proxy health measures instead.)

I mostly agree with you, but you seem to be ignoring this point: Medicaid decreased the probability of having an unpaid medical bill sent to a collection agency by 25 percent – which also benefits health care providers since the vast majority of such debts are never paid.

A lot of uninsured people can get medical care in emergency rooms which are not allowed to turn them away, but then the cost adds one more huge burden on top of an already-difficult struggle to get out of poverty.

Why would you blame the study authors for doing a correlation study? I know "CORRELATION IS NOT CAUSATION!!!" is the go-to takedown on the internet, but correlation studies actually can have significant value - otherwise peer-reviewed journals wouldn't publish them.

I'm also skeptical of Politifact's competence to adjudicate public health scholarship.

I'm also skeptical of Politifact's competence.

FTFY

I agree that correlation studies can be valuable, but I see many correlation studies as (intentionally or not) exploiting a propensity (arguably a bug) in human reasoning that conflates it with causation. In this situation, we have thorough documentation of multiple people clearly making the error.

I'm actually not sure the study authors should be blamed.

But: given how politicized this question is, they could have reasonably anticipated the misuse of their results, and thus could have written their results in such a way as to avoid this. Or they could have publicly corrected non-experts who cited them for causation. Or: they could have controlled for factors that would make mortality and insurance-status independent. This last option is difficult & requires complex judgement calls (see [1] for a reasonable attempt), but even if you feel the other two aren't required of academics, this last one very much may be.

(Separately, I agree that Politifact should not be trusted automatically, but these seem like reasonable analyses, and I did not quickly find anything better.)

[1] http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2739025/

Not only do they have significant value, but they're the only type of evidence you'll ever have for something like this - it's impossible, and almost certainly unethical - to run a randomized trial of keeping people from having health insurance.

Also, while "Correlation is not causation" is, as you mentioned, a tired canard on the internet, people often seem to forget that all things that do have a causal relationship with have some form of association. Association doesn't prove causation, but it's a damned fine first step, and miles above "guessing", which is what happens without evidence.

Sorry, but no. 100 times no. Correlation is very often linked to a third factor or multiple factors which are not visible, nor measured in observational studies. Besides, let's not disregard the fact that correlation still has some good chance to be pure luck. Even correlation with 95% confidence statistical significance can be a random result in a non-nil number of times.

So, no, you never prove anything nor imply anything at all with correlation. You're still guessing.

Find me where I said proof.

There are all kinds of things that we cannot prove, because it is either impossible or wildly unethical to conduct a randomized study. For those things, you can make a determined effort to control for as many "third factors" - the technical term for them is confounders - and that gives you a level of evidence which is well above guessing.

Since I can't reply to your comment, my responses here:

> "You didn't say proof but you said it's better than guessing, and I don't agree with you at all."

It is better than guessing. You're welcome to disagree, but a well conducted observational study is considerably firmer evidence than pulling it from your posterior.

> "What if there is a correlation between Vegetarian-lifestyle and Serial-killers ? Does it tell you that it's better than guessing ? Do you even question if the association/correlation makes remote sense ? Is there any underlying mechanism of action that would remotely explain rationally why this correlation could be linked to any real causation phenomenon ?"

All you've done is describe a really bad study. You can have really bad RCTs as well, by the way.

Of course you question whether or not an observed association has a clear biological or social mechanism. And you attempt to control for other variables that might influence the link between your exposure and your outcome. You run followup studies in different populations to try to understand if the result is a widespread phenomena, or a fleeting bit of statistical noise.

Basically, you do your job well. Which is why I used phrases like "a good first step".

Your example is about as useful as "Programming is useless because once I coded something poorly and corrupted my data".

You didn't say proof but you said it's better than guessing, and I don't agree with you at all. What if there is a correlation between Vegetarian-lifestyle and Serial-killers ? Does it tell you that it's better than guessing ? Do you even question if the association/correlation makes remote sense ? Is there any underlying mechanism of action that would remotely explain rationally why this correlation could be linked to any real causation phenomenon ?

Correlation is useless, and there's a ton of observational studies out there finding correlations every single day for which we have no rational explanation at all. Observational studies are full of variations in the way they are designed, the way they are reported and the subjects of the studies, it's rather a miracle if you actually detect a hint of causation based on the garbage noise that you get.

> make a determined effort to control for as many "third factors"

How can you tell if you missed one?

> [...] gives you a level of evidence which is well above guessing

> It is better than guessing

Why? Without any support, this seems to be an appeal to probability.

> a well conducted observational study

> a really bad study

The fallacy of moving the goalposts; also the no true Scotsman fallacy.

> How can you tell if you missed one?

Similar studies, using as many variables as you can find. Residual confounding is always and ever a problem, but the odds that something is both a strong residual confounder and has never been observed to have an association with the outcome or the exposure is pretty rare?

> Why? Without any support, this seems to be an appeal to probability.

It's really not - if for no other reason than it's forced you to think about your system more than a simple guess would. It's not an appeal to probability, its using data to update whatever prior you came in with. Guesswork is just using your prior.

> The fallacy of moving the goalposts; also the no true Scotsman fallacy.

Not really, no. Some observational studies are crap - this is just true. But that doesn't say anything about the potential quality of observational evidence, and many of the commonly raised objections to observational studies are actually objections to poorly run studies. The example used was a study that examined no potential confounding variables, looked at a correlation with no prior evidence suggesting any linkage between the two or biological plausibility, and then asserts that they've found a causal link.

That's a bad study. It's not 'No True Scotsman Fallacy' to say that the problems with a bad study don't generalize to all studies. If it is, then we're all screwed, because you can run a bad RCT too.

> It's really not - if for no other reason than it's forced you to think about your system more than a simple guess would.

Appeal to probability; appeal to common sense.

> Some observational studies are crap - this is just true.

Special pleading.

> But that doesn't say anything about the potential quality of observational evidence

Straw man.

> many of the commonly raised objections to observational studies are actually objections to poorly run studies

> That's a bad study.

Moving the goalposts. No true Scotsman.

> It's not 'No True Scotsman Fallacy' to say that the problems with a bad study don't generalize to all studies.

Straw man.

> So, no, you never prove anything nor imply anything at all with correlation. You're still guessing.

This is a totally unreasonable stance to take. You can't even imply anything at all with correlation? Really, nothing at all? It's no better than a random guess? Try actually doing some actual science with this attitude, and keep to it consistently, and let me know how far you make it. In fact, try the same thing with ordinary life, any kind of reality where your decisions have consequences in reality.

You keep saying 'unethical' as if it's the end of the discussion. But what if our current standard of medical ethics prevent us from finding the better policies or the better treatments? Is this standard set in stone and never to be doubted?

Probably taking away insurance from people who have it as an experiment is too extreme, and not implementable anyway. But is it really that unethical to take a subset of population without insurance, give it to a random subset for some time, and observe the differences? Why? No one from the control group is prevented from getting insurance on their own (compare with the candidate drug trials, where the control group can't just buy the drug on their own).

> You keep saying 'unethical' as if it's the end of the discussion. But what if our current standard of medical ethics prevent us from finding the better policies or the better treatments? Is this standard set in stone and never to be doubted?

No, it's not set in stone, but if you're talking about implementing studies now, you're not going to get major medical ethics reform first. You go with the system you have, and the system you have is probably going to push back pretty hard.

> No one from the control group is prevented from getting insurance on their own

This alone is a difference between the control and treatment groups that takes place post randomization, and knocks said experiment back into the realm of "correlational"

> correlation studies actually can have significant value - otherwise peer-reviewed journals wouldn't publish them

This is an appeal to authority. And unfortunately, the evidence is accumulating that there are problems with that authority. The success of peer-review depends on the quality of the peers and of the review. If those reviewing don't understand how to evaluate a correlation study, or do understand but don't take the time to properly evaluate it, then garbage will slip through.

It turns out, lots of garbage is produced and sent to the reviewers, as was noted in a recent Nature feature:

http://www.nature.com/news/scientific-method-statistical-err...

http://www.nature.com/nature/focus/reproducibility/index.htm...

---------

tl,dr; lots of crappy correlation studies are published in peer-reviewed journals. These studies later turn out to be irreproducible.

"But I don't think a statistic like this should be left unchallenged on HN."

Yeah, thanks for fighting for the cause! #sarcasm

Is HN supposed to be a place where we try to find a flaw on every statement and make sure it doesn't go unnoticed?

What you pointed out doesn't even diminish an ounce of what Brandon and his teammates are doing.

> Is HN supposed to be a place where we try to find a flaw on every statement and make sure it doesn't go unnoticed?

A recurring and important (to me, anyway) question here is how technical skill may be leveraged to provide real value to the world. This is a hard and unanswered problem. This [brandonb's] statement is so strong that, if it were true, it would eliminate a vast territory of alternate [possibly correct] paths to answers ("oh, you did X? My code saved 50 lives this year."). So it [brandonb's statement] is worth challenging (or correcting) more so than any random statement here. And yes, I do view promoting accuracy (even disillusion) as fighting for the cause.

I hope Brandon will accept my sincere thanks for working on something that is important. Irrespective of health impact, the financial risk borne by the uninsured is an important issue and not controversial.

edit: clarifications in []

I'm curious what do you think would have happened if you just let it go.

"This is a hard and unanswered problem. This statement is so strong that, if it were true, it would eliminate a vast territory of alternate paths to answers ("oh, you did X? My code saved 50 lives this year.")."

It seems like you are more of the problem rather than Brandon's statement. No matter what Brandon says, accurate or not, people will still find a flaw in it. People will see what they want to see.

If you want to eliminate alternate paths to answers, the sure way to do that is not say anything at all.

I hope my clarifications help explain what I meant in my parent comment.

I believe the following are potentially bad consequences of Grayson's/Mikey's claim spreading:

- People work on insuring others, at the expense of other activities that they would otherwise believe to be more valuable.

- Insuring people (or the ACA) is deemed a failure because mortality rates do not come down as "expected", plausibly leading to the ACA's repeal.

- A developer expends time working on the project expecting mortality rates to improve; when it doesn't, the uncritical idealist becomes an uncritical cynic, rejecting any future promise of saving lives/improving things.

Why would your second point be a bad consequence? If the ACA doesn't work and we repeal it, isn't that a good thing?

In the ideal world, we'd analyze all of the ACA's costs and benefits and decide whether or not to repeal. But realistically, the most visible "promised" benefits (not necessarily those promised by the authors of the bill) are overweighted in the analysis.

I believe the ACA should be understood to promise increased insurance rates leading to (a) less medical bankruptcy and (b) moderate improvement in certain healthcare measures (not mortality). I don't think it should be deemed a failure in any sense if it fails to reduce mortality amongst the newly insured.

> What you pointed out doesn't even diminish an ounce of what Brandon and his teammates are doing.

Right. It sounds like you're implying that was his intent, and he failed. I really think it was an noble effort to get to the heart of what might be a misleading soundbite. Very much in the spirit of HN.

But it's such a small part of Brandon's post.

Is the spirit of HN nitpicking the smallest things?

>It did not control for many factors that may be relevant (e.g. smoking).

Unless I'm misreading the polifact article you linked to in [2], it says that the 2009 study did control for smoking, and that they did a better job of controlling for such factors than previous studies.

> Still, their work stands out from previous efforts because it used more recent survey data and presented a more apples-to-apples analysis between the uninsured and insured populations. For example, it compared deaths rates for uninsured smokers with insured smokers, as well as other factors such as drinking, obesity, income and education.

You are right, the 2009 Wilpers paper does bucket out current and former smokers, as well as look at BMI and other factors. The authors should be credited for that. But when those factors are considered, the null hypothesis is only barely rejected at 95% confidence. The Kronick paper uses a much larger dataset and discusses the issue of what is controlled for more extensively.

So wait, now not only are we using hypothesis testing as your sole means of evaluating whether or not an effect exists, but you're moving the threshold around because...you want to?

No, I don't think I'm arguing that. It's not a question of any one methodology always being better than others, or correlation studies always being wrong. The question is what we should reasonably believe in light of several analyses of various strengths.