Go to the Wrong Hospital and You’re 3 Times More Likely to Die
nytimes.com
nytimes.com
Risk-adjusted outcomes were calculated after adjusting for population factors, co-morbidities, and health system factors. Even after risk-adjustment, there exists large geographical variation in outcomes
They are trying to measure differences in effectiveness for similar patients. GP gives a possible explanation for such a difference : the hospital probably gets good at the kinds of things it does a lot.
A hundred years ago a doctor named Ernest Codman suggested that hospitals should be rated by an "End Result System" [1] where paient outcomes were measured and compared, yet only now do we have the "first comprehensive study comparing how well individual hospitals treated a variety of medical conditions". And we still do not know exactly which hospital is best and which is worst.
To me the best way to judge a hospital is blindingly obvious: For each patient at admission, estimate the chance he/she will be alive after 3 years. For example, a 65-year-old woman who is a smoker, is overweight and has stage 2 lung cancer might have a 25% chance of being alive in 3 years. Then compare the predicted outcome with the actual outcome. For example, say both Hospital A and Hospital B both admit 100 patients with 20% predicted chance of surviving 3 years. If after 3 years there are 30 such patients from Hospital A still alive but only 10 such patients from Hospital B, it would strongly indicate that Hospital A is better. We could look then at Hospital A for ways to improve Hospital B.
What's so disheartening is that this process doesn't involve any special technology. It could have been implemented 100 years ago, albeit with less rigorous statistical methods. We spend a billion dollars on evaluating a drug which has a marginal, one-off benfit for a few hundred thousand people. But we ignore a process which is cheap and offers long-lasting benefits to millions of people. It seems that the medical establishment is too powerful in this case.
That said, it's still worthwhile to gather and use it for quality improvement. We just have to be careful to treat it with appropriate caveats and look at qualitative factors as well as quantified metrics.
Until we automate the variability out of healthcare, we are stuck with inconsistent results. Consider that roughly 70 years ago, lobotomies were considered a valid medical procedure for psychiatric problems including anxiety and depression.
This is subtly different to your analysis in two ways:
1. The trouble with using survival or outcome ratings is that they effectively punish providers who take on the hardest, riskiest cases. Not necessarily - if the professor underestimates the risk in easy cases and overestimates the risk in hard cases it would give the hospital an incentive to take on the hard cases and turn away the easy cases.
2. The reality is that's it's impossible to accurately quantify all the different factors which potentially impact survival. In some cases we have remarkably detailed understanding of a broad range of risk factors - take a look at table 3 from page 23 in this [1] report on stem cell transplants. There are detailed odds ratios for patient age, ethnicity, karnofsky score, CMV status, disease type and stage, interval between diagnosis and transplant, year of transplant, donor match type, donor/recipient sex parity and donor age. More importantly, the professor doesn't need to be particularly accurate in assessing patient risk - she just needs to be as accurate as the hospital. (Or close enough that the difference would not give the hospital a significant advantage)
Can the professor get close enough to hospital accuracy when estimating patient risk? Two relevant points spring to mind: (a) In the book Thinking Fast and Slow there is a description of how expert analysis can often be matched by simple rules of thumb. (b) The book Superforcasting describes how some lay people are able to make better predictions than the CIA about future geopolitical events, despite having no access to classified information.
My gut feel is that the hospital would not have a significant edge over the professor. But I'm willing to be proved wrong - let's conduct an experiment where a doctor estimates the 3-year chance of survival for each patient at the time the patient is admitted to hospital, and a professor also makes the same estimate based only on the patient's database record. The winner gets half a million dollars. Wouldn't this experiment give us valuable insight in determining whether hospitals can game an End Results system?
[1] http://www.seattlecca.org/sites/default/files/content_page/2...
And if we keep the identity of the hospital confidential, the number of hospitals willing to provide data on patient outcome grows substantially. The study referenced by the New York Times article [3] had 22 million hospital admissions, and the Cystic Fibrosis Foundation has gathered detailed data from all US Cystic Fibrosis treatment centers for 50 years [4]. Although it would be better for all hospitals to be completely transparent, there are still significant benefits to judging hospitals in anonymous fashion, as described in [4] (which is a fascinating article in its own right).
As for statistically significant results, it's true that we will be unable to judge the effectiveness of smaller hospitals (unless they are extremely good or extremely bad). But for medium-sized and large hospitals this is not a problem - in the stem cell study [3] there were 43 out of 173 hospitals whose 2015 results were judged to be outside the 95% confidence interval (26 were better than predicted to a statistically significant degree, while 17 were worse). Of those 43 outliers, 26 were also outliers in 2014 which suggests this is not just a fluke.
Regarding your question: "Did the patient die because his heart bypass surgeon was incompetent, or because he went right back to eating Big Macs as soon as he was discharged from the hospital?": Keep in mind that we can control for body-mass index, so the question is whether people with similar BMI will be more likely to eat Big Macs after treatment at Hospital A, rather than Hospital B? Yes it's possible, but it seems to me more likely to be an effect of the hospital. Also note that it may not be down to the quality of the heart bypass surgeon - it may be simply that Hospital A is more effective at encouraging healthier eating after patients are discharged.
[1] http://www.sts.org/quality-research-patient-safety/sts-publi...
[2] https://www.seattlecca.org/sites/default/files/content_page/...
[3] http://journals.plos.org/plosone/article?id=10.1371/journal....
[4] http://www.newyorker.com/magazine/2004/12/06/the-bell-curve
On top of that, the other commenter mentioned the unintended consequences of simplistic measurements like the one you describe. Look at measuring patient satisfaction - 'what could possibly be wrong with measuring patient satisfaction', people say! hold docs/hospitals/etc. accountable. but when you look at the data, patient satisfaction scores are weakly correlated with evidence based care and healthy outcomes but it is strongly correlated with things like how much time the patient sat in the waiting room or if the doctor prescribed them a medication.
Very much this. Two patients, same doctor, same proceedure, with somewhat similar medical backgrounds. Nearly the same progression in the hospital, and released to go home, and possibly to doctors that don't interact with that hospital.
Perhaps one is poor and sometimes skips medication. Only one starts eating healthily and starts exercise. One has a stressful home life. Maybe one, when the recovery isn't going as quickly as possible, starts mixing in alternative medicines but doesn't tell his GP. Maybe they were both women, in the hospital to give birth - and one got pregnant again in the first year while the other didn't. These sorts of things greatly affect the outcomes, but none of them are under the hospital's controls.
That is a good point - this thread has turned out to be more complex than I expected, and had I been a bit more open-minded I would have avoided the phrase 'blindingly obvious'.
Still, I'm not yet convinced this is an intractable problem. The New York Times articles says that patients are 3 times more likely to die at one hospital than another, even after controlling for patient sickness, income and age. This may be due to (a) a confounding variable; (b) a real difference between the quality of care provided by the hospitals. How much weight should we place on each possibility? Is it 50/50, or 60/40, or 10/90?
If you have to have a medical procedure and were told the survival rate of Hospital A is 98% and at Hospital B is 94% would you say, "there are so many possible confounding variables, this statistic will have no bearing on my choice of hospital"?
I found this for hospital-level comparisons (these should be much more accurate due to far larger numbers): https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4250275/#!po=2....
However: "Most commonly reported outcome measures have low reliability for differentiating hospital performance."
I have worked countless full arrest codes in the field... countless. Some go a lot better than others, and some are even successful. Most codes are do not end well for the patient whose heart has stopped beating or the drive to breathe has stopped. I can't imagine having stats of the codes we work pushed in our face everyday. There are sooooo many variables that we don't control which would make the whole "measuring" endeavor meaningless to anyone that really cared. From the weight of the patient to the cleanliness of the house are all variables that may or may not matter. Good luck controlling for all that!
I usually look at a doctor's reputation, experience level, education, publication, etc. as signals for "quality." It CAN vary drastically, as in, one doctor might overlook a diagnosis or even mis-diagnose something that another, more experienced doctor would catch correctly.
Tracking revision and mortality rates for surgeons would be much more helpful. I have been told (by engineers who have worked on joint replacements) that some surgeons have revision rates well over 50% (and frequently leave patients with one leg longer than the other), while others have revision rates well under 10%.
Well that's not too useful for the patient oh I mean "consumer"
In metro LA, the top-tier hospitals (Cedars-Sinai, UCLA) have noticeably better outcomes than regular hospitals. The difference is like 98.7% of patients not dying versus 97.9% in a valve replacement ("AVR"). Flip it around and it's 1.3% versus 2.1% -- definitely noticeable.
OTOH, UCSF (presumably top-tier, SF residents correct me if I'm mistaken) has 2.3% chance of dying. Maybe it's not a top-tier facility for AVR?
And further, a small midwestern hospital in Hays, KS (pop. 21000) has a 3.6% chance of death. That's huge next to Cedars-Sinai at 1.3%!
As a referral center for other referral centers, additionally UCSF will be getting some of the most complicated cases in the world.
I'm not holding a torch for Cedars, but they are of similar stature for referrals AFAIK (not directly on-point, but indicative: https://www.cedars-sinai.edu/About-Us/News/News-Releases-201...).
In the case of one example where NY started publishing outcomes the difficult cases switched to the Cleveland Clinic. The outcomes appearred to increase in NY and decrease at the Cleveland Clinic, but risk adjusted rates didn’t actually change for NY hospitals that transferred cases or for the Cleveland Clinic.
This is absolutely something that happens and Cardiac Surgeons are all scientists to some degree. The results and risk adjustments are very well documented and they can determine them with a fair degree of accuracy. I’m not aware of the info for this particular case, but if they did risk adjust (the patient) it right - there are still a lot of other factors that determine quality of care (speed of delivery of service) and quantity of surgerys performed (if you don’t perform enough surgeries per year your quality suffers). Incidently, this is true for surgical staff (nurses) as well. If the surgeon travels to numerous to hospitals… the hospitals with the most surgeries performed will have better outcomes (after risk adjustment for them doing higher risk surgeries).
This reminds me of a scene in the recent Dr. Strange movie where the main character -- a neurosurgeon -- turns down a case because it had a high risk of failure and he didn't want to tarnish his perfect record.
> They claim to be risk-adjusted, but in reality you cannot adjust for all possible risk factors. Lots of things in medicine are very unique. There's very little that's more complex than human disease.
Isn't it better to approach the problem by improving how we do these risk assessments so that the risk-adjusted ratings are fair, as opposed to leaving patients in the dark about where to get the best care?
I agree patients shouldn't be left in the dark. The most important thing is getting the best care. There isn't an easy solution.
However, I think the existing solution hurts patients (and I have seen these risk assessments first-hand numerous times).
And thinking about it, I realized how many projects I jumped in while others jumped out because, you know, they were risky in not succeeding...
Edit: Many patients are already assessed by performance status [1]
Put another way, a single death out of a small population of patients will have a larger impact on the statistics.
http://jamanetwork.com/journals/jamainternalmedicine/fullart...
Flip it back around and it is a 0.9% reduction in survival rates. ;-)
I'm going to guess that hospitals aren't exactly the biggest factor influencing outcomes...
1. make the good hospitals worse
2. make the bad hospitals better
3. combination of 1 & 2
4. ignore
Most people will want to choose 2 but inadvertently choose 3. When in reality we should have chosen 4.
If you go to a bad hospital, you're 3x more likely to die. But if you go to a good hospital, you're 99 / 97 or roughly 2% more likely to live.
Suppose 1 in 100 patients (for X) die at the best hospital and 3 in 100 die at the worst.
Keeping percents the numbers are much less scare mongering.
Best: 1% die, 99% live
Worst: 3% die, 97% live
n=1 tho
For me, that was a far better outcome than I would have received in the US for a similar condition. Of course, stitches are pretty well understood; I can't advocate for any other medical procedures in Cambodia.