What happens when patients find out how good their doctors really are? (2004)
newyorker.com
newyorker.com
"Matthews had started a cystic-fibrosis treatment program as a young pulmonary specialist at Babies and Children’s Hospital, in Cleveland, in 1957, and within a few years was claiming to have an annual mortality rate that was less than two per cent. To anyone treating CF at the time, it was a preposterous assertion. National mortality rates for the disease were estimated to be higher than twenty per cent a year, and the average patient died by the age of three. Yet here was Matthews saying that he and his colleagues could stop the disease from doing serious harm for years. “How long [our patients] will live remains to be seen, but I expect most of them to come to my funeral,” he told one conference of physicians.
In 1964, the Cystic Fibrosis Foundation gave a University of Minnesota pediatrician named Warren Warwick a budget of ten thousand dollars to collect reports on every patient treated at the thirty-one CF centers in the United States that year—data that would test Matthews’s claim. Several months later, he had the results: the median estimated age at death for patients in Matthews’s center was twenty-one years, seven times the age of patients treated elsewhere. He had not had a single death among patients younger than six in at least five years."
"In this short speech was the core of Warwick’s world view. He believed that excellence came from seeing, on a daily basis, the difference between being 99.5-per-cent successful and being 99.95-per-cent successful."
Many of the medical percentages that we see quoted in media are derived using a lifetime risk basis, _not_ a daily one. On a lifetime basis the difference between 99.95% and 99.5% _is_ negligible.
I wonder how many people reading this article are aware of the difference.
For instance, I'm from North Texas; I went to college in Oklahoma. Back then (1996) Texas had a type of insurance for children with CF (it actually extended to age 21) unavailable in Oklahoma. I ended up losing care, going off of my meds, and by the time I was back in Texas and had the income to support my needs (I came from a very poor family) I had lost 30 pounds (down to 98lbs) and had lost 25% of my lung capacity. Was this attributable to the clinic in Dallas? Of course not.
(FWIW, I'm up to 155lbs these days and my lung capacity is probably in the upper 5-10% of all CF patients)
Thanks :-)
There are a lot of challenges with healthcare metrics. Medical problems are more complex, arising from more factors than practically any other problem. Most other sciences can be resolved by theory or relatively convergent empirical data. Medicine is empirical, but the scatter is huge.
Data is not tracked well. When it does exist, it is in formats that are not well conducive to analysis and sharing. This seems to be slowly (and expensively) improving. Pressure for privacy adds to inertia for innovations that could use medical data for improved outcomes.
With the title of the article, I was hoping there would be some more discussion about implications of some 'rating system' for doctors. It would probably superficially be a good idea, but such a system would obviously result in some level of gaming. Does a doctor in a rated system only take easy cases to keep a good track record? Are hard cases given more weighting? Who/ how is such a weighting decided? Such a system could offer a results-based compensation scheme for doctors, which is lacking from most publicly funded systems.
The lower ranked surgeons would have the scope jostling around and the laparoscopic instruments would contact nearby tissues, while the higher ranked surgeons just seemed to have more elegance in his skill; a laser-precision focus to borrow from this thread and an intention and purpose for every movement.
I had a similar impression watching that video on surgery technique to watching high level Starcraft player replays versus some afternoon ladder replays by mid-level players. When placed side-by-side, the differences were fairly obvious even if you've never done the act yourself.
Surely, some doctors are better than others, but (for the sake of discussion :)) I don't think every professions or activities are equivalent with respect to 'rating'. Take the extreme example of walking. All human beings walk equally well. This is unlike playing music or solving math problems for instance.
I'm certainly not saying that medicine is as easy as walking, but maybe for a wide range of health problems, there are well known standard procedures or solutions that all doctors are competent enough to follow. And in doubt, they usually refer their patient to a specialist, who in turn can send them to a more competent doctor.
This is obviously not true.
An other example that comes to mind are regular airline pilots: nobody wonders whether they are good or bad. We never have to worry that the pilot is unable to fly the plane.
http://en.wikipedia.org/wiki/Professional
I just didn't like the walking example since it's neither true nor analogous.
Again, this is a terrible example. I'd say that a good third of passengers are wondering exactly that. (It's irrational to do so, but that's another story.)
Divergent, but any skill I can think of will have a distribution of abilities. Some of these skills have a threshold beyond which incremental improvement gives little benefit. Walking example: if you can walk 100m in one go, you are better off than someone who can walk 10m. Likewise, 1000m > 100m. Does 500km give any more benefit than 50km? Maybe, but probably only for extreme use-cases (maybe competitive extreme-distance walking? I'd rather drive!). Similarly with technique - if walking technique is horrible it matter for mobility and injury risk. Beyond a certain level, it probably matters less. And maybe walking extremely well is a visual-social signal, as a marker for someone who is in good shape? I'd read somewhere research done on dancing as a marker of fitness for attractiveness to the opposite sex.
"Doctor".
Well, buddy, that just means that if you have a good solution, it's stopping you from getting a perfect solution.
They were not forced to produce crap by overly active manager, it was not that situation. Manager did not cared for months.
Perfect is the enemy of the good refers to the above problem.
I would generally rather have a less than perfect system that can be used now than a "perfect" system 3 months from now as, in my opinion, you only really validate what you have built when it is in production and in use and delivering value.
In the context of this article: You have a patient who needs to be treated -- today. They can't wait for a hypothetical perfect treatment. Indeed, more fundamentally, the only way to discover a "perfect" treatment would be to use today's least-worst solution, with the determination to measure and improve it continually.
Also the point of the article was that it's not so much the techniques -- which all the CF centers know about and use -- but the aggressiveness of their consistent application.
This isn't about a perfect design. It's about doctors who are willing to coach/goad/persuade patients into doing consistently what they already know they need to do. (And who are willing to be sticklers for consistency to the point of being a bit of a PITA to their colleagues.)
One thing that stands out in my memory about this was the fact the he made sure he always had 3 (!) pens for the customer to sign the sales agreements. Just in case 1 pen stopped working and the backup pen malfunctioned. He brought this sort of laser focus to absolutely every aspect of the business.
How many salesmen would have a spare pen, already in place, let alone 2 spares. How many would even have thought about it and what the right number might be?
Focus, attention to detail and continuous innovation.
I'm in a company where the suit-wearing side says "we need innovation" to the tech side, but they don't actually want a discussion about what they see as opportunities or directions. They just want to invoke it by rote ritual, or order it as if we kept some on shelves.
I found this quote especially interesting. I've often found this to be true, that you look over at someone who's much better, and on one hand they're just doing the same things you're doing, but somehow they're just on a different level. It's very difficult to distinguish exactly how someone achieves better results, that is, until it's been explained. Here, we see this clearly in how the level of lung function is perceived. Both centers provide the same treatment, but at the Cincinnati center lung levels around 70% are "okay", but at Minnesota, any drop is unacceptable, even if the patient is already at above-normal levels.
It sort of reminds me of the "Don't deal with it, fix it" article that was also posted recently. You'll be coasting around at a certain stable level of engagement until one day you see something you've never considered before and all of a sudden you see a whole new world of possibilities.
Evidence based practice isn't really new, it's been a thing since '92. And while it is about using studies to make treatment decisions, it's not about forbidding doctors from thinking for themselves.
But what really got my goat was the fact that evidence-based medicine is the new thing, and it's an essential step in the evolution of medicine, and more people need to understand it. Evidence based medicine is about applying statistics to determine when treatments cause more harm than good. With modern medicine's advances in imaging techniques, we're hitting all sorts of new and dangerous problems where the benign abnormalities we all accumulate over our lives are treated at great expense, inconvenience and loss of health.
So someone has to be the person who ignores it, who tries new things. And that someone, in this field, is Warwick.
The evidence part comes in when that person also tracks their results! i.e. makes it a study, and Warsick certainly seems to do so.
i.e. Accounting isn't worthless. It's necessary, but not sufficient.
“We are used to thinking that a doctor’s ability depends mainly on science and skill. The lesson from Minneapolis is that these may be the easiest parts of care. Even doctors with great knowledge and technical skill can have mediocre results; more nebulous factors like aggressiveness and consistency and ingenuity can matter enormously... What the best may have, above all, is a capacity to learn and adapt—and to do so faster than everyone else.”
There are tests you can do to tell if the variance between care centers exceed that you would expect from chance, but the mere observation of a bell curve seems completely uninteresting.
Exponential is completely positively skewed and thin-tailed, and Pareto is roughly the same shape but very fat-tailed.
Medicare recently produced this data http://www.nytimes.com/interactive/2014/04/09/health/medicar... and the center that my wife works at used it to to produce this graph http://blog.parathyroid.com/parathyroid-surgery-medicare/. They are quite a bit better at this particular operation than anywhere else in the US, not surprising given how many more they do, however it is often difficult for patients to know or understand that - who should they trust? More data please! Interestingly this operation is one that Atul Gawande (who wrote this article in 2004) specializes in too.
tl;dr: If you don't adjust for the sample size, what may appear to be the best, or worse, hospital, may look like that only because of how they have (un)successfully treated a single patient, and thereby yield an unrealistic estimate of patient quality (ie; 100% cure rate of a single patient, which may not be reflective of the actual caliber of doctors).
Further reading, which is strongly recommended if you want to learn more;
http://nsmn1.uh.edu/dgraur/niv/TheMostDangerousEquation.pdf
I guess it's possible...but it's probably more likely that the New Yorker is not trying to be a reference on statistical methods. In any case, the statistical caveat you mention is arguably addressed in this catchall-paragraph that briefly describes the problem of quality-of-care statistics:
> In recent years, there have been numerous efforts to measure how various hospitals and doctors perform. No one has found the task easy. One difficulty has been figuring out what to measure. For six years, from 1986 to 1992, the federal government released an annual report that came to be known as the Death List, which ranked all the hospitals in the country by their death rate for elderly and disabled patients on Medicare. The spread was alarmingly wide, and the Death List made headlines the first year it came out. But the rankings proved to be almost useless. Death among the elderly or disabled mostly has to do with how old or sick they are to begin with, and the statisticians could never quite work out how to apportion blame between nature and doctors. Volatility in the numbers was one sign of the trouble. Hospitals’ rankings varied widely from one year to the next based on a handful of random deaths. It was unclear what kind of changes would improve their performance (other than sending their sickest patients to other hospitals). Pretty soon the public simply ignored the rankings.
Even with younger patients, death rates are a poor metric for how doctors do. After all, very few young patients die, and when they do it’s rarely a surprise; most already have metastatic cancer or horrendous injuries or the like. What one really wants to know is how we perform in typical circumstances. After I’ve done an appendectomy, how long does it take for my patients to fully recover? After I’ve taken out a thyroid cancer, how often do my patients have serious avoidable complications? How do my results compare with those of other surgeons?
(the author himself is a surgeon and has written a lot about the problems of reliably measuring quality of care performance)
I'm not suggesting that they're ignorant of basic statistical facts, but I am definitely suggesting that they're not immediately aware of the subtle assumptions implicit in many of the statistical models they use.
For example, given the large numbers we're talking about, not only is it possible that the "number 1" hospital in any particular field is there because of statistical fluke, it's actually likely that this is the case.
The (mis)use of statistics certainly isn't limited to medicine, but it is one of the places where its misinterpretation has the biggest impact.
That's not very Kantian. How are less-experienced doctors supposed to get experience?
A. Mister.
IOW, this joke/insight is only valuable if the standard for passing is set too low.
If it's set correctly, then everyone who passed has shown they are prepared to be a competent doctor.