Hacker News Reader

> We don't really need truely objective assessments. As an analogy, imagine we randomly added or subtracted 3 minutes to the finishing times of all the runners in the Boston marathon. We wouldn't have a truely objective assessment of each runner, but it's still a useful guide about who is likely to beat whom in the next marathon.

I don't think the analogy fits. There's no single metric (like runner speed/time) to evaluate competence.

> This problem can be alleviated to a large degree by risk-adjusting the outcomes: For each patient, estimate the most likely outcome and compare the actual outcome to the estimate.

Agree.

> I don't see this as an insurmountable obstacle. For example, you could have an independent body randomly sample the reported outcomes to check that they are accurate, and apply some kind of penalty if they are not.

That would be a good solution. Part of the problem is finding a truely independent body. Medicine is over-run with various governance and regulatory bodies (several of which have been shown to be little more than rent collectors). And again, there's the problem of deference to eminence combined with small in-bred communities within each speciality or sub-specialty (who would likely be the only people with the training to evaluate their peers reliably).

> I don't think the analogy fits. There's no single metric (like runner speed/time) to evaluate competence.

Let me suggest such a metric: A T-value. That is, the degree to which the actual outcome deviates from the expected outcome.

For example, say you have a hospital that does stem cell transplants. For each patient before the treatment you assess the "chance that the patient will die within 1 year" based on that patient's age, sex, BMI, heart and lung function, type of disease, time since last relapse, quality of donor match, etc. From this you estimate that 19.6% of the hospital's patients who are treated over a particular period will die within 1 year, with a standard deviation of 3.7%. The actual mortality rate for this group of patients turns out to be 26.2%. So the T-value is +1.78; this is the 'single metric' used to evaluate the competence of the hospital.

> there's the problem of deference to eminence combined with small in-bred communities within each speciality or sub-specialty (who would likely be the only people with the training to evaluate their peers reliably).

Keep in mind that expertise is only required in estimating pre-treatment "chance that the patient will die within 1 year". Determining whether a patient is alive after a year and calculating the T-value can be done by anyone. And even when estimating expected mortality, there is plenty of evidence that a simple algorithm can actually beat the experts - Daniel Kahneman devotes a whole chapter to this point in 'Thinking Fast and Slow' [1].

[1] Chapter 21: https://www.amazon.com/dp/B00555X8OA/ref=dp-kindle-redirect?...

Overall, I like the idea of a metric such as the one you describe. However, a change in one year mortality rate is unlikely to be a reliable indicator of the efficiency of most treatments. Most medical interventions have more modest effects than definite improvement in short term survival.

Sure, there might be better metrics than 1-year survival and different treatments could have different metrics.

Nice theory, but in reality there's no way to do accurate risk adjustment. We just don't have enough data and so many physicians would end up unfairly penalized due to factors outside their control. Perhaps in 100 years we'll be in a better position to do something like that.

Is this an argument against tracking and releasing data? Let people do their own risk adjustments.

If twice as many people who are going to one doctor or hospital are ending up dead or crippled, and there are no discernible confounding factors (age, socioeconomic, co-morbidities, etc), then I want to know. One might even say the patient has a right to know. The burden should be on the provider/facility to convince the patient to trust them nonetheless.

Yes this is an argument against releasing data. You and most other patients lack the skills and context to interpret the data in a meaningful way. And no provider currently captures all of the confounding factors. In fact, for most serious conditions we haven't even done enough research to understand what all the confounding factors are.

If we go with your proposal then the inevitable outcome is that the best providers (particularly surgeons) will engage in metrics arbitrage by refusing to treat patients whose co-morbidities and complications aren't adequately captured by standardized coding systems and clinical guidelines. Is that really the outcome you want?

As for putting the burden on providers to convince patients to trust them, good luck with that. We currently have shortages of providers in many areas and specialties due to price fixing and supply constraints. So most patients have to take what they can get regardless of trust.

Your views on bodily autonomy seem bizarre. Patients own their own bodies and have the right to make fully informed decisions about what happens to them.

If a provider's or facility's (or entire speciality's) numbers look bad, that's important information for the consumer to possess. Hiding the numbers should not be an option.

Or do you think that car crash test results, and airplane crash data, and health department inspection findings should also be kept secret?

More problematically, if you can't articulate a quantifiable standard that will indicate a provider's quality level, then then entire concept seems ill-defined. I.e., when you refer to the "best providers", what does that even mean?

> Your views on bodily autonomy seem bizarre. Patients own their own bodies and have the right to make fully informed decisions about what happens to them

Their statement only seem bizarre when you view it from the perspective of individualism. Another way to look at it is - let me (ab)use Star Trek quote: "The needs of the many outweigh the needs of the few". You seem to make absolute statements that in fact are relative and rooted in one ideology you chose to follow.

> If a provider's or facility's (or entire speciality's) numbers look bad, that's important information for the consumer to possess. Hiding the numbers should not be an option.

Forgive me the harshness - I believe this is a simplistic way of looking at the problem. OP clearly showed systemic forces at play, and those are important to consider. You're saying "hiding the numbers", as if they were perfect numbers hidden away in a safe. Problem is there are not, and even when they might be very nuanced. A point raised by OP: "best providers (particularly surgeons) will engage in metrics arbitrage by refusing to treat patients" - I think it is a perfectly reasonable threat, which you chose to ignore.

> Or do you think that car crash test results, and airplane crash data, and health department inspection findings should also be kept secret?

Again going back to OP's example - doctor's might be incentivised not to treat patients which make their numbers look bad. None of your examples share similarity in this sense (and possibly in many more).

>doctor's might be incentivised not to treat patients which make their numbers look bad.

If your concern is with metrics arbitrage, then it is not necessary to use quantitative metrics to satisfy concerns about bodily autonomy. One might instead require providers to furnish anonymized records of all the adverse events their patients have experienced, along with any mitigating factors they think absolve them of responsibility.

I think the issue of quantification is actually a red herring. A simpler example will indicate whether or not there are differences in our ethical intuition. I am scheduled for a "routine" surgery, but yesterday's patient who was having the same surgery by the same doctor had a major artery sliced, bled out and died. All the staff are aware of what happened, but no one tells me. In fact, even if I ask, people are instructed to say nothing. If I possessed this information, I would almost certainly decide not to proceed. I go ahead with the procedure based on the understanding that death is a much more remote possibility than those treating me happen to believe. Is this or is this not a violation of my bodily autonomy?

You've got to be kidding. Providers are never going to do the extra work to anonymize data and compile the reports you're looking for. We already have a shortage of providers. Who exactly do you think is going to pay for that extra work? I've dealt with clinical records firsthand and in the general case it's simply impossible to automate anonymization. Plus coding all those qualifiers and mitigating factors takes a huge amount of time.

>Who exactly do you think is going to pay for that extra work?

If it puts out of business doctors and facilities (perhaps even specialties?) whose patients agree to care based on grossly inaccurate understandings of their track record, then it would more than pay for itself.

Then we're fortunate that you're not in charge of anything important.

Bodily autonomy is irrelevant here. As a sane adult patient you're always free to decline treatment. You're also welcome to ask your providers for any data they can legally release. But in reality most providers simply don't have the data you're looking for and have no incentive to give it to you even if they did have it. Then if you need care what are you going to do?

You have hit upon the core problem though. Outside of a few limited areas where we have clear evidence-based medicine guidelines there is no reliable quantifiable standard for measuring provider quality.

What about, say a statistician? Good odds that a statistician or mathematician would have a better grasp of what the numbers imply than a medical practitioner.

Based on the doctors I know, the medical world does not have an especially profound understanding of how to deal with large amounts of data. Their specialities are diagnosis or specialist surgery, not data.

If the data is so scary we have to hide it then there are glaring problems that need to be addressed. Sure there are misleading edge cases; but believing that accurate data will be worse than word of mouth is, quite frankly, unwarrented.

I'm an epidemiologist that works on exactly this field, and I can tell you right now, with full access to the data, it's still hard to get a handle on.

Having statistical skills is helpful, but statistics only gives part of the picture. Patients often have bad outcomes despite providers adhering to the best current standard of care. And that doesn't just fall out of the noise because there are persistent differences in the patient populations between providers. We have no reliable way to identify and quantify all of those confounding factors. The data and the clinical research simply doesn't exist yet.

See: When Consumer Reports decided to rate the "Best Hospitals" based on their infection rates, while ignoring all kinds of other factors that meant some random rural clinics were awesome, while Mayo, Johns Hopkins, etc. got kicked in the teeth.

Yes that's part of the problem. But in fairness to Consumer Reports, some of the major teaching hospitals were actually terrible at adhering to EBM guidelines for preventing secondary infections. And this was simple stuff like washing hands and sterilizing treatment sites.

As someone working in one of "America's best hospitals" I can confirm that those consumer reports are a complete joke.

Here's a hypothetical question: Suppose I examine the performance of 100 doctors and I rate ten doctors as being the very best, and ten as being the very worst. Suppose that 9 out of the ten 'best' doctors are indeed the best, but one is in fact just average - she doesn't deserve to be rated in the top 10. Similarly 9 of the ten 'worst' doctors are indeed exceptionally bad, but one is actually no worse than average.

If I were to publish my ratings, one doctor will be unfairly penalized. However, hundreds of patients will benefit by being able to switch to a better doctor. In this hypothetical situation, do the rights of one doctor outweigh the rights of so many patients?

Even as a hypothetical that's an unrealistic question. Since we have a shortage of doctors in many regions and specialties due to price controls and educational pipeline bottlenecks, even if it was possible to rate doctors with a reasonable degree of accuracy (and in reality this generally isn't possible) some patients would still end up getting stuck with the worst performers. The only difference is that instead of the selection being mostly random as it is now, under your hypothetical the patients who would switch to the best doctors would be the wealthiest and best informed; the poor, elderly, and illiterate would continue to get screwed. Is that really the outcome you want?

>Since we have a shortage of doctors in many regions and specialties

We also have record levels of advertising by not only drugmakers but also doctors and hospitals. If there is a shortage of providers, why is so much being spent on demand generation efforts?

Because healthcare industry incentives are aren't aligned to optimize for Quality Adjusted Life Years per dollar. Some procedures and drugs are highly profitable because of high demand by patients with money, but those aren't necessarily the ones that society actually needs.

Now who is being unrealistic? I accept that demand for medical services is not as elastic as in other industries, but to suggest that exposing a group of poorly-performing doctors would have no effect on their patient numbers seems rather far fetched. Even if that were the case, the sudden loss of rich patients might make a few of these doctors consider early retirement.

The problem is not randomly adding a couple of minutes that you can average out again, but systemic biases like the one your parent mentioned: difficult cases are treated by different doctors than easier cases.