Resident physicians' exam scores tied to patient survival
hms.harvard.edu
hms.harvard.edu
Ultimately it boils down to what a given surgeon practiced in their career.
Is this a concept that exists? I thought "thin tailed" and "fat tailed" were defined by contrast to a normal distribution.
> Excess kurtosis, typically compared to a value of 0, characterizes the “tailedness” of a distribution. A univariate normal distribution has an excess kurtosis of 0. Negative excess kurtosis indicates a platykurtic distribution, which doesn’t necessarily have a flat top but produces fewer or less extreme outliers than the normal distribution. For instance, the uniform distribution (ie one that is uniformly finite over some bound and zero elsewhere) is platykurtic. On the other hand, positive excess kurtosis signifies a leptokurtic distribution. The Laplace distribution, for example, has tails that decay more slowly than a Gaussian, resulting in more outliers.
(Emphasis added.)
You can't have a bell curve with thin tails, because a bell curve has standard tails by definition.
even if hand skills is orthogonal to test scores, test scores could still be highly correlated with outcomes
When my mom need her second urgent brain surgery, one of the nurses, a friend of a friend, warned us that one of the surgeons on staff that night had dreadful results with basically any procedure. and that if we were stuck with him, that refusing the surgery until another one was able to come in. If the floor nurses know, the head on neurology knows, and yet, they were still OK letting them operate.
Any functioning system just has to eject surgeons that get bad outcomes. It's not as if they have to stop practicing medicine, but move to something a little less dangerous.
When it comes to bad doctors, I think a lot of people in the medical field would agree "you know one when you see one." But when you're talking about putting professional restrictions on someone, you need objective criteria.
The naive way, counting up bad outcomes, leads to a system where surgeons are incentivized to decline any case that looks technically difficult or where the patient has lots of preexisting conditions that put them at risk for complications after the surgery. We already see this to a degree in transplant surgery, where outcomes are followed closely to avoid wasting organs.
That said, I think true incompetence is pretty rare. I can't think of a single doctor I 've worked with professionally where I'd be concerned if I found out they were taking care of one of my immediate family.
Already happens all the time, although it's at the discretion of the first surgeon. Again, it's difficult to formulate objective criteria for when a surgeon should forward a case on to someone else.
I don't believe that surgeons obviously over operate.
In my career I've seen
- A case where the surgical team declined to operate for endocarditis with congestive heart failure, despite the fact that the society guidelines recommend surgery in that scenario
- A case where the surgical team declined to operate for a spinal cord injury that left a patient paralyzed from the neck down (and dependent on machines to breathe due to the paralysis affecting their diaphragm)
- A case where the surgical team declined to operate on an abscess even after a patient's blood stream infection failed to clear after two weeks of the strongest IV antibiotics
In all the first and third cases, the disease turned terminal after our surgical team declined to operate. In the second example the patient opted to die rather than live the rest of their life on a ventilator and I was left with the responsibility of arranging hospice for the patient.
I'll admit that these are extreme cases, but my point is the patients and family members in those cases likely had a very different view about whether surgeons should decline high risk surgeries as often as they do, let alone more often.
Part of this is the AMA cartel artificially limiting the supply of residencies.
Something has to change drastically in how medicine is organised because it's not working in its current iteration as the difficulty goes up and up.
They run pharmacy benefits management, health insurance and drug distribution.
The estimate is 4-5 Trillion flows throw these firms. Which is larger than the GDP of India. So this gigantic structure has emerged that doesn't really make too much profit btw (very similar to Amazon Platform Economics) but is layer upon layer upon layer of cash flow passing through middlemen.
Drastic change requires new ideas about what do we do about all these middlemen who shape the environment on top of which everything exists.
It's impossible to fix overly complex systems.
Simplify, simplify, simplify, and then the fixes become trivial.
In the US case, that means banning most of the middle-layers.
Banning the middle layers here (absent other changes) just means that the companies that replace their spots in the top 20 will be vertically integrated conglomerates that manufacturer, distribute, prescribe and provide insurance (i.e. payment plans) for pharmaceutical drugs.
Except these companies are already vertically integrated, to a large degree. All the biggest insurers have their own in-house PBMs.
CVS (the parent company of Aetna) has Caremark.
Cigna has Express Scripts.
Anthem (fine, Elevance) has CarelonRx.
UnitedHealth Group has Optum.
I don’t disagree that the system requires change and is extremely complex, however.
The real problem is that it’s nearly impossible to “scale” healthcare and keep it personalized, and people want personalized healthcare - because that’s shown to be more effective healthcare. Doctors can only see a limited number of patients a day, and they need to be paid some compensation commensurate with their skills and efforts. That alone makes it hard for everyone “healthy” to see a doctor often enough and for long enough to get deeply personal care. Most people realistically can pay out of pocket for preventative care. $100-200/yr for an American isn’t crazy. Even most drugs are super affordable out of pocket if the profit margins are kept low (which is started to be available, bits at a time).
The real complexity, of course, is the long-tail where a few people get cancer and car accidents and other serious conditions which swamp the costs of everything else.
What do you call the person that came bottom of their class in Med School?
Doctor.
A study that shows the board test is effective, sponsored by the board?
I might be reassured by more detailed statistics about the analysis. Even top 25% vs bottom 25% - how much actual variation in score are we talking about? What is the probability that someone scoring in the top 25% is actually in the top/bottom 25%? We imagine a big gap but that’s not necessarily true. Consider exam scores of 85 90 90 95…
I'd be more open to this line of reasoning if physician's salaries had kept pace with inflation over the last 30 years and if if we hadn't tacitly accepted a much, much lower standard of training in the form of DNPs, CRNAs and PAs who are now practicing independently in a lot of regions. You can't demand that people make extraordinary sacrifices without extraordinary compensation.
For contrast, most European countries have a much longer post-residency training process that is more humane. Caveat being that students enter medical school directly from high school and don't have student loans.
It's also worth pointing out that in the US a LOT of those 100 hours are not spent in direct patient care. They're spent doing chores ('scut') that are not directly tied to patient care. Think: Calling insurance companies for prior authorization for your supervisor or filling out FMLA paperwork for one of your supervisors' patients. As a resident you don't have the ability to say "no" to these tasks.
It's mixed, though. We don't know how much "squeezing as much training" helps or hinders future performance. We do know that sleep debt hurts retention of new knowledge and skills.
So I'm not positive whether "50% more training, but with not enough sleep during most of the interval" will result in better outcomes.
> I'd be more open to this line of reasoning if physician's salaries had kept pace with inflation over the last 30 years
Doctors in the US are artificially scarce and artificially expensive compared to the rest of the world. The artificial scarcity of residencies also contributes to the unusually harsh residency work conditions.
If a hellish residency improves patient outcomes by 0.1%, at the expense of every single resident suffering twice as much as they need to (and likely leading to some stimulant addictions and deaths among the resident/doctor population), that's not a fair tradeoff.
Medical workers don't exist solely to sacrifice themselves for others; they are humans also and their needs should be weighed as important like everyone else's.
As it so happens I think some of the strain of medical residency is related to supply shortages in the health care industry. If it's not crystal clear that working 80+ hours per week is necessary to significantly improve patient outcomes, and it is clear that working 80+ hours per week makes a lot of people choose other careers (limiting supply artificially), then reform here is imperative.
Apart from the bad real-time cognitive effects, long-term memory retention is dependent on regular, sustained sleep.
Just to point out the obvious, people doing 80 hrs/week for 2 years (lower end of residency term I believe) are going to have twice as much 'experience' as people doing 40hrs/week for 2 years.
I suspect most of us here know more hours worked doesn't directly correlate with more retention of information and best practices, but that's the thinking.
I'm arguing that even if 80hrs/week residencies was the optimal amount of pressure to turn our fledgling residents into battle-hardened physicians, if you can get 99% of the effect with 40hrs/week, maybe do that instead. And again, I'm not even suggesting this is actually the case.
https://magazine.columbia.edu/article/cocaine-addict-who-cha...
The Mayans used coca leaves to get more work out of their people.
I guess medical residency is kind of like a hazing ritual. Today's doctor's are like I went though it, why can't you?
My general impression is that the evidence overall is really not supportive of harsher residencies in terms of patient outcomes. I also think that rigor does not have to mean masochism or hubris; there seems to be this assumption that any change to residencies would mean dumbing it down or making it easier, as opposed to improving things overall. I'm also a little skeptical of minor tweaks to residency that might have happened somewhere now being representative of a more wholesale restructuring.
The often unacknowledged factor in the background is that hospitals and residency locations are getting free labor with no chance of repudiation of their situation by workers. Hospitals are getting physicians whose salaries are paid for by the federal government, where those physicians are essentially unfree to move if they're unhappy. So of course there's going to be an attempt to milk them for everything. It gets whitewashed as "selflessness" and physicians are encouraged to boast about it or something, instead of calling it out as exploitation. No physician wants to make that claim, for a whole host of reasons, even if it is true.
Imagine what would happen if hospitals had to bear the costs of residency training completely, like just about any other healthcare profession, and residents were able to move freely like most employees.
I get despondent about so much in US healthcare. There's so much focus on invoice costs per se, and payment by insurers, and not enough on monopolies in service delivery, and problems with educational structures. Any attempt to address these issues is met with resistance by various groups with conflicts of interest, who aren't called out on these conflicts of interest.
Another thing about residencies constantly on my mind from other settings (institutional tracking hours in the moment versus recalled hours later) as well as personal experiences with residency in the past is that people are notoriously bad about reporting past work hours and conditions, and tend to exaggerate. I'm not saying that anyone in particular is necessarily being dishonest in describing their residency experience, but I suspect there has been drift over time in conditions that reflects a kind of biased memory of things on the part of residency directors. "I worked 120 hours a week" when that wasn't actually the case, or is distorted, then becomes residency policy for the next generation.
Sometimes I feel like the logical conclusion, given the way these discussions go sometimes, is the only one being legally able to practice is someone with an MD who has completed a residency working 140 hours a week for 6 years, with perfect board exam scores. It just doesn't add up.
> he was able to hide his addiction under a veil of eccentricity and a pyramid of residents
Which means "created an environment to allow himself to be high at work" to me. It's not impossible that he held it off at home, but I don't see why he would.
Also, he's clearly Dr. House; Ctrl-F "Leaving much"
High exam scores are an indication of discipline and good prioritisation - factors that evidently reflect on the physician's professional performance.
Whether it's surprising or not, it's up to you. But it's something that should be measured once in a while.
Likewise, there's been an element of testing = racial discrimination in all sorts of fields, such as:
https://fairtest.org/article/legal-attack-biased-firefighter...
And:
https://www.theguardian.com/world/2009/jun/29/connecticut-fi...
The fact that this study alleges a direct link between exam scores and performance is itself bucking the zeitgeist.
Healthcare worker here. Sheriffofsodium did a great video poking holes at this study: https://youtu.be/JKS9Y-nCnKs?si=VPsUNSoepltbg4Hu
It's 45 minutes so I don't expect people to watch it, but he makes several important points, including:
- This study was performed by USMLE insiders, the only ones with access to this private data. USMLE does not share this data publicly so it's impossible to verify.
- As the USMLE makes millions of dollars from these exams, they have a clear conflict of interest.
- The differences in patient outcome are AT BEST of marginal clinical significance, which the authors of the study even state in the paper.
There is better scientific evidence that female surgeons have better patient outcomes on average: https://pubmed.ncbi.nlm.nih.gov/37647075/
In the U.S. med schools been matriculating many unqualified "underrepresented minority" (black, hispanic, native American, Hawaiian) medical students for a long time. This is unfair to patients and doctors, especially competent brown doctors, because it is now the case that you get a very strong signal about how how good a doctor is simply by the color of his or her skin. Which is messed up.
AAMC has the data (https://www.aamc.org/data-reports/students-residents/data/fa... , table A-18). This is after the 2023 Supreme Court decision, so the spreads are a little wider in e.g. 2022 data. MCAT scores range from a minimum of 472 to a max of 528, which is stupid and a deliberate tactic to make the differences between groups seem small. Subtracting 472 from each average score, 2024 average MCAT scores look like this for matriculants:
41.9: Asian
40.2: White
36.9: Hawaiian
34.4: Black
33.9: Hispanic
31.3: American Indian
These are very large differences which you can absolutely expect to show up in doctor performance. Everyone has to pass the same boards during / after med school, but that's just going to cut out some of the worst. Among those who pass, the unqualified minority students who were admitted to med school because of their skin color will still be concentrated at the bottom of the distribution.
Do you know what they call the guy who finished last in his med school class? "Doctor".
??? The NPI registry doesn't indicate the race of registered providers, only their sex. Really bizarre to call a limitation of the available data "deliberate".
1) Board scores are strongly linked to patient outcomes (this paper)
2) We already know test scores vary strongly with observable characteristics like race
3) It's a very safe bet that board scores vary with race in the same way that MCAT scores vary with race
Therefore,
4) We can have a very good idea of how good a doctor is based on observable characteristics like race
Which is a thing the article immediately, obviously, and loudly implies but of course couldn't say for fear of censorship, losing jobs, etc.
Show the actual hard data that correlates board certification exam results and race for this study. As it stands now, we can at best associate this with physician sex.
If I'm using your logic, then, without any evidence whatsoever, I can say that obviously because the correlation between MCAT scores and Step 2 scores is weakened compared to Step 1, then it's a "very safe bet" that there will be little to no correlation for Step 3 and almost entirely eliminated by the time they take the BCE.
Or I can be rigorous and not make data points up in my head to fit some worldview.
> Which is a thing the article immediately, obviously, and loudly implies but of course couldn't say for fear of censorship, losing jobs, etc.
No, it doesn't, because it can't, because they don't have any information about the races of the physicians in the study.
"Or I can be rigorous and not make data points up in my head to fit some worldview."
It seems clear to me that you're sticking your head in the sand, not me. I'm believing the thing that is dangerous to believe, not you. I believe it because it's obviously true.
"actual hard data" would be best. It would be best if we just had board scores split by race. But we don't. We do, however, have lots of other information that makes it very, very clear that there will be significant disparities by race in USMLE boards in pretty much exactly the same pattern we see in MCAT scores.
Here's the meat of it, you can look to the other comments here for the potatoes:
This: https://www.sciencedirect.com/science/article/abs/pii/S00904... Suggests MEDIAN USMLE step 1 scores for White, Asian, Hispanic/Latino, and Black applicants were 242, 242, 237, and 232. It's urology specific, and practice specific, though.
This: https://onlinelibrary.wiley.com/doi/full/10.1002/hsr2.161 Says The mean (±SD) USMLE step 1 score was significantly greater among non-[Black or Hispanic] applicants as compared to URiM applicants (223.7 ± 19.4 vs 216.1 ± 18.4, P < .01, two-sample t-test). This is at a specific medical school.
...and this is just the result of a casual search.
The people who are best at this sort of thing are economists. They are trained to do causal inference based on patchy, far-from-perfect data. It's totally normal to come to a conclusion (even a very strong one!) using a "patchwork of studies". That's just life. You don't usually get "actual hard data". It's very clear what the pattern in the USMLE data would look like. I bet the effect size would be a little attenuated.
Your epistemic stance, which seems to be "Well we don't have perfect, incontrovertible proof, which means we must act like we don't know anything at all!" is unworkable. You don't do this, I don't do this, the world doesn't permit of this. As a rhetorical move, I can see where you're coming from. It gives you license to not think about the hard thing, and to punish those around you who might. But I'd argue that's not a way forward for us as a whole.
The GP has chosen not to reply, but I would hope they learn from your comment. Especially this part:
> Your epistemic stance, which seems to be "Well we don't have perfect, incontrovertible proof, which means we must act like we don't know anything at all!" is unworkable. You don't do this, I don't do this, the world doesn't permit of this. As a rhetorical move, I can see where you're coming from. It gives you license to not think about the hard thing, and to punish those around you who might. But I'd argue that's not a way forward for us as a whole.
When the right takes swipes at “DEI”, going after bar lowering in medical school is very high on the list of legitimate targets for them to attack. I don’t want to care about the race of my doctor, but do gooders gave me no choice by passing so many bad doctors.
This: https://www.sciencedirect.com/science/article/abs/pii/S00904... Suggests MEDIAN USMLE step 1 scores for White, Asian, Hispanic/Latino, and Black applicants were 242, 242, 237, and 232. It's urology specific, and practice specific, though.
This: https://onlinelibrary.wiley.com/doi/full/10.1002/hsr2.161 Says The mean (±SD) USMLE step 1 score was significantly greater among non-[Black or Hispanic] applicants as compared to URiM applicants (223.7 ± 19.4 vs 216.1 ± 18.4, P < .01, two-sample t-test). This is at a specific medical school.
But more generally...imagine what would have to be true for us to go from BIG differences in g-loaded test performance to small / no differences. Either people fundamentally change somehow (get smarter / dumber), people's test scores systematically differ because they e.g. got better / worse at "tests" or something, independent of their underlying knowledge of the content or abilities, or it's attrition (e.g., very many minority med students wash out, leaving only those who should have been admitted in the first place).
None of those things seem plausible to me. The little glimpse we have from the two studies above is consistent with the obvious thing happening. Things are mostly the same, though I'd bet URM have higher wash-out rates, so differences get attenuated somewhat by the time they're practicing. Of course, URM vs non-URM will sort differently into specialties and geographies so there's that...you'll see bigger or smaller differences depending on how they sorted. A good question, as well, is why the USMLE people don't split reporting by race. I bet one of the reasons is they'd get a lot of flak because there would be big disparities. And good on them (maybe!) because one reason they might care about that is they want to produce good doctors, and watering down their test won't help with that.
The entire point of these programs is to make up for the lack of educational access for minorities by giving them a chance to prove themselves by admitting them with lower scores. But if they complete the same program, doesn't that mean they are just as good?
Now, in light of this study, it would be super interesting if this divide holds up in exit exam scores. But until we actually have that data, I'm not sure your claim is valid.
You could probably back out at least some bounds from the data here: https://www.aamc.org/data-reports/students-residents/report/...
This: https://www.sciencedirect.com/science/article/abs/pii/S00904... Suggests MEDIAN USMLE step 1 scores for White, Asian, Hispanic/Latino, and Black applicants were 242, 242, 237, and 232. It's urology specific, and practice specific, though.
This: https://onlinelibrary.wiley.com/doi/full/10.1002/hsr2.161 Says The mean (±SD) USMLE step 1 score was significantly greater among non-[Black or Hispanic] applicants as compared to URiM applicants (223.7 ± 19.4 vs 216.1 ± 18.4, P < .01, two-sample t-test). This is at a specific medical school.
But more generally...imagine what would have to be true for us to go from BIG differences in g-loaded test performance to small / no differences. Either people fundamentally change somehow (get smarter / dumber), people's test scores systematically differ because they e.g. got better / worse at "tests" or something, independent of their underlying knowledge of the content or abilities, or it's attrition (e.g., very many minority med students wash out, leaving only those who should have been admitted in the first place).
None of those things seem plausible to me. The little glimpse we have from the two studies above is consistent with the obvious thing happening. Things are mostly the same, though I'd bet URM have higher wash-out rates, so differences get attenuated somewhat by the time they're practicing. Of course, URM vs non-URM will sort differently into specialties and geographies so there's that...you'll see bigger or smaller differences depending on how they sorted. A good question, as well, is why the USMLE people don't split reporting by race. I bet one of the reasons is they'd get a lot of flak because there would be big disparities. And good on them (maybe!) because one reason they might care about that is they want to produce good doctors, and watering down their test won't help with that.
For example, maybe burn unit care is high-mortality and low-barrier, compared to sleep medicine which is low-mortality and high-barrier (I don't know how accurate this is, just providing some hypotheticals for clarity)
>The researchers compared outcomes for patients within the same hospitals who were cared for by doctors with different exam scores. This allowed the researchers to eliminate, or at least minimize, the effect of differences in patient populations, hospital resources, and other variations that might influence the odds of patient death or readmission, independent of a doctor’s performance.
> The researchers compared outcomes for patients within the same hospitals who were cared for by doctors with different exam scores. This allowed the researchers to eliminate, or at least minimize, the effect of differences in patient populations, hospital resources, and other variations that might influence the odds of patient death or readmission, independent of a doctor’s performance.
Good docs are humble, meticulous and knowledgeable. Stellar docs are excellent communicators.
. Many inputs can be placed in the machine by physicians, and the outputs are known. The biological machines can easily be isolated from environment, or monitored with high technology, and assigned numbers in databases to be processed in data centers.
Value is extracted from the biological machines mostly from government and 3rd party sources, so there is no real need to rely on machines having a means or will of their own.
There is no compelling reason to treat humans any different from automobiles for the purposes of medicine and medical treatment. In fact humans are less genetically diverse than motor vehicles, and A new model year will always produce a bumper crop of lemons to work on.
> Many inputs can be placed in the machine by physicians, and the outputs are known. The biological machines can easily be isolated from environment, or monitored with high technology, and assigned numbers in databases to be processed in data centers.
We aren't even close to that level of understanding.