Comment by NoImmatureAdHom | Hacker News Reader

Comment by NoImmatureAdHom | Hacker News Reader

NoImmatureAdHom

It isn't exactly news that doctors with better test scores are better doctors, but this is additional evidence. The article doesn't touch on race, but very deliberately. To anyone on the inside, the silence is deafening.

In the U.S. med schools been matriculating many unqualified "underrepresented minority" (black, hispanic, native American, Hawaiian) medical students for a long time. This is unfair to patients and doctors, especially competent brown doctors, because it is now the case that you get a very strong signal about how how good a doctor is simply by the color of his or her skin. Which is messed up.

AAMC has the data (https://www.aamc.org/data-reports/students-residents/data/fa... , table A-18). This is after the 2023 Supreme Court decision, so the spreads are a little wider in e.g. 2022 data. MCAT scores range from a minimum of 472 to a max of 528, which is stupid and a deliberate tactic to make the differences between groups seem small. Subtracting 472 from each average score, 2024 average MCAT scores look like this for matriculants:

41.9: Asian

40.2: White

36.9: Hawaiian

34.4: Black

33.9: Hispanic

31.3: American Indian

These are very large differences which you can absolutely expect to show up in doctor performance. Everyone has to pass the same boards during / after med school, but that's just going to cut out some of the worst. Among those who pass, the unqualified minority students who were admitted to med school because of their skin color will still be concentrated at the bottom of the distribution.

Do you know what they call the guy who finished last in his med school class? "Doctor".

Was wondering how long I’d have to scroll for this. The reality is that it’s unhealthy not to be “racist” when selecting health care providers right now due to historical policies like this.

When the right takes swipes at “DEI”, going after bar lowering in medical school is very high on the list of legitimate targets for them to attack. I don’t want to care about the race of my doctor, but do gooders gave me no choice by passing so many bad doctors.

Did they pass bad doctors? The first post referenced entrance exams but cited no data about those that actually complete their medical training.

You could probably back out at least some bounds from the data here: https://www.aamc.org/data-reports/students-residents/report/...

This: https://www.sciencedirect.com/science/article/abs/pii/S00904... Suggests MEDIAN USMLE step 1 scores for White, Asian, Hispanic/Latino, and Black applicants were 242, 242, 237, and 232. It's urology specific, and practice specific, though.

This: https://onlinelibrary.wiley.com/doi/full/10.1002/hsr2.161 Says The mean (±SD) USMLE step 1 score was significantly greater among non-[Black or Hispanic] applicants as compared to URiM applicants (223.7 ± 19.4 vs 216.1 ± 18.4, P < .01, two-sample t-test). This is at a specific medical school.

But more generally...imagine what would have to be true for us to go from BIG differences in g-loaded test performance to small / no differences. Either people fundamentally change somehow (get smarter / dumber), people's test scores systematically differ because they e.g. got better / worse at "tests" or something, independent of their underlying knowledge of the content or abilities, or it's attrition (e.g., very many minority med students wash out, leaving only those who should have been admitted in the first place).

None of those things seem plausible to me. The little glimpse we have from the two studies above is consistent with the obvious thing happening. Things are mostly the same, though I'd bet URM have higher wash-out rates, so differences get attenuated somewhat by the time they're practicing. Of course, URM vs non-URM will sort differently into specialties and geographies so there's that...you'll see bigger or smaller differences depending on how they sorted. A good question, as well, is why the USMLE people don't split reporting by race. I bet one of the reasons is they'd get a lot of flak because there would be big disparities. And good on them (maybe!) because one reason they might care about that is they want to produce good doctors, and watering down their test won't help with that.

> The article doesn't touch on race, but very deliberately. To anyone on the inside, the silence is deafening.

??? The NPI registry doesn't indicate the race of registered providers, only their sex. Really bizarre to call a limitation of the available data "deliberate".

It's possible to put together multiple data sources. There are certain things everyone reading this will already know. It's like reporting "educational attainment" rather than g or IQ in studies...everyone knows what it implies, you just can't say it. Anyway:

1) Board scores are strongly linked to patient outcomes (this paper)

2) We already know test scores vary strongly with observable characteristics like race

3) It's a very safe bet that board scores vary with race in the same way that MCAT scores vary with race

Therefore,

4) We can have a very good idea of how good a doctor is based on observable characteristics like race

Which is a thing the article immediately, obviously, and loudly implies but of course couldn't say for fear of censorship, losing jobs, etc.

Either you want me to make conclusions based on data or you don't. If you want me to make conclusions based on any of the data you provide, then you must provide all the data necessary to make an end-to-end connection to your claim. You can't use a patchwork of studies and say things like "it's a very safe bet" and "we can have a very good idea" to "put together multiple data sources". That's not science, that's "trust me bro".

Show the actual hard data that correlates board certification exam results and race for this study. As it stands now, we can at best associate this with physician sex.

If I'm using your logic, then, without any evidence whatsoever, I can say that obviously because the correlation between MCAT scores and Step 2 scores is weakened compared to Step 1, then it's a "very safe bet" that there will be little to no correlation for Step 3 and almost entirely eliminated by the time they take the BCE.

Or I can be rigorous and not make data points up in my head to fit some worldview.

> Which is a thing the article immediately, obviously, and loudly implies but of course couldn't say for fear of censorship, losing jobs, etc.

No, it doesn't, because it can't, because they don't have any information about the races of the physicians in the study.

I appreciate you engaging.

"Or I can be rigorous and not make data points up in my head to fit some worldview."

It seems clear to me that you're sticking your head in the sand, not me. I'm believing the thing that is dangerous to believe, not you. I believe it because it's obviously true.

"actual hard data" would be best. It would be best if we just had board scores split by race. But we don't. We do, however, have lots of other information that makes it very, very clear that there will be significant disparities by race in USMLE boards in pretty much exactly the same pattern we see in MCAT scores.

Here's the meat of it, you can look to the other comments here for the potatoes:

This: https://www.sciencedirect.com/science/article/abs/pii/S00904... Suggests MEDIAN USMLE step 1 scores for White, Asian, Hispanic/Latino, and Black applicants were 242, 242, 237, and 232. It's urology specific, and practice specific, though.

This: https://onlinelibrary.wiley.com/doi/full/10.1002/hsr2.161 Says The mean (±SD) USMLE step 1 score was significantly greater among non-[Black or Hispanic] applicants as compared to URiM applicants (223.7 ± 19.4 vs 216.1 ± 18.4, P < .01, two-sample t-test). This is at a specific medical school.

...and this is just the result of a casual search.

The people who are best at this sort of thing are economists. They are trained to do causal inference based on patchy, far-from-perfect data. It's totally normal to come to a conclusion (even a very strong one!) using a "patchwork of studies". That's just life. You don't usually get "actual hard data". It's very clear what the pattern in the USMLE data would look like. I bet the effect size would be a little attenuated.

Your epistemic stance, which seems to be "Well we don't have perfect, incontrovertible proof, which means we must act like we don't know anything at all!" is unworkable. You don't do this, I don't do this, the world doesn't permit of this. As a rhetorical move, I can see where you're coming from. It gives you license to not think about the hard thing, and to punish those around you who might. But I'd argue that's not a way forward for us as a whole.

Thanks for doing the analysis.

The GP has chosen not to reply, but I would hope they learn from your comment. Especially this part:

> Your epistemic stance, which seems to be "Well we don't have perfect, incontrovertible proof, which means we must act like we don't know anything at all!" is unworkable. You don't do this, I don't do this, the world doesn't permit of this. As a rhetorical move, I can see where you're coming from. It gives you license to not think about the hard thing, and to punish those around you who might. But I'd argue that's not a way forward for us as a whole.

This is the data on entrance exams, not exit exams. Is there any data that actually shows minorities that finish med school and pass boards are any less competent?

The entire point of these programs is to make up for the lack of educational access for minorities by giving them a chance to prove themselves by admitting them with lower scores. But if they complete the same program, doesn't that mean they are just as good?

Now, in light of this study, it would be super interesting if this divide holds up in exit exam scores. But until we actually have that data, I'm not sure your claim is valid.

As above:

You could probably back out at least some bounds from the data here: https://www.aamc.org/data-reports/students-residents/report/...

This: https://www.sciencedirect.com/science/article/abs/pii/S00904... Suggests MEDIAN USMLE step 1 scores for White, Asian, Hispanic/Latino, and Black applicants were 242, 242, 237, and 232. It's urology specific, and practice specific, though.

This: https://onlinelibrary.wiley.com/doi/full/10.1002/hsr2.161 Says The mean (±SD) USMLE step 1 score was significantly greater among non-[Black or Hispanic] applicants as compared to URiM applicants (223.7 ± 19.4 vs 216.1 ± 18.4, P < .01, two-sample t-test). This is at a specific medical school.

But more generally...imagine what would have to be true for us to go from BIG differences in g-loaded test performance to small / no differences. Either people fundamentally change somehow (get smarter / dumber), people's test scores systematically differ because they e.g. got better / worse at "tests" or something, independent of their underlying knowledge of the content or abilities, or it's attrition (e.g., very many minority med students wash out, leaving only those who should have been admitted in the first place).

None of those things seem plausible to me. The little glimpse we have from the two studies above is consistent with the obvious thing happening. Things are mostly the same, though I'd bet URM have higher wash-out rates, so differences get attenuated somewhat by the time they're practicing. Of course, URM vs non-URM will sort differently into specialties and geographies so there's that...you'll see bigger or smaller differences depending on how they sorted. A good question, as well, is why the USMLE people don't split reporting by race. I bet one of the reasons is they'd get a lot of flak because there would be big disparities. And good on them (maybe!) because one reason they might care about that is they want to produce good doctors, and watering down their test won't help with that.

Reply on news.ycombinator.com