AI has the worst superpower medical racism

> Racial identity is a social, legal, and political construct that consists of our own perceptions of our race, and how other people see us.

This seems a pretty US-centric conception. I for one wouldn't know how to assign a "racial identity" to myself, and most of my acquitances would surely say the same thing. I doubt that this study holds much meaning outside the USA.

Now if it was about genetic ancestry it would make some sense (and not really be surprising). But if it is about "self-reported identity" I guess it's only finding the correlation of that with actual genetically transmitted features.

I can think of some reasons why this might be a problem, but the article doesn't seem to make a strong case? Without extrapolating a bit with my imagination, the only conclusion I can draw is "medical ai considers race in it's diagnosis, and it is neither a good nor a bad thing". They seem to leave it as a given that it's bad, but tbh as a layman I could imagine arguments both ways[1]. An article from a professional like this one is where I would expect to find them, along with a summary that effectively "picks a side", with a justification for picking that side.

1: Eg: using race as a clue that some reading might indicate a disease common in that race could be good, on the other side there might be some very expensive scan that could throw off the diagnosis because the training data is skewed towards old white men by the sheer price of doing the scan.

I'm sympathetic to the questions being asked, but I think the author is too quick to assume the worst here.

The key assumption is shown in the diagram that "shows a hypothetical pneumonia detection model that, during optimization, has learned to recognize racial identity." -- the author asserts that this is obviously incorrect and dangerous.

Look at the HPF 50 results, and the fact that the location being imaged is nearly irrelevant -- it's clear that image shape features are irrelevant to the results. It seems like the signal itself is picking up (is modified by) racial indicators. If that's true, it only makes sense that being good at detecting pneumonia is going to require correcting for that signal. That makes (this use of) AI anti-racist in a very important way.

This is nonetheless a fascinating result!

I sometimes think 'race' is nothing more than a fantastical vanity construct. Really its tribalism. Furthermore, I find it hard as well to comprehend how it holds weight in the medical industry. Race is not real science. Race is entertainment science. AI is mostly entertainment science.

This is bewildering and concerning. I absolutely applaud the authors for releasing their code and asking the public to have a look, as well as raising the alarm. Given the degree of due diligence they've performed, it seems too much to hope that there's just some really nightmarish data leakage occurring; but even if that were the case, the difficulty of identifying its origin or even determining the true characteristics of what the models are doing is extremely problematic regardless of whether this particular instance of bias can be explained.

Disclaimer: Worth noting that this paper is apparently pre-print [1] and was submitted to the Lancet, who are not particularly reliable in recent times (flaws hydroxychloroquine study [2], stamped out early investigation in COVID lab-leak possibility [3]).

Regarding the blog and paper:

> AI can trivially learn to identify the self-reported racial identity of patients to an absurdly high degree of accuracy

If it is 'trivial' for a model to learn race, then what they are detecting is likely pretty obvious.

In their paper they use 'private and public datasets' [4]. There is probably exactly where the problem occurs. People of the similar race tend to live in groups together, if you collect data in some area, there's a good chance you source data from the same ethnic pool. What it appears they are detecting is the difference between different machines and practices. To test this, they should collect their own medical data on the same machine. They also need to figure out how to remove the bias of the medical staff themselves.

Another assumption is that just because the medical doctors cannot see differences in the anatomy of races, it doesn't mean there isn't one. One of the commentators on this blog indicated that dark skin may affect the absorption of x-rays, for example.

> AI does learn to do this when trained for clinical tasks

Just because AI models can detect race from medical data, doesn't mean they will. They may in the early stages learn such simplistic relationships, but as accuracy increases, it will be forced to look for more complex classifiers/indicators.

Finally, this blog really skirts around the important thing - how does this affect patient care? Is it beneficial to actually have race-based medical plans? We know that some medicines affect different races - for example in pain and pain management [5].

I think the point to be made here isn't 'racism' but the possibility of bias that people training models should be aware of. Some of this bias will have a positive outcome, some will have a negative outcome.

[1] https://www.bibsonomy.org/bibtex/c3e2ede2f835d35c08c54a67095...

[2] https://www.nbcnews.com/health/health-news/lancet-retracts-l...

[3] https://www.washingtonexaminer.com/news/wuhan-lab-collaborat...

[4] https://arxiv.org/pdf/2107.10356.pdf

[5] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3654683/