Hacker News Reader

Comment by bravura | Hacker News Reader

This is not overfitting precisely because of the bias/variance tradeoff.

A model overfits if it is unnecessarily COMPLEX for the training data.

If there is bias in the training (and validation and test) data that allows a SIMPLE model to fit the data because of a spurious correlation, that is not overfitting.

View on HN

There is no reason to believe that an X-ray model's correct estimation of age is due to "spurious correlation". Rather, it seems to be "undesirable correlation".

> Rather, it seems to be "undesirable correlation".

More specifically, politically undesirable correlation - as in, "it's there, but its existence upsets some people". It's pretty obvious and self-evident that there are meaningful biological differences related to age, sex, and other demographics. Whether or not they're clinically relevant for a specific diagnosis under question is one thing, but they are clinically relevant for great many diagnoses; trying to "de-bias" reality here will only lead to unnecessary suffering and loss of life.

It seems that the models not only use bone structure (or similar) itself, but improve their prediction with "forbidden" knowledge, such as "green people have an overall lower or higher rate of bone cancer than others" or "people who come to this specialized hospital have bone cancer anyway, so I don't even look at the image"

Now you can say that this is perfectly fine and represents the most likely real-world use case. Or you might prefer a model that looks at the image only, with the implicit assumption that this "forbidden knowledge" will be added by human doctors later on in the pipeline. This is beneficial because the "forbidden knowledge", such as whether patients from Hospital A always have bone cancer, might change overnight! Imagine the hospital gets assigned a new name in the system and the prediction is shit now.

This second, "unbiased" AI system will always have a worse performance, because you lobotomize it when you kill the forbidden knowledge with a sledgehammer. This study just showed that "group fairness" is at odds with optimal predictions" and how much it is at odds.

PS: You might even prefer a society where everyone is worse off, but every protected group is equally bad off. You'd also ban the humans from applying the forbidden knowledge. Whether that is desirable, is, of course, out of the scope of the paper.

> Or you might prefer a model that looks at the image only

These models are only looking at the images. They are inferring demographics.

Yes, these things can be a factor for a specific diagnosis but why use AI when (just) going back to conditional probabilities based on groups instead of making a true individual diagnosis? You want each diagnosis to be correct and not just a good average.

You always diagnose on conditional probabilities. The diagnosis is conditioned on your belief in occurrence of symptoms, which is conditioned on the observations and results of tests you make. In an ideal case, you observe well enough to make a definite diagnosis; in real case, there's always some uncertainty, plus you can't do all the tests simultaneously - which is where all those proxy factors like demographics are useful: they help prioritize tests and narrow down the diagnosis quicker.

You don't always diagnose on conditional probabilities (a simple example is looking at an x-ray of a broken bone - no priors needed to spot the broken bone).

Your knowledge guides, but it also doesn't (or shouldn't) blind you.

Do the sorts of diseases ML is being used to detect have a flat incidence profile over age? Even if they do, negating other diagnoses that are age dependent would still mean ML models would acquire a measure of patient age, say.

Almost every noncommunicable disease of adulthood becomes more common with age. (The diagnoses in this paper were things like "cardiomegaly" which are mostly just X-ray findings and, while they have ICD codes, are not a meaningful diagnosis that a practicing physician would care about.)

Reply on news.ycombinator.com