Hacker News Reader

Comment by twanvl | Hacker News Reader

> The problem is that they fail differently than humans do

That is a great argument for giving such a model as an aid to a human doctor. Together they will be better then either one alone.

View on HN

In Thinking, Fast and Slow -- the author details a double blind trial where the did this. It was worse with humans and AI than with just AI. Humans think they can use AI as a guide and move it in the right direction. But the movements they made, on average, were bad.

Surely in this type of instance (looking at a scan to answer a yes/no question) the human and AI act independently, with the computer being a useful aid because it separately picks up a few of the human's false negatives. Assuming false negatives are a lot worse than false positives, this can only be a good thing.

If they lead to an unnecessary mastectomy then false positives are pretty bad. Not as bad as dying, obviously, but still a severe blow to a woman's identity and sense of self worth.

It's going to be a hard pill to swallow if you have to tell a woman "sorry, we removed your healthy breast because the computer made a mistake."

Even a false positive that leads to telling the patient that they may have cancer is bad. It leads to a life-long anxiety for many people.

I think the idea of "screening" is that you don't just race off to a mastectomy the minute some AI model goes off. Of course, putting more false positives through a fallible process of review does run the risk you speak of.

It does cause unnecessary biopsies for sure. And some stress on the patients.

It sounds like a smart hospital would run a patient through both human and AI screenings separately, and a different doctor to examine both results and evaluate the discrepancies. This way you would keep the strengths of both approaches, lowering the failure rates, and depending on the countries health care funding can be good business from the hospital's POV as they get to charge for the extra work as well as the better success rates to drive business.

And I wonder what happens if you apply machine learning to looking at the difference between AI and human screening results.

Radiologists are really bad at detection, even after many years of study. That's quite often due to coarse level of details of scans when only large tumors can be observed or recognized with some certainty. Surpassing humans there is not so difficult, but improving accuracy from e.g. 32% to 34% doesn't really sound like a win :(

2% more accuracy could still be millions of people if it's a common enough cancer like breast cancer.

> 32% to 34% doesn't really sound like a win :( We are talking about human lives here, not about beating some CPU benchmark. Detection improvement by 2% is huge in almost any sickness.

remember when ensembles were the cool word before they got erased from collective consciousness and replaced with deep things? it can't even be a decade, was it 2012 or something?

They haven't got erased, but more like subsumed? If you use dropout to train your model that is basically equivalent with using an ensemble of deep neural networks.

That is not even close to the same thing.

If you train an ensemble of models with random dropout, you have an ensemble. Models trained with dropout will still have significant variation from run to run.

> That is not even close to the same thing.

It's a common interpretation: https://arxiv.org/abs/1706.06859

There may be a paper on it, but it’s not a common view.

In particular, this paper neglected to do the obvious thing: ensemble networks trained with dropout. It improves performance over dropout alone.

Why shouldn't you employ an ensemble of deep neural networks?

Correlated errors. Naive averaging will lead to overconfidence and it is not trivial to model the correlation. Boosting is worth a shot though.

My point was that ensembles of deep neural networks are commonly used and yield higher accuracies.

More importantly, what happens if you put a radiologist opinion (or multiple) in such an ensemble?

This. The argument was never to replace doctors. These are valuable tools that augment what doctors can do.

No, the point very much is to eventually replace doctors. You just can't easily get there before first going through a doctor-machine cooperation period.

Automation is a friend of society, but is not a friend of individuals working particular jobs. I think doctors are acutely aware of that.

Reply on news.ycombinator.com