Comment by haldujai | Hacker News Reader

Comment by haldujai | Hacker News Reader

Parent Full thread

AI in medical imaging has been very limited and ineffective thus far.

My institution pays for an expensive lesion detection / mammography CAD software package but it generates so many false positives I don’t use it in my practice. Overall having to review an extra n pseudolesions per scan results in decision fatigue and increases my miss rate, at least in my experience.

There are several issues with AI when it comes to radiology that I can’t address with a single comment but given your question my biggest criticism would be that we don’t have a sensitivity problem in medicine. What we have is largely a specificity problem, but that’s inherent with imaging which is not a ground truth representation of a physical entity but rather how this tissue attenuates photons (for mammo).

AI, like self driving cars, has to be an order of magnitude “safer” which generally translates into decreased specificity.

What I mean by this is that I don’t miss a significant number of breast cancers, in fact it’s very very low. So an AI to improve my detection is pointless to me, what I really want is to recall less patients but AI won’t necessarily help with that because it’s inherent limitations of the imaging modality (scar and cancer look the same).

I think you are in the right track that AI looking at information humans are not currently looking at is the future / next step. Not to replace a task already handled by physicians who have the ability to integrate disparate health records (I.e. does this patient have easier access to breast MRI or biopsy given location/insurance/biopsy schedule/MRI schedule/hospital resources and what are her personal goals of care to help me decide what to do with this lesion) that is currently silo’s and inaccessible to AI.

AI in medical imaging has been very limited and ineffective thus far.

A) I think it's too early to make a call here. The first big paper for AI in medical imaging with modern methods came out in 2016, and we are just now getting large scale prospective studies where algorithms outperform specialists: https://www.thelancet.com/journals/landig/article/PIIS2589-7...

My institution pays for an expensive lesion detection / mammography CAD software package but it generates so many false positives I don’t use it in my practice. Overall having to review an extra n pseudolesions per scan results in decision fatigue and increases my miss rate, at least in my experience.

B) Totally agree that old CAD has not been effective, and this has created a huge trust gap for the new generation of technology. The new generation of technology using deep learning is also easier for anyone to create so it will take some time to separate the good from the bad.

There are several issues with AI when it comes to radiology that I can’t address with a single comment but given your question my biggest criticism would be that we don’t have a sensitivity problem in medicine. What we have is largely a specificity problem, but that’s inherent with imaging which is not a ground truth representation of a physical entity but rather how this tissue attenuates photons (for mammo).

C) This claim seems overly broad. If we can improve sensitivity it can certainly be a good thing. That being said, improving specificity in an assisted read scenario seems almost impossible in my experience, because if a doctor is convinced something is cancer, no output from an algorithm is going to get them to downgrade. We did an analysis on exactly what was going on when our algorithm would give a lower, non-recall score vs. a panel of radiologists, and we found that the most common situation was missing that a lesion was a scar. https://www.nature.com/articles/s41591-019-0447-x

It turns out that radiologists will in general not look at a lesion from all different possible reconstructions in 3D, because it's too time consuming. But sometimes there is only one reconstruction/view where the scarring is obvious. AI obviously does not have this issue, so in principle it is certainly capable of improving on specificity and this has been seen in many peer reviewed retrospective studies.

As I mentioned, the issue is how to make this happen in clinical practice where assisted read is the only option that will be tolerated for deployment for at least the next 5-10 years. In that time period, I think the only thing we can really hope for is a boost to sensitivity without impacting specificity. I believe eventually enough strong evidence (prospective observational multi site studies) will accumulate, and the algorithms will improve enough that for something like mammography, at least a portion of the reads can be done without a radiologist. This is probably the only way we will see improvements to specificity AND sensitivity in my opinion.

Sensitivity and specificity are interrelated.

Increased sensitivity (I.e. in CTPA) often leads to over diagnosis and bad outcomes.

We don’t routinely look at all three planes because most imaging and size criteria were developed and validated on axial sections (I.e. RECIST). Research done on volumetric measurements is in its infancy and of questionable added benefit.

Show me an example where sensitivity is a problem in modern day radiology and I might buy this argument.

Otherwise this is a solution in search of a problem

One thing engineers often don’t appreciate about medical interpretation is that you may THINK you’ve designed a better wheel but unless you can show better OUTCOMES (not increased detection) this is probably irrelevant.

Reply on news.ycombinator.com