Breast cancer detection in mammography using deep learning approach
arxiv.org
arxiv.org
1) Mammograms are not interpreted in a vacuum. In fact mammograms are usually the first in a long line of tests before a breast cancer or other diagnosis is ultimately made. In fact, it's probably more accurate to refer to mammography as a screening exam for which patients need a biopsy rather than a diagnostic test for cancer (there are rare exceptions, but overall this point holds).
2) Speaking frankly as a radiologist myself, tests like mammograms aren't even that good in terms of overall diagnosis. Thats why ultrasound, tomosynthesis and MRI are often used as supporting evidence and/or alternative exams.
3) There is controversy over the overall utility of mammograms, particularly in the screening context. Radiologists more than anyone would like the sensitivity and specificity of these studies to be higher.
It strikes me that the people that push these "radiology is ripe for disruption" or "AI outperforms radiologists" hyperbolic arguments are clearly people that have never seen the inside of a clinic. I'm sure they love this rhetoric though when pitching to VCs or sitting around the conference table coming up with 'breakthrough ideas' to turn into power-points for the other administrators.
I wasn't just commenting on the abstract presented. I was commenting on the comments I see here, as well as comments I see related to similar papers all the time.
I also interact with AI/ML researchers all the time. Most of them are typically some combination of: 1. Poorly informed about the appropriate context and utility of medical imaging. 2. Trying as hard as they can to push AI/ML as the most important technology in medicine today. 3. Pursuing a very task-specific project which they claim is massively generalizable in some (incorrect) way.
In fact, the area of medicine most in need of 'disruption' imho is healthcare enterprise software. Doctors are literally killing themselves because the interfaces they have to deal with on a daily basis are so appallingly poor.
Of course, the solution is less technological than political as it wouldn't take much to come up with better software than the legacy alternatives but you'd have a very hard time getting past the entrenched relationship interests of crony bureaucrats that run hospital administrations.
I wonder how long people think about these sorts of claims before posting them.
Do they really think an AI is more likely to appropriately interpret an MRI scan (and all the anatomic, physics and pathophysiological data therein contained) in the context of a specific clinical work-up more easily than triage patients the way a family practitioner or ER doctor does?
The paper specifically talks about mammography, it does not claim to replace a complete diagnosis.
> 2) Speaking frankly as a radiologist myself, tests like mammograms aren't even that good in terms of overall diagnosis. Thats why ultrasound, tomosynthesis and MRI are often used as supporting evidence and/or alternative exams.
From the abstract: "2) successfully extends to digital breast tomosynthesis"
> 3) There is controversy over the overall utility of mammograms, particularly in the screening context.
> It strikes me that the people that push these "radiology is ripe for disruption" [...]
The paper, which I just skimmed over, does not read hyperbolic, for that we'll have to wait for popsci journalists.
OTOH, if one leaves the 1st world context, any type of successful diagnosis automation in medicine is a blessing for areas where you simply don't have enough trained medical staff.
I also interact with AI/ML researchers all the time. Most of them are typically some combination of: 1. Poorly informed about the appropriate context and utility of medical imaging. 2. Trying as hard as they can to push AI/ML as the most important technology in medicine today. 3. Pursuing a very task-specific project which they claim is massively generalizable in some (incorrect) way.
My engineering mind just can't come up to terms with this. Why wouldn't you collect all information you possibly can? You can always ignore irrelevant data you have, but you cannot consider data that you don't have!
The closest I've been to rationalizing this: diagnosing is a stochastic process so complex (and with the search space so large) that the random noise in extra data is likely to point you towards wrong directions. Plus you can always collect more data afterwards if your initial diagnosis turns out to be wrong. This is of course very simplified, but it makes sense.
However, I just can't turn off my inner voice from screaming "more data is always better". I guess that's why I'm not an MD :)
You run a screening company. You take people at high risk of lung cancer -- people who smoke a lot and have smoked a lot for many years -- and you provide low dose CT scans of their lungs.
Bob comes in. You scan his lungs and you find spots.
What do you do now?
You're probably going to start providing treatment to Bob. Will this help Bob live longer? Will it improve his quality of life? It might not.
https://blogs.bmj.com/bmjebmspotlight/2019/02/15/understandi...
Hopefully get other tests done, to confirm diagnosis.
I'm with GP here. I can't understand this attitude either. Having more information should never make you more wrong. This holds for uncertain information, because uncertainty can be quantified and tracked (if you're not doing this, then you're doing voodoo, not science).
I can see two reasons why you wouldn't want to gather more information in medical context. One, many tests carry risk to patient's health and well-being, so there's no point of doing them if that risk outweighs the expected value of evidence gathered. Two, I suspect that gathering information also gathers legal obligations and risks to doctors.
Those other tests involve things like "needle biopsy" -- they shove a needle through your chest into your lung into the suspect tissue to get a sample. This carries risk. We can justify that risk if it saves life. But this is the problem with screening -- often it doesn't save life (of course, it depends on the type of screening).
https://www.radiologyinfo.org/en/info.cfm?pg=nlungbiop
> Having more information should never make you more wrong
But you can see how having lots of low-quality information could make someone more wrong -- these are not clear signals, because if they were it wouldn't be a problem. These are almost noise. We're taking data from a large population ("4 in 100 people with this result have this disease") and trying to apply it to the individual, and when we try to get more information we subject this person to more radiation in scans or invasive procedures or both. We increasing the risk, but not necessarily saving life.
> there's no point of doing them if that risk outweighs the expected value of evidence gathered
Yes, this is exactly the balance that doctors are making. They're looking at all cause mortality and seeing if life is saved.
Everything we know about human psychology says you can't.
From a decision theoretical viewpoint, you would certainly want all the information you could get. For humans running a business in today's medicolegal environment, it's a very different set of issues:
1) Collecting information costs time and money.
2) Making good decisions requires the most precious resource of all, which is doctor brain-time. There isn't enough of it to spend on information with little probability of benefit.
3) If you get sued for malpractice, the unneeded data you collected probably would not have helped the patient, but it could help the attorneys arguing that you missed something. Juries struggle to understand the cost of false positives.
Even though there are valid issues here, doctors don't always make the right tradeoff in this regard. Oftentimes, I think it is more an issue of lack of training or experience that leads a doctor to consider a test to be unneeded. In the case of mammography, if doctors spend too much time doing screening and not enough time doing diagnosis, their screening performance degrades, which I think is due to a lack of feedback on their decision making[1].
In the short term, more information might cause harm because doctors are risk averse & scared of lawsuits and err on overbiopsy/overtreat, and many of our treatments aren't as good as we think they are, and all of this makes patient anxious.
In the long term, turning the information firehose on full blast means we can work out which incidental findings are best ignored or pursued and overall more data will help us.
The problem is that it is unethical to do #2 in the short term even if it is the long term ethical thing to do.
I completely get your attitude, I think I agree with you overall and if I was not this lazy I could comb through my bookmarks and find the studies supporting what you said.
But I was just responding to your comment in the context of the paper linked. Which, at least when skimming over it, does not read like what you (IMHO, rightfully) criticize in the broader debate.
And yes, read the first paragraph as a tongue-in-cheek response, we both know that overgeneralizations don't help any debate ;)
We don't use deep learning - we use Biophysical models. We hate using the term "AI". This is a very challenging discipline to explain to VCs.
Also, speaking to point 2 here - the "value" of building tools for ultrasound is often dismissed by VCs because "ultrasound isn't used for screening or diagnosis". This is an insane perspective from our position when we are practically based within hospitals, collaborating strongly with radiologists and medical oncologists who work with ultrasound on a daily basis.
We are very embedded within the hospital and look to understand the clinicians workflow and decision making processes first, as well as understanding what's possible given the hurdles involved in data access (which can still be tricky even when you are through IRBs and ethics).
We have found that telling VCs the reality about working with hospitals and doctors can often limit their excitement about your company prospects. Our success to date has largely been as a result of doctors and hospitals who believe in us, see the value in what we are doing. They have put time and effort into collaborating because they are impressed with what we have been able to do results wise by bootstrapping as a small team, rather than as a VC funded shiny startup.
In a weird way i would say that at it's best times medtech can be one of the "purest" industries to work in. By this i mean ultimately your technology works or it doesnt (at least from the medical communities perspective - again VCs are a different story). There are obviously exceptions to this (Theranos anyone) and there are issues around the 510K process but on the whole there is a big price to pay for making unsubstantiated claims (say compared to aspirational lifestyle marketing).
I don't think taking the human all the way out of the loop is a great idea given the state of models we've seen to date (in medicine, anyway).
I think a better direction would be looking at how to make these systems more complementary with human operators, surfacing interesting features or distant connections humans tend to miss and guarding against the big errors anyone might make staring at grayscale images on the night shift.
Yes, I agree it should be another, yet increasingly important tool for a while. Perhaps not until a near-AGI is invented that machines can completely automate away doctor jobs.
In some cases they also spend a lot of time annotating and measuring small areas of the images and having a model generate suggestions for them would save a ton of time.
If the NNS for a screening test is 5,000, those who advocate screening must make the ethical argument that the large benefits to 1 individual justify the sum of the harms to which 4,999 people are exposed. Whether this holds up to moral scrutiny depends on the nature of the harms.
The problem that I think the medical community misses is that the unwashed masses are generally completely unable to access procedures that aren't recommended by a doctor. So in the above assertion, that someone 'advocating for screening' must take on the ethical burden of harms coming from that screening, in my estimation, is complete and utter bullshit. What I would prefer to see is that screening comes with patient education so they understand the potential risks and inaccuracies and then let 'er rip. If something goes south, it's on them.
Gerd Gigerenzer has done a lot of work on this, so his books are useful. Reckoning With Risk or Risk Savvy are good.
> What I would prefer to see is that screening comes with patient education so they understand the potential risks and inaccuracies and then let 'er rip. If something goes south, it's on them.
Informed choice should already be built into all healthcare systems because it's a feature of international human rights laws and patients are usually allowed to decline to have a test done. And there's usually a doctor somewhere who'll perform an unnecessary test if you're prepared to pay.
Communicating risk is difficult.
We know that merely giving people information and letting them take full responsibility won't work. We know it won't work because we already study whether people understand the risks of testing and treatment, and we find a disturbingly large number of people don't, and that includes the HCPs recommending the testing and treatments.
Most people struggle with this question: "A machine has been invented to scan a population for a disease. The machine is good but not perfect. If you have the disease there is a 90% chance it will return positive. If you do not have the disease there is a 1% chance it will return positive. About 1% of the population have the disease. Mr Smith is tested, and the test comes back positive. What's the chance Mr Smith actually has the disease?"
But the problem of lack of numeracy is more severe: only 20% - 25% of people understand that 0.1% is 1 in 1,000. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3310025/
People do not understand the difference between "absolute" vs "relative" risk increases, so we see newspapers reporting "100% increase in risk from eating X" when the numbers are an increase from 1:100,000 to 2:100,000 deaths.
https://bestpractice.bmj.com/info/toolkit/practise-ebm/under...
https://www.eufic.org/en/understanding-science/article/absol...
100% agreed.
A significant dimension of the regulatory dynamics is accountability: Who will sign off on and ultimately be responsible for findings from radiologic studies?
Ceding this responsibility to corporations is a terrible idea. After all, one of the things about a corporate entity is that there isn't really anyone responsible. The GFC and Boeing are recent perfect examples of this. Automating medicine will result in making healthcare more like trying to get tech support. Yes doctors are imperfect, they make mistakes, some definitely shouldn't be working, they are territorial and monopolistic etc etc, but when the system is working you walk into a room with another person who wants to listen to you and help you, and we shouldn't ever try to take that away.
And in the case where the machine learning algorithms don't find anything suspicious, the GP again won't have the training or experience to confirm those results. Now if the person was otherwise healthy and this was just a screening that might be enough, but if the GP was suspicious enough to order the test in the first place, it won't be.
What will probably happens is that this kind of technology increases productivity for radiologists, and maybe increases the number of screenings done on healthy people. But it's not going to reduce the demand for radiologists.
Basically the problem is that to be able to interpret the output of a neural network you need to be an expert. What we need is AI that can present a fully formed argument that is easy for a non expert to follow and validate, but we are nowhere near that in most cases.
I also think we need to watch out for the human-attention issues illustrated in almost self driving cars. If a radiologist gets used to the computer being right 9/10 times, they could miss the 10th which would usually have been caught.
Overall we need more CV/ML/AI (choose your acronym) in this space, but it definitely requires some care.
Doctors make mistakes all the time and people in the medical profession work odd hours so there's already a ton of room for errors. Having a machine check their work or provide a second opinion will help them a lot.
Having a system that can do the primary screening and prioritize patients before a radiologist is available will save a ton of lives.
> Doctors make mistakes all the time and people in the medical profession work odd hours so there's already a ton of room for errors.
That's my point - this is the training data. Unless we're careful, we're just going to approximate what we're already doing.
I'm working on a similar system but for dietaty feedback based on images and it's amazing to see the model outperform all of the dietitians because it's able to see how all of the coaches respond to similar items.
They've been successfully pushing back against scope of practice changes for RAs, PAs, ANPs, PTs and other midlevels, let alone allowing foreign doctors in without an expensive residency medallion. And on the data side, hospitals are doing all they can to make sure that is as locked down in their silos as possible, because all these AI papers have been posing a fundamental threat to their bottom line (Medicare and employer reimbursement for physician and related services).
Basically every single decision-making regulatory body on state and Federal levels is full of MDs, with inherent conflict of interest.
Now new FDA chief is an old-school MD again, and his first order of business was to call a conference studying the potential dangers AI can pose to patient safety.
As such changes continue to be signed into law, I assume you mean, overall, unsuccessfully doing so.
This stuff will be used to do initial screening to prioritize cases and provide an initial analysis for the radiologist to confirm. Once they're deployed it won't be too long before they're as good as the top practitioners.
It will be a long time before they completely replace radiologists in America but we'll probably see them on autopilot in third world countries where there's a shortage of doctors and data privacy laws are not as stringent. I've met a Chinese guy doing medicine in the states who claimed to have access to all medical data for a bunch of hospitals back in China.
Radiologists are likely to see a lower demand as a result of these technologies and will either A) Spend more time on complicated cases or B) Be let go. Nobody is saying ALL radiologists are going to be out a job. Look at dosimetry as a recent example of how software improved, and the time to contour per patient decreased, causing many health systems to shrink their dosimetrist staff or offload the responsibilities to the physician office.
This isn't the first time technology has been applied to healthcare, change will come slowly and eventually people will have to find new fields to work within healthcare.
I had some coworkers who went to RSNA this year. The AI companies are still desperate for data. There was direct discussion of the disappointment in AI in more than one talk.
It'll happen, but like anything else it'll take a lot longer than people were predicting.
You can always choose not to act on what you learn. You can't act on what you don't know.
You can also act on what you think you know but in fact don't. False negatives kill people but so can false positives.
There is presently some set of screening tests with varying levels of sensitivity and specificity, and they aren't all appropriate for mass screening. However, if millions of people started regularly getting non-ionizing imaging done through MRI or ultrasound or infrared or whatever, we would learn a shitload about predicting maladies and likely save quite a few more people than we kill in the process.
I couldn't say, but I wage the medical science community / industry is cognizant of the phenomena and, not being an expert in the field myself, I trust them to handle the matter reasonably. In the specific case of breast cancer, I've heard that the cost of false positives is high enough to be a major consideration in recommended breast cancer screening schedules.
What we want is prevention, not early detection. Bear in mind that prevention is a proven outcome, because some people never get Alzheimers, and some people never get cancer. Detecting cancer or Alzheimers early and then intervening to cure the condition is actually not something that is definitely possible.
Even taking your example, getting to the point of removing a pre-invasive adenocarcinoma is actually very complicated. You need a motivated patient, you need a healthcare team of specialist doctors or nurse practitioners, pathologists, radiologists, anaesthesiologists, nurses, an endoscopy suite, a recovery area, you need a system to follow up and track these cases etc. The full embedded cost of this undertaking is huge, I don't think it is scalable, and at the same time, will never capture everyone. Randomised studies usually just barely show that these screening approaches are better than not screening, once false positives are accounted for. Furthermore, you have chosen an example where there is a useful pre-screening test (fecal blood tests), a relatively non-invasive diagnostic test (colonoscopy) and a relatively painless intervention (removing a polyp with the scope). This is not broadly applicable. A pancreatic lesion for example is very hard to diagnose for sure, and the intervention is a massive and life changing operation.
Cancer prevention is best, I agree, but it's not always clear that it can be perfectly prevented. So then you have to move to detection and therapeutic intervention to either eliminate the cancer or prevent it from materially impacting the health of the individual. And with cancer specifically, early detection is one of the best predictors of a good outcome. This is not always possible of course, some cancers are very difficult to detect, but some aren't. It just seems like we are making perfect the enemy of good.
The strongest argument is an economic one. If we can develop preventative treatments that we can just give to everyone, it is a much more justifiable expenditure. The risks and benefits are much clearer. This is in contrast to what we do now, which is apply extremely expensive and complex care to a group of people that are diagnosed with cancer. There are only a small number of people in the world that can even get this care. Consider that a preventative therapy is the only type of therapy some people in the world will ever get for their cancer. Papua New Guinea for example has zero oncology specialists in a country of 8 million people.
In case it didn't come through I'm 100% agreement with you on this. Learning about TIL and CAR-T therapies now, but in general feel that cancer vaccines would rival any pharmaceutical on the market in terms of profitability and net decrease in global misery.
Just trying to optimize the time between now and that (IMHO) inevitable day.
1. Even the best tests have a chance that the scan will produce a false positive. If we indiscriminately test everyone for every disease under the sun, most of the findings will be false positives.
2. The benefit must outweigh the cost of the test. Both in terms of expense, and intrinsic factors such as radiation exposure.
3. Certain things that show up in a scan would never cause a problem if they were just left alone. In these cases detecting it early has little benefit, and may lead to unnecessary interventions.
4. If a cure is not available, testing for the disease may not always be appropriate, from an ethics point of view.
Especially 3D US - so far it's mostly been used for "cosmetic" purposes of getting a 3D picture of your baby. There hasn't been much clinical evidence to its usefulness from what I can tell.
Anyone have a good reference point for the reader selection of 5 specialists with 5.6yrs avg experience? That population seems small. Another opportunity for licensing bodies or national institutions to grow a publicly available dataset -- including annotations from a wider selection of imaging specialists.
There's another completely (to me) unintuitive angle as well. Andrew Lo and some folks from MIT Sloan have published a paper about using clinical trial data to predict which medicines will be approved by the FDA in order to help reduce investment risk and unlock dollars. He does a pretty good talk about it here - https://www.youtube.com/watch?v=AzELyaVf0v8
He's on a recent episode of Linear Digressions discussing this as well. http://lineardigressions.com/episodes/2019/12/8/using-data-s...
Most of that are statistical methods in statistic and not ML. We're talking about survival analysis, longitudinal analysis, clinical trial, nonparametric statistic, etc...
From my experiences ML is too dependent on large dataset. Medical data are often high dimensional and small. My thesis papers have leverage two statistician works to make decision trees and ensemble leverage more statistic to handle medical data (high dimensional data). As noted by Dr. Harrell, statistician, ML is much more suited with less noisy data, medical image. Also inference is most more important in the medical field that just prediction.
https://old.reddit.com/r/MachineLearning/comments/ehpllt/dee...
And also linked blog[edit]:
https://lukeoakdenrayner.wordpress.com/2017/12/06/do-machine...
TL;DR; There are sooo many subtlties to stuff like this that this things really shouldn't be taken at face value. This is far from replacing doctors in anything.
The sensitivity is also still not as high as you would like ideally... this is a limitation of mammography.
2) I see nothing in the Luke Oakden-Rayner blog that calls this study into question. This study actually avoids the pitfalls that he mentions.
3) This paper said nothing about replacing doctors.
I'm not sure that day will ever come.