I think in some ways making the model available online can be good, but in other ways could be harmful too. Very complicated topic.
恭喜coolwulf， 祝你继续成功。
And especially because in the future, most radiology work will be done by software. It's just a matter of whether it's 10 years or 100 years from now.
Surely concern for the well-being of the patient figures in there somewhere...
Or imagine this: A liver lesion is incidentally discovered on your abdominal CT performed for unrelated reasons. Its radiographic characteristics are equivocal. Additional imaging studies fail to completely exclude the possibility of a liver malignancy. You undergo a biopsy. But the biopsy is complicated by hemorrhage. Surgery is required. You develop a post-operative nosocomial infection. etc. etc.
To the extent that risks along this chain of unfortunate events is known, yes, warnings could put some of the quantified decision making in the patient's hands. Well, except for the rampant innumeracy in the general population...
This topic (breast screening) is a good example due to the sheer scale. If you increase the work-up rate by even a smallish amount, you are statistically pretty much guaranteed to kill people who did not have the disease. How this balances about gain (i.e. save other lives) is not obvious. Figuring out the "right" way to do this is real work.
Gøtzsche PC. Mammography screening is harmful and should be abandoned. J R Soc Med. 2015;108(9):341-345. doi:10.1177/0141076815602452 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4582264/
It's not the case the more screening is always better. There are plenty of screening regimes that you've never heard about because the trial data didn't support it.
"I have no medical background whatsoever but I do not buy undisputed medical science."
> I'd get a CT scan every year if I could convince someone to sign off on it
20mSv of unnecessary ionizing radiation exposure a year, what could possibly go wrong? That whole-body CT scan is equivalent to getting at least EIGHTY chest x-rays, and ten times what a uranium miner receives in a year, and well within the range where an increased risk of cancer is noticeable in epidemiological data.
Case and point why this tool should not be generally available.
You would choose annual body CT even if outcomes-based evidence showed no benefit?
Even among those at highest risk for lung carcinoma, studies of annual screening chest CT do not uniformly show improved disease-specific survival as compared to an unscreened but otherwise matched cohort.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3762603/
What I found mind blowing when I first read these studies (particularly the second link) is that the positive-predictive value of the CT screen was only 3.8% for lung nodules that were 4mm or larger in size. Basically just means we find all kinds of lung abnormalities all the time that "could be cancer" but aren't.
Keep in mind, that the screening criteria was people over age 65 who had a 30+ pack-year smoking history (1 pack-year = smoking 1 pack of cigarettes per day in one year), and had to have smoked within the last 15 years.
I can also tell you from doing CT lung biopsies that it's not a procedure to take lightly. Lot of important structures in the chest.
I mean, plus I have extreme health anxiety, and I've had doctors say "nah you're too young, we don't need to test for that" and later it turned out I did have kidney failure at 35.
It really isn't. Your chances of negative consequence for unnecessary follow up procedures would rapidly become significant.
Does a patient take a negative exam as "I'm good!" and forego actually seeing a doctor, or having the exam read by a radiologist?
A positive result is also a challenging situation. I've talked to patients before after diagnosing them, and as you could expect most people are shocked, scared or they want to know what they should do next.
In breast radiology specifically (at least in the US) there is a well defined reporting lexicon and classification system. From what I can tell, the system does not use the lexicon or BI-RADS classification. https://www.acr.org/-/media/ACR/Files/RADS/BI-RADS/BIRADS-Re...
Breast cancer can be very subtle, and can be hard to figure out, especially if a patient has had surgery or other benign abnormalities. https://radiopaedia.org/cases/development-of-dcis?case_id=de...
Diagnostic and screening images also undergo rigorous QA for the entire system. Does the model have some kind of QA built in?
There is a just a small disclaimer at the top that says "we will not store your data on the server. please do not worry about privacy issues". Not exactly a formal or legal binding agreement.
In fact, you can't even connect over https. http://mammo.neuralrad.com:5300/upload
These are just a few things that come to mind off the top of my head. Like I said before, I do think that what coolwulf works on and is trying to do is good, and in the long run can help doctors better characterize findings and help patients. But like anything in health care, there are a lot of edge cases and side effects that you have to think about. The stakes are also very high.
So we don’t have to worry as much about the “Does a patient take a negative exam as ‘I'm good!’”, because in the normal flow of medicine that gets this scan the doctor will check the scan for them either way. The website probably could have better explanations of this, but most likely negatives will already be double checked independently by doctors, and positives will most likely be handled correctly by bringing it to their current current attention to double check the scan or run more tests.
I wonder how the false positive/false negative rates for this tool compare to that of a trained human. At some point we may reach parity, in which case what's the harm of trusting the automated result?
I don't think that a human reader in the loop will be going away for a long time. Sure, they can be at parity, but does a combined read have better results? This is an active research question.
I think what will happen is that over time, human interpreters and AI systems will "co-evolve" in a sense, where people will pick up on where models are wrong and also learn how to use models to understand their own blind spots. These are also active research topics that are in their infancy.
EDIT: To be clear, I believe that this is a different group of authors from coolwulf. Pubmed link:
It's our job (as radiologists and engineers) to shape that in a way that benefits patients.
This is very true. Data availability (and moreso, label availability) is the biggest barrier to improvement here I suspect [ thanks for labeling!]. Access being another. Using a public site to bootstrap that could do very interesting things. On the other hand, public access to a poorly RA/QA'd algorithm could also cause more trouble than help, easily.
I am hopeful though. Few-shot learning and self supervision are very active questions right now and there are a lot of papers in the medical AI field that are getting published on these topics.
I'm personally interested in liver cancer, which does not have large, well-curated and shared database of cases.
Sharing data gets tricky, especially in the US. Labels I'm working with and creating (at least at this point) are for my own research, which won't be shared publicly any time soon.
Most of what I've seen isn't very promising. The energy in these research areas are because it would be so much cheaper than the "right" way, far more than because of the likelihood of success. And also perversely, because it's hard for the academic researchers to get enough data to do other studies :) NB: I'm not saying there is nothing useful coming out of the learning literature in last few years, just that it a) isn't a silver bullet and b) is often being misapplied in these areas anyway.
Label quality and availability isn't the only big problem though. Many data sets exhibit problematic sampling bias, as well as being order(s) of magnitude too small, because of the way they are gathered and how access is granted.
Not to mention imaging protocols are not standardized, and the imaging technology is also evolving so scans we do today may not be "correct" or standard in 5-10 years.
I suspect we have similar overall views of the problem, but I'm pretty strongly in camp that recent advances in ML/AI are mostly really driven by data & label availability, not algorithmic advances - this colors where I think the wins to be had in medical ML can happen most easily. Either way though the non-technical barriers seem clearly higher than the technical ones still.
I’m a first year radiology registrar (PGY3) in Australia looking to find others doing interesting work in this domain, if you think I could help with your efforts feel free to DM
Results are not impressive until they are :)
It's certainly not a solved problem, and it's easy to have a pessimistic view now but I'm generally bullish on where things will be 10 years from now.
True! I certainly wouldn't discourage anyone from trying.
On the other hand, I think it would be a huge mistake to trust that fancy learning approaches will solve everything so we shouldn't try and improve access and labling. Getting better there is still by far the most high probability of successful impact, imo.