Radiology-specific foundation model
harrison.ai
harrison.ai
But radiologists are very busy and this could help many people. Put a strong disclaimer in there. Open it up to subscriptions to everyone. Charge $40 per analysis or something. Integrate some kind of directory or referral service for human medical professionals.
Anyway, I hope some non-profit organizations will see the capabilities of this model and work together to create an open dataset. That might involve recruiting volunteers to sign up before they have injuries. Or maybe just recruiting different medical providers that get waivers and give discounts on the spot. Won't be easy. But will be worth it.
Like, that's the only REAL reason? Not the technological or ethical implications? The dangers in providing people with no real concept of how any of this works the means to evaluate themselves?
The infantilization of the public in the name of “safety” is offensive and ridiculous. In many countries, you can get the vast majority of medicines at the pharmacy without a prescription. Amazingly, people still pay doctors and don’t just take random medications without consulting medical professionals.
It’s only “necessary” to limit access to medical tools in countries that have perverted the incentive structure of healthcare to the point where, out of desperation, people will try nearly anything to deal with health issues that they desperately need care for but cannot afford.
In countries where healthcare costs are not punitive and are in alignment with the economy, people opt for sane solutions and quality advice because they want to get well and don’t want to harm themselves accidentally.
If developing nations with arguably inferior education systems can responsibly live with open access to medical treatment resources like diagnostic imaging and pharmaceuticals, maybe we should be asking ourselves what is it, exactly, that is perverting the incentives so badly that having ungated access to these lifesaving resources would be dangerous?
The TLDR is that most people when interacting with anything other than their GP family doctor, are probably interacting with someone "fresh out of school."
They've crammed it, yes. They need some extra time learn how to make use of that knowledge in day-to-day practice.
You wouldn't drop a fresh college CS grad by themselves in a group a developers and expect them to just figure it out. Just like medical school doesn't really teach you how to be a doctor, a CS degree doesn't really teach you how to code. They're both much more academic than the day-to-day of the job you're getting that degree for. They'd still get mentorship from colleagues, supervisors, and others. The only difference is medicine has the ACGME and all the government regulations to make it much more structured than what you need for most everything else.
Well, the conditional in this if statement doesn't hold.
Yes, pharmaceuticals are open access in much of the developing world, but it has not happened responsibly. For example, Carbapenem-resistant bacteria are 20 times as common in India as they are in the U.S [1]
I really don't like this characterization of medical resource stewardship as "infantilization" because it implies some sort of elitism amongst doctors, when it's exactly the opposite. It's a system of checks and balances that limits the power afforded to any one person, no matter how smart they think they are. In a US hospital setting, doctors do not have 100% control over antibiotics. An antibiotic stewardship pharmacist or infectious disease specialist will deny and/or cancel antibiotics left and right, even if the prescribing doctor is chief of their department or the CMO.
[1] https://www.fic.nih.gov/News/GlobalHealthMatters/may-june-20...
It comes from dealing with the public.
> In many countries, you can get the vast majority of medicines at the pharmacy without a prescription. Amazingly, people still pay doctors and don’t just take random medications without consulting medical professionals.
I see people on this site of allegedly smart people recommending taking random medications ALL THE TIME. Not only without consulting medical professionals, but _in spite of medical professional's advice_, because they think they _know better_.
Let's roll out the unbelievably dumb idea of selling self-diagnosis AI on radiology scans in the countries you’re referring to and ask them how it works out. If you want the freedom to shoot from the hip on your healthcare, you've got the freedom to move to Tijuana. We're not going to subject our medical professionals to deal with an onslaught of confidently wrong individuals who are armed with their $40 AI results from an overhyped startup. Those startups can make their case to the providers directly and have their tools vetted.
So again, if you want to ignore the safeguards that we've built for good reason - take your business to Tijuana.
I'm curious what you think the problem is, concretely, with a tool like this in the hands of the public which you clearly have such disdain for. Let's assume I buy this thing (the horror). I have to actually get access to my scans, which despite being legally required to provide most providers will be loathe to actually do. So I get my scans, I get this AI tool, I ask it some questions. It's definitely going to get some answers right, and it's very likely going to get some answers wrong. I'd be shocked if it's much less accurate than a resident, and if they're commercializing it there's a decent chance it's more accurate than the average experienced attending.
What is your doomsday scenario now that I have some correct data and some incorrect data? What am I going to do with that information that is so "unbelievably dumb" that I need the AMA to play daddy and prevent me from hurting myself? I can't get medication based on my newfound dangerous knowledge. I can't schedule a surgery or an IR procedure. I can't go into an ER and say "give me a cast here's a report showing I need one."
Unfortunately we have a lot of people that despite an abundance of ignorance are arrogant enough to think that they know better than most, but still are incapable of applying basic logic and critical thinking in their reasoning. (which if you think about it is sensible, since wisdom is largely a matter of comprehension of the depth of ones ignorance)
These people are a huge PITA to work with and every tool they wield inflicts pain on those trying to help them.
It was a serious 30 something woman who collected something like 50 pesos (around $3), listened to me for about 30 seconds, and told me to make sure I slept and ate well (I think she specifically said chicken soup). I asked about antibiotics or medicine and she indicated it wasn't necessary.
So I rested quite seriously and ate as well as I could and got better about a week later.
During the time that I was in Playas de Tijuana I would normally go to nicer pharmacies though, and they didn't ask for a prescription for my asthma or other medicine which was something like 800% less expensive over there. They did always wear nice lab coats and take their job very seriously if I asked for advice. Although I rarely did that.
I do remember one time asking about my back acne problems at a place in the mall and the lady immediately gave me an antibiotic for maybe $15 which didn't cure it but made it about 75% better for a few months.
Another time at the grocery store I asked about acne medicine and the lady was about to sell me something like Tretinoin cream for a price probably 1/4 of US price. She didn't have anything like oral Accutane of course. It was just a Calimax Plus.
There are of course quite serious and more expensive actual doctors in Tijuana but I never ended up visiting any of them. I was on a budget and luckily did not have any really critical medical needs. But if I had, I am sure it would have cost dramatically less than across the border.
EDIT: not to say the concession-stand office lady wasn't an actual doctor. I don't know, she may have had training, and certainly had a lot of experience.
They go to the doctor because the healthcare system here works, for the most part, and they value and respect the expert counsel in matters of their health.
True! And, aside from people with chronic conditions like diabetics, who are forced to know how their glucose levels work, nobody uses those. So it certainly does change the cost, but I don't think it would be any more useful in the US.
What was the potential harm that was greater than the reward?
It's like, not being obese enough for your insurance company to pay for medical intervention doesn't mean that your weight is optimal enough to enjoy a long retirement.
It’s probably very low in the context of CGM and diabetes as the potentially harmful treatments require prescriptions.
Device prescription requirements are usually due to product labelling and the manufacturers application. There are OTC fingerstick glucometers and CGMs approved.
To correct this, though. You can buy all those in the US as well. Holter and FirstBeat are selling clinically validated and FDA approved mutli-lead ECG, Derxcom is selling an over the counter CGM, as is Abbott with the Libre 2, and a Chinese company has recently joined there, too.
Low calorie meal replacements are all over the store, too.
If you're a member of this orthorexia/orthovivia crowd, you have the same access to tools as you do in the EU, often more so.
https://www.fda.gov/news-events/press-announcements/fda-clea...
On the surface those all sound like additional reasons not to make it available. But they are also great rationalizations for those who want to maintain a monopoly on analysis.
Personally I found all the comparisons to other AI performance bothersome. None of those were specifically trained on diagnostics AFAICT. Comparison against human experts would seem to be the appropriate way to test it. And not people just out of training taking their first test, I assume experts do better over time though I might be wrong on that.
Comparison against human experts is the gold standard but information on human performance in the FRCR 2B Rapids examination is hard to come by - we've provided a reference (1) which shows comparable (at least numerically) performance of human radiologists.
To your point around people just out of training (keeping in mind that training for the FRCR takes 5 years, while doing practicing medicine in a real clinical setting) taking their first test - the reference shows that after passing the FRCR 2B Rapids the first time, their performance actually declines (at least in the first year), so I'm not sure if experts would do better over time.
1. https://www.bmj.com/content/bmj/379/bmj-2022-072826.full.pdf
- X-Ray: $20
- Radiologist Consultation: $200
- Harrison.AI interpretation: $2000 - X-Ray: $15
- Radiologist Consultation: $125
- Harrison.AI interpretation: $20
Not to mention that in particularly sick patients problems tend to compound one another and exams are often requested to deal with a particular side of the problem, ignoring, perhaps, the major (but already known and diagnosed) problem etc.
Often times factors specific to a hospital play crucial role: eg. in hospitals for rich (but older) patients it may be common to take chest X-rays in a sited position (s.a. not to discomfort the valuable patients...) whereas in poorer hospitals siting position would indicate some kind of a problem (i.e. the patient couldn't stand for whatever reason).
That's not to say that automatic image reading is worthless: radiologists are, perhaps, one of the most overbooked specialists in any hospital, and are getting even more overbooked because other specialists tend to be afraid to diagnose w/o imaging / are over-reliant on imaging. From talking to someone who worked as a clinical radiologist: most images are never red. So, if an automated system could identify images requiring human attention, that'd be already a huge leap.
Yes, in principle, if people taking the images had infinite time and could foresee what kind of accompanying data will be useful at the analysis time, and then had a convenient and universal format to store that data, and models could select the relevant subsets of features for the problem being investigated... I think you should see where this is going: this isn't going to happen in our lifetime, most likely never.
Of course, there are lots of remaining challenges around integration and actually getting access to these data sources e.g. the EMR systems, when trying to use this in practice.
Hospital systems are atrocious for providing useful information anyways. They are often full of unnecessary / unimportant fields that the requesting side either doesn't know how to fill, or will fill with general nonsense just to get the request through the system.
It gets worse when it's DICOMs: the format itself is a mess. You never know where to look for the useful information. The information is often created accidentally, by some automated process that is completely broken, but doesn't create any visible artifacts for whoever handles the DICOM. Eg. the time information in the machine taking the image might be completely wrong, but it doesn't appear anywhere on the image, but then, say, the research needs to tell the patient's age... and is off by few decades.
Any attempt I've seen so far to run a study in a hospital would result in about 50% of collected information being discarded as completely worthless due to how it was acquired.
Radiologists have general knowledge about the system in which they operate. They can identify cases when information is bogus, while plausible. But this is often so much tied to the context of their work, there's no hope for there to be a practical automated solution for this any time soon. (And I'm talking about hospitals in well-to-do EU countries).
NB. It might sound like I'm trying to undermine your work, but what I'm actually trying to say is that the environment in which you want to automate things isn't ready to be automated. It's very similar to the self-driving cars: if we built road infrastructure differently, the task of automating driving could've been a lot easier, but because it's so random and so dependent on local context, it's just too hard to make useful automation.
PS genuinely appreciate the engagement and don’t see it as undermining.
As a senior developer I routinely use LLMs to write boilerplate code, but that doesn't mean that the layman can get something working by using an LLM. And it's exactly the same for other professions.
It's more likely that regardless of disclaimers people will still use it, and at some point someone will decide that that outcome is still the provider's fault, because you can't expect people to not use a service when they're impoverished and scared, can you?
Unfortunately, it's the other way around. The tech sector understands very little about clinical medicine, and therefore spends its time fighting windmills and shouting in the dark at docs.
I'm curious whether this AI model would have been able to detect my issue more competently than the shitty doctor.
Surely your results went to a requesting physician who should have been following up with you? Radiology doctors don’t usually organise follow up care.
Or was the inaccurate result from the requesting physician?
I guess the reasoning is that they want to provide „good service“ by giving the patient something to work with directly after the exam and the workload is so high that they couldn’t look at the images so fast. And they accept the risk that some people are getting angry because their exam wasn’t normal in the end.
But on the scale a typical radiology practice operates today, the few patients who don’t have a normal exam don’t matter (the number of normal exams in an outpatient setting is quite high).
I find it highly unethical, but some radiologists are a little bit more ethically relaxed I guess.
What I want to say is that it might be more of a structural/organisational problem than incompetence by the radiologist in your case.
(Disclaimer: I’m a radiologist myself)
A quick look at the paper in the BMJ shows that the model did not sit the FRCR 2b examination as claimed, but was given a cut down mock up of the rapid reporting part of the examination invented by one of the authors.
https://www.bmj.com/content/bmj/379/bmj-2022-072826.full.pdf
Were the same tests also used here?
However, the actual FRCR 2B Rapids exam question bank is not publicly available and the FRCR is unlikely to agree to release them as this would compromise the integrity of their examination in the future- so the test used are mock examinations, none of which have been provided to the model during training.
"The Fellowship of the Royal College of Radiologists (FRCR) 2B Rapids exam is considered one of the leading and toughest certifications for radiologists. Only 40-59% of human radiologists pass on their first attempt. Radiologists who re-attempt the exam within a year of passing score an average of 50.88 out of 60 (84.8%).
Harrison.rad.1 scored 51.4 out of 60 (85.67%). Other competing models, including OpenAI’s GPT-4o, Microsoft’s LLaVA-Med, Anthropic’s Claude 3.5 Sonnet and Google’s Gemini 1.5 Pro, mostly scored below 30*, which is statistically no better than random guessing."
But if someone is able to connect a network to the raw data outputs from CT or MR machines, one may start seeing these AI's radically outperform humans at a fraction of the cost.
For CT machines, this could also be used to concentrate radiation doses into parts of the body where the uncertainty of the current state is greatest, even in real time.
For instance, if using a CT machine to examine a fracture in a leg bone, one could start out with a very low dosage scan, simply to find the exact location of the bone. Then slightly higher concentrated scan of the bone in the general area, and then an even higher dosage in an area where the fracture is detected, to get a high resolution picture of the damage, and splinters etc.
This could reduce the total dosage the patient is exposed to, or be used to get a higher resolution image of the damaged area than one would otherwise want to collect, or possibly to perform more scans during treatment than is currently considered worth the radiation exposure.
Such machines could also be made multi modal, meaning the same machine could carry both CT, MR, ultrasound sensors (dopler + regular). Possibly even secondary sensors, such as thermal sensors, pressure sensors or even invasive types of sensors.
By fusing all such inputs (+ the medical records, blood sample data etc) for the patient, such a machine may be able to build a more complete picture of a patient's conditions than even the best hospitals can provide today, and a at a fraction of the cost.
Especially for diffuse issues, like back pains where information about bone damage, bloodflow (from the Doppler ultrasound), soft tissue tension/condition etc could be collected simultaneously and matched with the reported symptoms in real time to find location where nerve damage or irritation could occur.
To verify findings (or to exclude such, if more than one possible explanation exists), such an AI could then suggest experiments that would confirm or exclude possibilities, including stimulating certain areas electrically, apply physical pressure or even by inserting some tiny probe to inspect the location directly.
Unfortunately (or fortunately to the medical companies), while this cold lower the cost per treatment, the market for such diagnostics could grow even faster, meaning medical costs (insurance/taxes) might still go up with this.
I still see somewhat of a product gap in this whole area when selling into clinics but that can likely be solved with time.
“AI is a bubble”
We’re still scratching the surface of what’s possible. I’m hugely optimistic about the future, in a way I never was in other hype/tech cycles.
- one of the speakers at a recent health+AI event
I'm wondering what others in healthcare think of this. I've been skeptical about the death of software engineering as a profession (just as spreadsheets increased the number of accountants), but neither of those jobs requires going to medical school for several years.
Radiology remains one of the most competitive and in-demand specialties. In this year's match, only 4 out of ~1200 available radiology residency positions went unfilled. Last year was 0. Only a handful of other specialties have similar rates.
As comparison, 251 out of ~900 pediatric residency slots went unfilled this year. And 636 out of ~5000 family medicine residency slots went unfilled. (These are much higher than previous years.)
However, I do somewhat agree with the speaker's sentiment if for a different reason. Radiologist supply in the US is roughly stable (thanks to the US's strange stranglehold on residency slots), but demand is increasing: the number of scans ordered on a per patient continues to rise, as does the complexity of those scans. I've heard of hospital systems with backlogs that result in patients waiting months for, say, their cancer staging scan. One can hope we find some way to make things more efficient. Maybe AI can help.
You make it sound like the reporting radiologist is given a referral with helpful, legible information on it. That this ever happened doubtful.
https://harrison.ai/news/reimagining-medical-ai-with-the-mos...
I'd interpret it as a foundation model in the radiology domain
NB. In all claims I've seen so far about outperforming radiologist, the common denominator was that people creating these models have mostly never even seen a real radiologist and had no idea how to read the images. Subsequently, the models "worked" due to some kind of luck, where they accidentally (or deliberately) were fed data that made them look good.
This seems generally aligned with AI realities today: it won't necessarily replace whole job functions but it can increase productivity when applied thoughtfully.
How is chatgpt the competion? It’s mostly a text model?
I'd be 2x as productive if I could just speak and it auto filled my template in the correct spots.
And while you’re at it, the current ‘integrations’ between RIS and PACS are so jarring it sets my teeth on edge.
https://www.adsc.com/blog/what-are-the-differences-between-p...
Schedules radiology studies, exchanges information with the modalities (the radiology devices) so the studies have proper metadata, exchanges information with the PACS (the radiology image storage), it might be used by radiologists and/or transcriptionists to add the reports for the studies...
It might overlap a bit with the HIS (hospital information system) that's the more general hospital management software.
The manual data introduction, the shortcuts and workarounds, the differences on roles and even meanings! The words RIS, PACS or HIS doesn't mean the same to every person and have different functions on different places. Just ask somebody to compare a PACS to a VNA and run away! :D
In a way I miss the field, as it felt more productive than whatever stupid consulting firm that can reach me through LinkedIn.
I recently joined [Sonio](https://sonio.ai/platform/), where we work on AI-powered prenatal ultrasound reporting and image management. Arguably, prenatal ultrasounds are some of the more challenging to get right, but we've already deployed our solution in clinics across the US and Europe.
Exciting times indeed!
Prenatal ultrasounds are one of the most rote and straight forward exams to get right.
Acquiring the images is the hard part in obstetrical ultrasound, reporting is very mechanical for the most part and lends itself well to AI.
From their benchmarks it's looking like a great model that beat competition, but I will see the third party tests after they get released to determine the real performance.
"We have proprietary access to extensive medical imaging data that is representative and diverse, enabling superior model training and accuracy. "
Oh, I'd love to see the loicenses on that, :^).
> Filtered for plain radiographs, Harrison.rad.1 achieves 82% accuracy on closed questions, outperforming other generalist and specialist LLM models available to date (Table 1).
The code and methodology used to reach this conclusion will be made available at https://harrison-ai.github.io/radbench/.
I'd imagine access to the model itself will remain pretty exclusive, but would love to see them adopt a more open approach.
Deeplesion is another one out of nih.
Segmed is a yc company that sells access to radiology datasets