Production AI systems are hard

Almost 90% of the diagnostic studies I read could be pre-drafted by AI. That's where the money is and where AI-in-radiology companies should focus. The money is not in detecting hemorrhage or pulmonary embolism. It's a classic fallacy to think that life-saving means money-saving. Rather, the money in radiology is reads per day.

Here's a user story: A private practice radiologist reads 20 abdomen and pelvis CT scans with contrast per day. In each of these studies, he must write a short description of each organ. For example, "Gall bladder: Unremarkable" or "Gallbladder: Cholecystitis without evidence of cholecystitis" or "Gallbladder: Dependent sludge." There are around 15 such organs (liver, gallbladder, pancreas, spleen, adrenal glands, kidneys, etc.). The AI should auto-populate the radiologist report with an appropriate description of each organ system.

The job of the radiologist is to confirm what the AI says in each section, and to go into further detail as needed. It's essentially just customizing the existing template to each patient. This type of pre-drafting is exactly what radiology residents do and what companies like vRad do.

This would actually be a long term reason to go for AI / database diagnosis.

I had a personal case where a close family member, a young child, almost died. The doctors didn't understand condition and last minute it suddenly calmed down.

Im sure a few doctor's in the world seen it before, but they weren't working in my hospital that week. If we can start sharing in-depth diagnosis worldwide of obscure cases using AI to make it easy to query that would probably be of great benefit.

Yes. There is still so much low hanging fruit for software everywhere.

Author starts off by saying that you can't generalize conclusions based on observations in a limited domain:

> Geoff made a classic error that technologists often make, which is to observe a particular behavior (identifying some subset of radiology scans correctly) against some task (identifying hemorrhage on CT head scans correctly), and then to extrapolate based on that task alone.

Author then goes on to suggest that there are a number of issues with radiology which AI will not solve.

Then, based on this lack of ML's complete coverage of the aspects of radiology that he finds interesting, he then extrapolates:

> Should you be worried GPT5 is going to interact with social systems and destroy our society single-handedly? No absolutely not.

I don't see how the conclusion remotely follows from the argument. And more to the point, the refuting argument is embodied in the article itself.

Also, neurologists are quite specialized (e.g., stroke vs. MS vs. acute encephalopathy vs. optic neuritis). In the majority of cases, the specialist neurologist is better at reading the MRI of their target conditions than the general radiologist. This is handled by the neurologist phoning the radiologist and guiding them to re-work their findings.

The vast majority of AI in software is in radiology. Decades ago radiology was among the first practice groups to fall under private equity patterns because of the cost of the machines. In some cases, specific radiology groups develop protectable expertise around some protocols (i.e., how to get the machine to give better data), but long machine lifecycle creates opportunities for technical one-upmanship.

Knowing that the labeling quality is variable and the images are not entirely comparable, you realize the bulk of AI-radiology may be built on shifting sands, and amounts to workflow optimization.

Worse, the effort could go into addressing what really plagues all providers: the unusability of their EHR's. But that's not particularly monetizable relative to critical diagnostics, and harder than production AI for MRI.

They dive straight into embeddings(they have the freedom in the interview to design something the are comfortable with), which is important to the quality but not necessarily the most(or even the only) challenging part of what the team works on.

Notes:

1. The caveat at the top states "NOTE: This post is not up to my normal writing standard, just felt compelled to get this down in some form. This is more like a blog post entry than a real newsletter addition."

2. The article's final paragraphs are: "Long story short / Thinkers have a pattern where they are so divorced from implementation details that applications seem trivial, when in reality, the small details are exactly where value accrues. / Should you be worried about GPT5 being used to automate vulnerability detection on websites before they’re patched? Maybe. / Should you be worried GPT5 is going to interact with social systems and destroy our society single-handedly? No absolutely not."

P.S. Pedantic timestamp: the quotes above are taken on 12:24 pm eastern time, May 29. Well, this may seem pedantic until the author updates the blog post and then this comment seems erroneous. It is notable that we still don't have a well-accepted standard for snapshotted content.

It's worth remembering exactly what Geoff Hinton said, and how he said it:

“I think that if you work as a radiologist you are like Wile E. Coyote in the cartoon,” Hinton told me. “You’re already over the edge of the cliff, but you haven’t yet looked down. There’s no ground underneath.” Deep-learning systems for breast and heart imaging have already been developed commercially. “It’s just completely obvious that in five years deep learning is going to do better than radiologists,” he went on. “It might be ten years. I said this at a hospital. It did not go down too well.”

https://www.newyorker.com/magazine/2017/04/03/ai-versus-md

So Hinton didn't "decry the decline of radiology". His comment brims with exuberant sarcasm about how his chosen approach to AI is about to not only leave radiologists without a job, but also leave them looking like the short-sighted fools he thought they surely were.

Hinton was cock-sure and arrogant, like an undergraduate student who just trained his first neural net. That is quite unbecoming of the man whom the New York Times, The Guardian, and many other publications have been calling "the godfather of AI", and a Turing award winner, to boot.

That is something he should be called out on, and not allowed to forget. Nor he, nor anyone else who thinks that leaving highly-trained workers without a job is a laughing matter. Nobody should be allowed to make fun of the harmful consequences of the technology they create.

Is that going to move the needle on better healthcare for patients? Will it allow our hospital systems to bring in more throughput of patients at higher quality of care and lower costs?

Thats not what doctors (and hospitals) want. They won't be an ally in "AI for healthcare" unless it means their paychecks/revenues are protected. The author is being misguided by seemingly smart people who have skin in the game and a lot to lose.

Personally, I only favor AI solutions that will drive down the cost of all healthcare, raise the quality of service, and overall expand high-quality healthcare to underserved populations. Anything less than that is only serving the needs of the industry and not patients.

Sharing patient data harms privacy, but in the long run more data sharing is probably the best route for reducing physical harm. The body is just too complicated for individuals to fully grok.

As AI's get exposed via API's and ever easier access we will see more of proliferation of this where jobs which we felt never could be AI'fied are more rapidly done so.

Reminds me of the Lee Sedol vs Alpha Go game where famously it was said at the end "All but the very best Go players craft their style by imitating top players. AlphaGo seems to have totally original moves it creates itself.."

I do echo though the production grade rollout of AI. When we want to jinx the game, we can notoriously play spoil sport - regulation, data protection et. al.

I think the natural excitement we have with any new AI model (or technology) leads us to assume that it will magically get integrated in any existing system and adopted by any user.

Radiologists are certainly performing 2d pattern recognition, as the input they are processing is only 2d even if the details of how that recognition is performed relies on some deeper understanding. Likewise, an AI system performs recognition of 2d patterns based on some deeper "understanding" - in the case of a neural net that "understanding" lies in the complex configuration and weightings of its neural connections built up from vast training datasets.

Even if this dataset is only a subset of the dataset that humans are trained on, we still can't a priori claim that this subset lacks patterns and correlations that escape humans and that allow an AI to make certain determinations better than a human might.

I don't think most people are scared of GPT5. It's AGI that they're scared of.

And GPT5 can destabilize a society because of how fast it could replace workers.

I've been developing software solutions for several years and the fact that a computer can do cool stuff, automate processes, be more accurate... always need users who know and want to use it. So my takeway is that with AI solutions the human part also needs to be considered.

I have no doubt that AI is not going to kill us because 95% of companies won’t even be able to build it.

What others are out there?

datasets are proxies, measurements are fuzzy.

This is something that is drilled into students learning statistics and I can't for the life of me figure out why this changed with respect to ML (best guess is not requiring statistics courses).

Datasets are proxies: they represent the world, but aren't. They should generally be seen as narrow subsets too. Your dataset quality and type matter a lot! Things like medical image datasets also have tons of correlating factors that can easily invalidate all results without you being aware. There are simple datasets we use to prove a concept (toys, mnist, cifar, etc). There are large scale datasets that have internal inconsistencies (imagenet, flowers). There are huge datasets that haven't been properly filtered/deduplicated (LAION). (there are also just shitty datasets (HumanEval)) Thinking of datasets as proxies helps internalize the frustration that literally every production engineer faces (even outside ML and software): real world results are inconsistent with lab results. Dataset engineering is an underappreciated art that is extremely difficult. But everyone needs to internalize that datasets are just a map, not the territory, and your navigation will only be as good as the map (many are poorly drawn maps, often on purpose)

Measurements are fuzzy: Benchmarkism is running rampant in the ML world and it baffles me that a field who's bottomline objective deals with alignment isn't able to align how we evaluate ourselves. No measurement is perfect, many are far from them. You can train two language models to the same NLL and one might sample well and the other outputs garbage. You can train two image models to have identical FIDs and one samples clearly and the other doesn't. Likelihood also doesn't guarantee sharpness and I can go on. You must think about the limitations of your measurements and know them in depth. This also has seem to have gotten away from us and people just use the measurement libraries and call it that. We've reached a point where ImageNet classification accuracy has decoupled from downstream performance (object detection and segmentation), and things like this are confusing to production people because just taking the model with the highest score doesn't always result in the best performing model for their work (even before we consider things like throughput and memory usage). It is a Goodhart problem through and through.

ML is at a serious point where we've gotten away from our basic stats learning. That's going to pose a real danger to society, not AGI. It's like handing powertools to chimps (who don't know how powertools work), it won't end well. But that is happening because we've shifted focus to meet targets, not to measure our work. Targets are easy, science is hard. Unless we bring these nuances back to our evaluation of works then we are just handing powertools to chimps without any quality assurance.

I don't know who Geoffrey Miller I'm pretty sure if there's a Geoffrey who notably predicted the decline of radiology, it was Geoffrey Hinton a few years ago...