AI is making critical health care decisions, but is unregulated | Hacker News Reader

AI is making critical health care decisions, but is unregulated

politico.com

35 pointsisaacfrond49 comments

Should we set a requirement that AI models used in certain decision making have to be "static" and "deterministic"? Meaning it would be possible to inspect and re-run cases through the specific version of model that was in use at certain point in past?

You could take the model that was in use 2024-19-02T10:54:00Z, feed in data and get always the same results. Then tinker with the inputs and observe the results to find out for example potential biases or errors.

Why?

Surely the key question is "Is it correct?", not if the same words were used?

In very few situations you get a "yes" or "no" to that kind of question or you have a lot of things that are correct but not all of equal quality etc.

Yep. Far more important to, 1 - get a result consistent with both reasonable diagnosis and standard of care for the condition in question. And, 2 - get the same result given the same input every time. The FDA doesn’t like when you say “amputate the left leg” one day, and then say “amputate the right leg” the next day. Particularly when that happens after you’ve already amputated the left leg and there was nothing wrong with it.

Believe it or not, that amputation scenario has happened before. It wasn’t with AI software, but rather imaging software that hosed an orthonormal conversion on a DICOM dataset. But the same rules should obviously apply to AI.

This sounds a bit like using a standard manual (not the wrong leg part, though).

Well, they don’t generally think it’s necessary to put “don’t amputate the wrong leg” in the books in med school, but yes, if it helps to think of it that way, you can.

In your leg-example your condition #2 was not met, let's imagine that still happens despite using "AI magic":

At the point when "amputate the right leg" was found to be the correct diagnosis, an authority should be able to audit and reconstruct the wrong "amputate left leg" diagnosis, trace the root-cause of this decision and verify the corrective action.

For this, it is important to be able to reconstruct the process of any computer-aided recommendation used. A corporate statement like "we don't know why the model recommended the left leg back then, now it correctly recommends the right leg when feeding the same data" should not be sufficient.

Of course it should. In other tech, if somethung is tested with a measurment tool, there are also numbers to calibration certs on your cert. So you can follow back the tools to the national standarts. So yes of course you should know what version checked your data.

Or, in the words of the European Union:

"AI systems [need to be] safe, transparent, traceable, non-discriminatory and environmentally friendly."

And:

"AI systems should be overseen by people, rather than by automation, to prevent harmful outcomes."

Source: https://www.europarl.europa.eu/topics/en/article/20230601STO...

I sometimes feel people overestimate the ability of doctors as a final decision maker. I used to make voting software for a large medical conference, and spent over a decade or so attending these events. Doctors would often present two (nearly) identical scenarios, i.e. a severe condition a patient has, and then proceed to try two different approaches in treatment. The follow-ups on this are presented at these conferences, to educate others and continue figuring out which procedures may be better in certain scenarios. The most shocking part about this all, to me, was the fact that they often just have their "own opinion" about the best treatment, only to get lectured (if not roasted) shortly after (in the Q&As) about why the method was long outdated, or not a good choice because of the patient showing X, Y and Z, etc. — all while two photos are up on the projector, of two children, one with "less than ideal results" and one with "perfect recovery" — two stories that could have been identical, but weren't, because we humans need to experiment and figure things out, slowly. And two other doctors present afterwards, doing the exact same thing, slightly differently, and thanking the first person about sharing it so they can also try it next time. Long story short - it felt SUPER ancient and error prone, at the cost of human life (and quality of life), and I often found myself thinking about what would be possible if we could let computers work with collective data and suggest based on probability rather than "winging it based on an (educated) hunch" the doctor has, just to have something to present next year. Side note: they were all absolute experts in their fields, of course, I am painting this a bit more black than it may be, but I remain quite shocked when I learned how doctors come to decision making in medical procedures.

I had similar experiences working in health tech. I am still shocked to have learned that it can take decades for new (proven) treatment procedures to make it into general application in healthcare.

tbh the same probably is true of tech and any other industry? People generally risk adverse (No one gets fired for buying IBM)

Risk aversion would mean you try to be on the forefront. As medical advances are often rigorously tested against the status quo. So the new version more often than not should have less risk than the old, no?

> all while two photos are up on the projector, of two children, one with "less than ideal results" and one with "perfect recovery"

Seems like you still could argue the bad outcome have better a priori estimated outcome with that small sample size though?

I fear "the other extreme" would be doctors chasing fads.

Just a reminder that some doctors are still giving covid patients anti-biotics.

In the EU if a software is influencing diagnosis decisions of a doctor in any way it needs to be registered as a medical device software. Probably a similar regulation can be expected in the United States too.

People already use ChatGPT to determine if a doctor's visit is necessary, so AI is already in the decision chain.

From my own experience, most medical software is sold in the USA and must be certified by the FDA which is equivalent to the CE process.

This is an interesting case to analyze. Health web pages has a disclaimer saying something alike: "The information provided on this website is for general informational and educational purposes only and is not a substitute for professional medical advice, diagnosis, or treatment. Always seek the advice of your physician or other qualified health provider with any questions you may have regarding a medical condition. Never disregard professional medical advice or delay in seeking it because of something you have read on this website.".

It doesn't matter if it is AI or any other technology the point is about relying on computers instead of humans and breaking the rule number #0. Not saying that AI couldn't be better than humans in the future but saying that you cannot jump over rule #0 easily.

Sure, but even with that disclaimer the website still has to display at least an attempt at not providing you with blatantly false information. If it literally said "in case of a cold, drink a glass of bleach" then no disclaimer would save them in the resulting criminal case(or at least I hope that it wouldn't).

Not saying that the disclaimer is enough, I tried to express the usin AI is a concern because you are bypassing doctors.

Recently my girlfriend was sick and after seeing multiple doctors in Germany who all had long delays, unfriendly assistants, little time, and conflicting conclusions, I have high hopes for AI. It’s easy to improve the situation from where it is now.

I'm sorry but surely it is even easier to see AI worsening the situation? Would you be happier if your girlfriend received the wrong treatment from an unaccountable system based on decisions made for unknowable reasons, as long at was in a timely manner?

Can't the system be held accountable? Unknowable reasons holds for doctors too, by the way.

Well no they're quite famously unaccountable. IBM even had a mantra that stated "A computer can never be held accountable, therefore a computer must never make a management decision". Re the second, if you are prescribed a certain treatment by your doctor, it is entirely possible to find out why that treatment is proposed.

This seems to be a rather superficial article:

> When the FDA approves drugs and medical devices, it doesn’t need to keep tabs on how they evolve.

The FDA clearly keeps an eye on ongoing developments.

If your company produces an FDA approved drug, be aware you might get a visit by them to check on you, especially if you reported - oh yes, that’s mandatory - an unexpected deviation from your normal FDA approved processes.

And from the reactions I have seen by such companies, those FDA visits are no fun thing to look forward to either, even here in Europe.

> Doctors are already using unregulated artificial intelligence tools such as note-taking virtual assistants and predictive software that helps them diagnose and treat diseases.

That sounds to me like the doctors are making the decisions. And they are regulated.

If they are concerned about malpractice suits due to AI dependence, I'm sure they'll curtail its use or make sure the AI-makers are liable. And that liability is what causes meaningful regulations. That kind of liability is also what separates actual engineering professions from most "software engineering."

It is regulated in Europe via the AI Act, effective in 2026 (iirc).

It’s regulated everywhere with sensible health regulation. If you intent to have your software to make decisions about diagnosis or treatment. You are a medical device with all that follows from that.

i'd genuinely rather take my chances with an unregulated AI than what is currently available in the UK.

Frankly, looking at the trajectory of UK healthcare and the huge lack of skilled and well-paid staff, I wouldn't be surprised if the government embraces AI already tomorrow and creates favorable laws for the supplying companies around that, just to score a quick win with the public and press...

Something along the lines of:

#1: If an AI creates a diagnosis and the AI-model was audited by a doctor, each diagnosis shall be treated like it was made by a doctor.

#2: If an AI creates a diagnosis, the issuing doctor's office is eligible for 10% of the contractually agreed insurance compensation-fee.

People are making critical health care decisions, but are unregulated.. Regulate them!

That’s right. AI should not be regulated, people and organizations using it should be

I'm sure companies would love that.

Business model of big-corp X:

"We deploy an AI model to make 100 million health care decisions per day and the AI-model is supervised by a independent certified doctor we contracted. Legally it should be treated like that doctor made those decisions, since he supervised the model.

In case of any regulatory breach, the doctor who was hired for the task should be prosecuted, we replace that doctor, tweak the model and keep going.

This way we can charge 100 million units of health diagnosis, and based on custom parameters we can decide i.e. which brand of painkiller should be prescribed.

If that's not legally abstracted enough: Let's license the model to doctors based on a per-decision-license, allowing them to boost their daily insurance-claims for a small revenue-share, but requiring them to approve each diagnosis in some kind of UI"

Just because you hired A doctor doesn't mean your company isn't liable. Otherwise companies would pull this any time they get sued.

And in your second case insurance companies wouldn't insure doctors that did that if they were liable. Assuming the diagnosis comes out bad.

In a world where "AI should not be regulated", there are lots of possible corporate constructs where the "people and organizations using them" are decoupled from the entity which owns the AI, securing it from ever becoming liable for its actions.

Already today it's really hard to prove liability for a car company providing self-driving services and not just the driver using them. And that area IS REGULATED.

Ok so add the people/orgs running it as well as those using it.

Otherwise you risk neutering the tech itself rather than neuter bad/ignorant actors.

As long as unregulated AI doesn't spend a decade causing opioid addictions, collude to reduce labor supply of doctors, and cause a decline in lifespan while increasing costs then I'm willing to give it a shot over what regulation has given us.

Or... the reduction in regulation could rather increase the problems? I'm not definitively saying that'd be the case, but there have been cases in other industries that I've at least heard about, such as the correlation, perhaps causation, between decreased railway regulation and large train crashes and disasters.

The beauty of this is, such "unregulated AI" would most likely be created by the companies making huge sums of money in the current flawed regulated market.

Why on earth would they fund the development of an AI which serves against their own interest of securing and further growing their business and revenue...?

It's more likely that they are such experts already in how these pesky regulations are hindering their prospects that they would jump on the opportunity of using AI to further redefining the rules in their favor...

And this is a good thing. I know MANY doctors who are getting out of healthcare and going into business in the USA due to the many problems that range from regulation to BU problems to - well, many other issues. People should be able to take risks as long as they know what the risks are and regulatory clamps need to be released.

How do you make sure people understand the risks in a more "free for all" world and who pays for when those risks realize?

There are many ways to do this - AI naturally facilitates this very thing in a robust manner. Frankly I have used it to create various "studies" with incredible success when you go deep enough with it. What we really need is MedGPT and not regulations - a ChatGPT that is simply highly specialized and gives warnings out the wahoo.

How exactly? How did you test the results from your studies? Even for experts not easy to understand or articulate the risks in a lot of situations. Listing a long set of warnings isn't usually helpful in making an good decision.

Navigating the complexities of AI in healthcare isn't about embracing regulation or throwing it to the wind per se; it's about smart integration. "MedGPT" wouldn't just be a tool—it's a leap towards democratizing medical knowledge, with the power to sift through data and present nuanced insights that can guide decisions. There are definitely ways to do this that go far outside the scope of this format of discussion. When it comes to understanding risks, the clarity and depth AI can offer are unparalleled. By leveraging AI, we're not just throwing caution to the wind; we're arming ourselves with a precision tool. Testing? It’s through iterative, real-world application and feedback, refining our approach as we learn. This isn't about replacing human judgment; it's about enhancing it with comprehensive, data-driven insights. Let’s focus on how AI can transform healthcare, driven by innovation and guided by wisdom.

The ONLY advantage of going to doctors or large hospitals is that they tend to have more data, but they (the doctors) admit that their hands are tied by so many regulations that they are being prevented from healing people. This has created a very toxic ecosystem for healthcare globally that is not driven by sincere and true interests of healing people. My grandfather convinced me to get out of healthcare due to his many decades in the industry and seeing it just completely disintegrate from circa 1950-2000. He implored me to go into a different field so as to not be caught up in an industry that does not truly cure. I'm grateful for his wisdom. His entire family were doctors and owned a hospital even. He had a very deep and well aged perspective of the development we have seen over the past century.

Nuanced insights rarely work with people, if it works in your setting: great. On the side of the physician, perhaps it can work better, but even then for some fields I am not certain nuance works well.

Doctors not only have (potentially) more data, they can also do stuff like touch, smell, etc. Most doctors I know don't seem to have the issues you describe - they are allowed to work on curing/treating patients (yes, there is paperwork, economic limits, etc., but overall the loss-making side of, for example, large hospitals is just accepted).

An airline was sued and lost because their chatbot provided incorrect information. What do you think will happen if MedGPT makes a mistake, not to mention the risk of killing someone?

Who will be liable?

You don't want another https://en.wikipedia.org/wiki/Therac-25

Many more will go out of healthcare once the medical insurance industry is allowed to demand that their customers use their AI software for the initial visit instead of going to their general practitioner, and once hospitals start using a few doctors as rubber-stamps for decisions made by their AI software. I suspect that the eventual net benefit to society will be positive. But the transition will be bumpy, with most people supporting deregulation until another Elixir Sulfanilamide event shifts the pendulum in the other extreme direction. I might be wrong, but I think that people aren't great aiming for the pragmatic balance.

People are pretty bad at understanding risks.

Sounds more like blame shifting.

Reply on news.ycombinator.com