UnitedHealth uses AI model with 90% error rate to deny care, lawsuit alleges | Hacker News Reader

UnitedHealth uses AI model with 90% error rate to deny care, lawsuit alleges

arstechnica.com

163 pointskurthr58 comments

In my previous role, I worked on something similar to this. We built a model to predict which claims(CMS 1500) would be denied by the carrier(Insurance Provider) before they were submitted and allowing the hospital staff to adjust the Diagnosis/Procedure codes and validate the combination. The client had 20% denials, our model brought that denial rate down to 10%.

We wanted to compare the performance of our model with commercial third party offerings and we enrolled into an early beta for a similar offering from ChangeHealthcare one of the big players in the healthcare industry, their model was wildly off(<50% accuracy), we thought we were missing something and went back and forth with their team, turns out they didnt know what they were doing, they just slapped AI on the API and tried to see if anyone would actually use it. That model never saw the light of day. After studying the data model and seeing the number of variables involved I would be skeptical of prediction models retaled to insurance in the healthcare industry.

Now Change is owned by United Healthcare, so maybe the model you mentioned did live on..

you either die the benevolent language model, or live long enough to become the…

no, don’t like that. do not like that.

> Though few patients appeal coverage denials generally, when UnitedHealth members appeal denials based on nH Predict estimates—through internal appeals processes or through the federal Administrative Law Judge proceedings—over 90 percent of the denials are reversed, the lawsuit claims. This makes it obvious that the algorithm is wrongly denying coverage, it argues.

The '90% error rate' in the title sounded like a pretty serious twisting of some numbers so I scanned the article for where it's quoted from ^, which... makes a mockery of the title.

In case it's not obvious and to save parsing that, 'few people appeal' (which makes sense given the internal appeals process or Administrative Law Judge proceedings). Then, /of those cases/ 90% are overturned.

- If 50 / 10,000 cases and 45 were overturned or 500 / 100,000 cases appealed and 450 were overturned... would all meet this '90% error' mark.

- Similarly, 'overturned != error', because it may be as simple as a policy on United's side to not fight super hard and allow overturns (for any number of reasons, including just saving costs on the proceedings, it needn't be altruistic).

Did Ars used to be better than this, or were they always like this on some issues?

The /business/ side of medical insurance isn't really a topic that I know enough to weigh in on, but hopefully people can read the above sentence and parse what it's actually saying for themselves before reading the article.

Irrespective of how I/we feel about the outcomes, this headline is awful.

Of the only claims that have been adjudicated 90% were found to have been erroneous.

Without analysis there is no particular reason to believe that other denials were more justified.

Few doctors on average make nonsensical requests for patient care while insurers issue spurious denials as a matter of course.

You dont need to know the total number of denials instead you need a neutral party to analyze a random sample of denials.

We could do this as matter of course and tie failing an adequate benchmark to ruinous penalties to company and CEO.

I think the OP is suggesting that hypothetically speaking, people would only go through the hassle of appealing if they were pretty sure they would win to begin with

I think that logic is just as faulty as the assumption that 90% of the un-appealed claims would also be overturned.

I suspect many people just don't know that they can appeal. Those that do might think it's too difficult to do so, or believe it requires some specialized knowledge to do properly.

And this is a perfect example of the type of conversation that happens when the correct answer is that we don't know the answer but everyone keeps talking in circles pretending there's a way to know with any certainty other than testing all (or a carefully chosen random sample) of the other denials and getting the actual data. Whenever there's a disagreement where both sides seem reasonable it means that both sides are wrong because the correct answer is that the information present is inadequate to distinguish. All the potential reasons for things going one way or another are also just hypothesis to test since the gut feeling could be right and the reason wrong, and just getting a percentage on the rest isn't enough to figure out why that percentage is what it is

> testing all or a carefully chosen random sample of the other denials and getting the actual data

We should build an Ai to test this.

Funny.

For a model whose only output is allow/deny, 90% error rate is pretty good, you can just flip the answer and then you have a 10% error rate.

If you know the first thing about AI (or even statistics in general), that headline sounds very suspicious.

What's an acceptable error rate for such a model? What's the human baseline?

I ask these questions[1][2] for nearly every model I'm requested to develop. It usually throws off PMs who aren't used to any acceptable errors.

It's always fun to get a request that's several orders of magnitude better than human performance. Sometimes it's actually possible to deliver, but often it shifts the conversation into feasibility and expectations.

Anyone developing a model like this will have errors and definitionally makes systemic. Regardless of performance it's important to address that.

1. In this order 2. Sometimes I study human baselines but ask before revealing because it prevents anchoring.

Absolutely -- an academic environment I was in used to have a lot of government callers come by, asking for predictive power to solve any number of humanitarian crises, terrorism, crime and so on.

We collectively used to ask them every time,

1. What data they had to work with.

2. What was the human baseline that they were currently executing with.

3. What performance would make a difference to this process.

... and as you say, the answers we would get would tend to sound something like.

1. Almost nothing.

2. Unclear, un-measured.

3. Something well beyond incredible, well-intentioned humans with incredible data.

In that case the lesson was often to simply not try, but it does sometimes feel like other areas of government like E2E-breaking legislation, in that well-intentioned folks will try and try and try again and eventually get someone to fund and execute on the project, feasibility notwithstanding.

In this instance with United... the data that we're getting via Ars and this case is so unclear and one-sided that it's hard to know how well things actually work... it'd be fun to consign them (for now) to "don't even try" if we had the info to do so, but it feels like it's a matter of time either way and so... I think the ideal with most of these cases is to eventually ship something that's actually high quality.

> What's an acceptable error rate for such a model? What's the human baseline?

Funny, I though it was socialized medicine that had "death panels" and such /s

If anything, the "death panels" on private insurers (whatever form they might actually take) are likely much worse than a government-run "death panel".

I haven't heard the death panel canard brought out since Canada started it's death panels. It's like the classic idea, but instead of letting you die from lack of medical care they drive you into depression prior to killing you.

> Are Canadians being driven to assisted suicide by poverty or healthcare crisis?

https://www.theguardian.com/world/2022/may/11/canada-cases-r...

> Why is Canada euthanising the poor?

https://www.spectator.co.uk/article/why-is-canada-euthanisin...

> What's an acceptable error rate for such a model?

Ideally, 0%. We're talking about people's lives and health here.

But, right, realistically:

> What's the human baseline?

Good question, but I would hope it's much better than a 90% error rate. And if it's not, any sane company in any sane industry would figure out why things are that bad, and fix it. Of course, insurance companies are the scum of the earth, and have a huge incentive to get things wrong in this way.

Why is there a different standard between the model and a human?

From the perspective of the business, whether a model or human causes the error is immaterial.

> Any sane company in any sane industry would figure out why things are that bad, and fix it.

The function of a thing is its consequences. So we have to conclude that the reason the model had a 90% error rate was to have a 90% error rate. It's aligned with business incentives to be so.

What kind of an educated scientific person, working in health insurance and dealing with sick people, develops anything that makes this possible?

Regular people will do horrible things to keep their paycheck coming. It's an underrated phenomena of humanity.

I strongly disagree with how this is presented. Doing half-assed or even passionate work for greedy insurance companies is not a horrible thing to do. Not thinking of how these companies can be reigned in and acting accordingly is far worse. We need to stop offloading moral responsibility from the people in charge to the regulae folks.

They are all complicit, the people in charge have no power without their underlings.

They accepted the role and fulfilled the duties that were asked of them by the company. They had all the chances to intervene or divest before such a model made it to production.

This horrendous behavior of health insurance companies in the U.S. is widely understood, reported on, and documented to cause harm and death. If you choose to work for them, you’re of the same ilk of people who work for pyramid schemes and arms manufacturers.

Health insurance companies have a long and documented history of doing horrible horrible things. Are you really saying the rank and file at health insurance companies who are collecting middle class salaries at a white collar job denying people's claims for an MRI for a possible brain tumor have no moral responsibility?

If you can't start with being the change you want to see in the world there is no hope

I think the rank and file at health insurance companies need to be held accountable when breaking the law. The law itself must be designed in a way to prevent such behavior eg.: Any action such as trying to deny medical service or outsource decision making into a gray zone such as ai shall be punished hard- also on an individual level. It is utterly unnecessary to discuss the individuals moral obligations or to start another meta discussion on capitalism. This type of reasoning is extremly harmful as it deflects the needed discussion about the law.

You probably agree with me. It just struck me that our misunderstanding comes from the different approaches europe and the us are following. While we tend to make up rules in advance, the american strategy is often about giving the market more options and having it regulate itself by class action lawsuites down the line. When we talk about health you must admit the „free market“ strategy has failed and europe is doing a far better job.

Haha, literally "I was just following orders" argument.

Moral responsibility exists on both ends, but it's amplified by agency. Those in charge have more agency to make their own decisions, and those at the bottom have less agency because they need to eat. That doesn't completely absolve them of responsibility for their own actions, even if their options are limited in scope.

If your job is to make people's lives worse, at some point you need to take responsibility for choosing to do that job anyway. Very few of the people actually implementing models like this are without options when it comes to providing for themselves in ways that don't cause harm.

One person doesn't do this, a company does, a hierarchy of jobs. An entity without a conscience built from entities with a conscience. Very clever.

The banality of evil.

The irony here is that the entities with a conscience are very much aware of what their doing. Very very much aware.

So there's not much difference is there?

Actually there is. The difference is in a quirk of moral evolutionary psychology. The tragedy of the commons. As an individual each of those entities contributes negligible evil to society. In aggregate that negligible evil becomes prominent.

Morality is divided up into fractional shares and sold like a collateral debt obligation. Every time you step into a car and let that thing spew green house gases into the air you are contributing to the destruction of the world. The individual is still guilty. We call corporations evil, but what we don't realize is that corporations are a mirror of humanity.

There is only one rule: shareholder profits

You're making a lot of assumptions here. Insurance is there to make a buck. Nothing more.

"Would you do X for $Y?"

Most of the time you can increase Y to get the answer "yes".

In many cases Y can even be factored out by changing it to "Would you do X so you wouldn't get fired?"

Empirical answers differ a lot from people's theoretical principles.

Plenty of folks here work on bombs for killing people, surveillance instruments for govt and police to use for spying on citizens, etc - people take paychecks and they take orders

The business of insurance is to make money by denying care that people paid for

What on Earth makes you think that there are less assholes in educated part of the population?

It's a common expression on HN and Reddit that people don't "care" about their job or their coworkers. They produce output, they get paid. Simple exchange. Code for money. So why does it surprise you that someone would see building this as just an exchange?

I worked on healthcare software for over a decade. The healthcare industry attracts the lowest quality people generally. Software engineers aren't giving up working at FAANG to go work in healthcare. A significant portion of UHG's revenue comes from billing and other healthcare software. Most of it is under their Optum brand.

I predict this that AI will be (intentionally) used as a scapegoat for plausible deniability in some industries. "We just followed our AI model." "We thought it was doing the job." "It's too complex to understand."

Obviously once you know or believe that your AI model is wrong, and you are doing a disservice to your clients/patients... and especially once you begin discussing this internally, then you are aware of the problem and should not (get caught) sweeping it under the rug.

You don't need AI for this, software is already treated as something God given by people. Government somewhere wrongly denying foodstamps to a huge amount of people? Software error, nothing anyone could possibly be responsible for. Getting hacked and all your customers data stolen (because you don't follow basic security best practices)? Well that just happens, just like a taiphoon or earthquake, nothing anyone could've done about this. But bonus points for demanding the government do something about those hacker gangs.

"Computer says no" is a catchphrase first used in the British sketch comedy television programme Little Britain in 2004.

https://www.youtube.com/watch?v=0n_Ty_72Qds

CEO of Brawndo: "The stock has dropped to zero and the computer did that auto-layoff thing to everybody!"

That's the power of system defaults. Make "No" the default response, regardless of the validity of the request, and there's a good chance patients that are trying to manage the logistics of appointments, prescriptions and treatments will simply give up.

AI will be used to hold up these existing practices; when glittering visions of lowered healthcare costs are trotted out by the op-ed writers and techno-optimists, this is really how that sausage is being made.

Bureaucracies and authorities love black box models because they can always point to them, shirk their own responsibility for decision making and say they're only following orders and that computers aren't biased.

It's just a coincidence that the model they're paying for keeps making decisions that are beneficial to their own interests and reflect their own biases.

66.5% of US bankruptcies are caused directly by medical expenses.

https://www.retireguide.com/retirement-planning/risks/medica...

We need regulations about what computers should and should not be allowed to do. Computers should not be allowed to make healthcare decisions. Computers should not be allowed to make legal threats or actions.

There needs to be a legal respponsibility of users of AI. AI isn't responsible, those acting with the aid of AI are.

Was denying 90% not the desired outcome?

For an insurance company, this sounds like the ideal model.

Stupid question—if an insurance provider keeps denying claims which could keep elderly patients healthy, and those patients then die, isn't the insurance provider just cutting off its own source of income?

But I agree that it's about incentives, and insurance carriers are incorrectly incentivized for the service they provide vs. what they're expected to provide.

The old adage goes: “those who don’t use insurance, pay too much. And those who use insurance, pay too little”.

It becomes a math problem for insurance companies to keep that equation positive. And since claims far exceed the premiums any single member pays, the easiest way to stay positive is to deny claims. Even if that sacrifices losing a few premiums.

>die, isn't the insurance provider just cutting off its own source of income?

No because that person will have paid right up the the point they become expensive.

Most people incur the vast majority of their health costs in the final year or even months of life. This is why insurance based medicine will always be problematic - there’s just too much incentive to cheap out on people when they are weakest and most powerless.

Not really, the highest medical costs are incurred in old age. Insurance companies would have to make less costly payouts if they didn't have to cover older people.

Another angle is premiums paid vs claims awarded. If an elderly patient has paid in 50k over a lifetime and is making a 100k claim, now is an okay time to lose that customer.

So glad to be a United customer!

Less known for it, but EU GDPR allows for asking for a non automated decision.

It's call UnitedHealthcare

I guess the stupid HN title rule limited the characters

Reply on news.ycombinator.com