Hacker News Reader

Comment by bearjaws | Hacker News Reader

It's not just employers, its everywhere.

In healthcare, insurers are using "AI" to deny insurance claims. The doctors are then just ChatGPT to generate the messages refuting the denial.

View on HN

Using unvalidated algorithms (including AI) when life is on the line should not be legal.

This is clearly the direct case for medical devices, but there should also be a standard when algorithms are being applied that moderate access to healthcare too - because they have effects exactly like that of doctors making medical decisions.

The issue is that a case worker that has been instructed to delay processes (or has incentive structures in place to do so) is equally bad.

I think we just need regulation. Not regulation that takes AI into account.

Agreed, I think of a standardized clerical process (even if wholly administered by humans) as an algorithm.

I’m not a lawyer but I suspect there are regulations that apply to unjustified denial of insurance healthcare services - they don’t seem to be closely enforced and moreover there is also a frustrating magical enforcement loophole when software becomes involved where even previous precedents seem to need to be reestablished just because tech doing the denial is somehow different than people following a process.

> Using unvalidated algorithms (including AI) when life is on the line should not be legal.

EU AI Act forbids that.

If an employee is pressured to deliver faster, and the employer looks the other way when chatgpt comes out - then there is little for the regulator to do. If you penalize the employee, another will just do the same.

As a regulator, you don't penalize the employee; you penalize the company as though it were a wilful violation.

The company is much larger and incentivized to make it look like an employee issue. Unless there are active mechanisms to prevent this, it will be the default outcome.

In European bank I worked at, it was worked around by having an employee "reviewing" the "recommendation" given by AI, and making the final decision. The final decision was of course 100% in line with AI "recommendation".

The AI act is less than a month old; you were likely dealing with either older or local legislation, or some attempt at corporate responsibility (possibly risk management; if you tell your financial regulators “yeah, an unverifiable magic box makes the lending decisions, unreviewed”, you will likely get in trouble, at least in the post-noughties-financial-crisis era).

The EC is usually rather sceptical of attempts to work around the rules.

GDPR Article 22 has been in force for a long time:

> The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her.

https://gdpr-info.eu/art-22-gdpr/

Yup, that's my worry about these things, too.

What would a "validated" algorithm even look like? (I suppose you're not thinking of mathematical formal proofs)

Well if there are high denial rates for procedures that are within the accepted body of medical practice then the claims process is standing in the way of individual healthcare.

Who validates that a procedure is accepted practice, doctors or insurance accountants.

That sounds more like black-box testing rather than validating the algorithm itself

Insurance claim processing is already so bad that it needs drastic regulation, even if there no AI. I have a feeling these companies save money by exhausting patients. How many times can you call and be on hold for an hour only for them to tell you they’re still looking into it and need 30 business days?

Claims processing is working exactly as designed. Money comes in, it does not go out. If money goes out the process is optimized further.

I don’t get why we aren’t able to regulate them further. Fines for each day a claim is open past some limit. Fines for average wait times that are too long. Jail time for executives. Etc

I remember reading an article about some company trying to sell their service to school districts, it uses "AI" to grade assignments, add comments, etc... the safeguard was that teachers have to read the AI comments before basically clicking on an "Accept" button.

This is the future (present?), students using ChatGPT to write their papers, teachers using ChatGPT to grade.

It's gonna be turtles all the way down.

Southpark covered this already - https://en.wikipedia.org/wiki/Deep_Learning_(South_Park)

I know it's not an amazing safeguard, but providing that alternative would be a deterrent for some teachers just blindly feeding it into an AI and not reviewing the outputs critically. I don't think many teachers would do that, but they are quite overworked as well.

Any source for the chatGPT claim? Typically (in the US) when doctors disagree with an insurance denial they have to do a "peer to peer" call, and plead their case with a doctor employed by the insurance company.

AI is a gamechanger. For grifters and con-artists.

I work in AI because I think it’s a cool/fun piece of technology.

I also believe that AI is a terrible business/product.

I think it is a wonderful product because it can be a great accessibility tool. I use it for speech recognition, cleaning up my writing(speech mis-recognition errors) and programming.

Reply on news.ycombinator.com