Google removes AI health summaries

There's a difference between a doctor (an expert in their field) using AI (specialising in medicine) and you (a lay person) using it to diagnose and treat yourself. In the US, it takes at least 10 years of studying (and interning) to become a doctor.

> This "random output machine" is already in large use in medicine

By doctors. It's like handling dangerous chemicals. If you know what you're doing you get some good results, otherwise you just melt your face off.

> Should I trust the young doctor fresh out of the Uni

You trust the process that got the doctor there. The knowledge they absorbed, the checks they passed. The doctor doesn't operate in a vacuum, there's a structure in place to validate critical decisions. Anyway you won't blindly trust one young doctor, if it's important you get a second opinion from another qualified doctor.

In the fields I know a lot about, LLMs fail spectacularly so, so often. Having that experience and knowing how badly they fail, I have no reason to trust them in any critical field where I cannot personally verify the output. A medical AI could enhance a trained doctor, or give false confidence to an inexperienced one, but on its own it's just dangerous.

> This "random output machine" is already in large use in medicine so why exactly not?

Where does "large use" of LLMs in medicine exist? I'd like to stay far away from those places.

I hope you're not referring to machine learning in general, as there are worlds of differences between LLMs and other "classical" ML use cases.

The "well we already have a bunch of people doing this and it would be difficult to introduce guardrails that are consistently effective so fuck it we ball" is one of the most toxic belief systems in the tech industry.

- The AI that are mostly in use in medicine are not LLMs

- Yes. All doctors advice should be taken cautiously, and every doctor recommends you get a second opinion for that exact reason.

LLM is just a tool. How the tool is used is also an important question. People vibe code these days, sometimes without proper review, but do you want them to vibe code a nuclear reactor controller without reviewing the code?

In principle we can just let anyone use LLM for medical advice provided that they should know LLMs are not reliable. But LLMs are engineered to sound reliable, and people often just believe its output. And cases showed that this can have severe consequences...

Because privatized $$$ and public !!!

depends if the cost of training can be recouped (with profit) from the proceeds of usage. Plenty of inventions prove themselves to be uneconomic.

And that becomes obvious when you go to look for it. These work best in situations where false positives have low cost/impact, but true positives are easily verifiable and have high impact. in other words, problems in epistemological p-space. :)

Agreed. I'd also argue that there will always be the issue of adverse selection, which in any system that doesn't mandate that all individuals be covered for healthcare regardless of risk profile, will continue to raise costs regardless of whether or not margins are good or bad. That dream died with the individual mandate, and if the nation moves even further away from universal healthcare, we will only see costs rise and not fall as companies shoulder more and more of the relative risk.

The solution is single payer. Any attempt to solve this with technological band aids is completely futile. We know what the solution is because we see it work in every other developed nation. We don't have it because a class of billionaire doners doesn't want to pay into the system that allowed them to become fabulously wealthy. People who are claiming AI is the solution to healthcare access and affordability are delusional or lying to you.

Compare: Google's founders can buy all the yachts they could possibly eat, yet Google Searches are offered for free.

If we could get healthcare to that level, it would be great.

For a less extreme example: Wal-Mart and Amazon have made plenty of people very rich, and they charge customers for their goods; but their entrance into the markets have arguable brought down prices.

IDK, the owners of retail clothing chains buy yachts and yet that sector is jaw-droppingly efficient at delivering clothes to people. Executives can be annoying tools but I don't think their pay is the problem.

Many hospitals are already non-profit. That doesn't seem to bring down prices. Why would you think that this would work for insurance?

Profit isn't even a big part of the overall revenue.

> Mandate at least decent minimal coverage standards

I assume you want higher coverage standards than what currently exists? Independently of whether that would be the morally right thing to do (or not), it would definitely increase prices.

> and large insurance pools that must span age groups and risk groups.

Why does your insurance need a pool? An actuary can tell you the risk, and you can price according to that. No need for any pooling. Pooling is just something you do, when you don't have good models (or when regulations forces you).

These solutions are often proposed as easy fixes but I'm skeptical that they actually will do much to reduce healthcare costs. Healthcare is fundamentally expensive. Not-for-profit hospitals and for-profit hospitals don't really substantively differ in terms of out-of-pocket expenditures for patients; I find it difficult to imagine that forcing insurance companies to be nonprofit would do much to reduce costs.

> large insurance pools that must span age groups and risk groups.

What you describe (community rating) has been tried and it works. But it requires that a lot of young, healthy people enroll, and seniors receive most of the care. In an inverted demographic pyramid like most Western economies have, this is a ticking time bomb, so costs will continue to rise.

> Mandate at least decent minimal coverage standards

I think a better solution is to allow the government to threaten in negotiating prices with companies as Canada does; it greatly reduces rent-seeking behavior by pharmaceutical companies while allowing them to continue earning profits and innovating. (I understand a lot of the complaints against big pharma but they are actually one of the few sectors of the economy that doesn't park their wealth and actually uses it for substantive R&D, despite what the media will tell you, and countless lives have been saved because of pharma company profits)

Essentially the gist of what I'm saying, as someone who has been involved with and studied this industry for the better part of five years, is that it's much more complex than what meets the eye.

I just looked it up, and apparently health care costs for pets has gone up in price even faster than for humans.

Pets typically don't have medical insurance, and any insurance that does exist there has a radically different regulatory regime than for humans.

Since 1980 for the US:

CPI has gone up by 3.16% on average per year (x4.17 in total). Human healthcare costs by 4.9% per year (x8.96 in total). And pet healthcare costs by 6.49% (or x17.87 in total).

Dental and vanity surgeries aren't happening in a vacuum. There are baseline costs eg. anesthesia, recovery medications, medical machinery etc which are all bloated due to the rest of industry not being under price pressure (rising tide lift all boats)

It's similar to how AI data center buildout race is raising the prices for consumer electronics in 2026 and beyond. The suppliers have no incentive to sell lower cost products to tiny niche

The uninsured medical market is actually quite efficient. e.g. lasik surgery can actually be done for very reasonable rates with full price transparency.

> This argument doesn’t make sense to me. Insurance companies are structurally incentivized to minimize payouts across the board. They want hospital bills lower, physician compensation lower, and patient payouts as small as possible. If insurers had unilateral power, total medical spending would collapse, not explode.

They absolutely do not.

They have their profit levels capped at 15% by law and regulation. That means if the insurer wants more absolute dollars of profit, prices must go up.

It also means that if they push prices down they necessarily have less funding to administer those plans, even if the needs are the same (same number of belly buttons, same patient demographics and state of health).

As you note there's also other variables, but this claim: "Insurance companies are structurally incentivized to minimize payouts across the board" is absolutely and categorically not so.

Physician reimbursement is only ~9% of national healthcare expenditures.

I tell you this with certainty as a 3rd year medical student: If physician wages go down and tuition stays as is, no one will do this. Intrinsic motivation to help people evaporates as soon as you see how enshittified healthcare in the US has become.

I do agree that medical school is far too restrictive to get into (For MD schools at least). However, if you want to make medical school easier to get into: Where will all those students rotate at for their clinical years? There aren't enough spots in hospitals to jam students in.

Stop taking aim at the people that sacrifice so much to help you. Take aim at the real drivers of healthcare expenditures: administrative bloat.

I don't think physician compensation per se is a good metric for capturing the effect of lack of providers, because some of the increased costs are due to the bottlenecks in the services per se, in terms of procedure costs and types of procedures offered. I also don't think the number of providers per se under the current regime, without deregulation or reregulation of practice boundaries, is representative of what would happen if there were changes in those boundaries. Adding more optometrists 5 years ago isn't the same as changing what they're allowed to do. It also doesn't address what cost increases would have been without an increase in the number of providers.

9% might also seem pretty big to me if it's out of all spending and doesn't include other provider compensation? What if overall healthcare costs went down, but physician compensation stayed the same? Would that then be a problem because it was an increased proportion of the total costs — fat left to be trimmed, so to speak?

There are many problems that don't have anything to do with providers per se, but I also don't think you can glean much by extrapolating to more of the same, especially compensation per se.

It's really just not there yet. I've been in medical school for >3 years now and have been using the latest models with good prompting. They have gotten much better, but I still see misses that my classmates would easily catch. This is not acceptable in healthcare. It's certainly not getting 100% on all my assignments, which are a step below the complexity of real-world clinical practice.

Before medical school, I was not so sure of the quality of your average doc. Now having spent a year in clinical practice across various settings, I am extremely reassured. I can say with certainty that a US trained doctor is miles ahead of AI right now. The system sucks really bad though and forces physicians to churn patients, giving the impression that physicians don't pay attention/don't care/etc.

> wronging

Not sure if this is a typo (of wringing) or a pun, but it's apt either way.

Salaries for healthcare workers make up only a small portion of expenditures. You do not want to avoid a trip to your GP for an AI system.

It'll be great at first while in development. But when profits need to be generated, seeing a specialist will get harder. There will be less wiggle room. I predict we will see more GP utilization.

That's a different class of problem. It will do just fine on text based queries spanning a few pages. Probably better than the average physician (average over all countries).

I do agree that LLM's are not there yet in the image part.