Hacker News Reader

Comment by imiric | Hacker News Reader

Good lord. The idea that a system that produces pseudo-random output without any semantic understanding can be relied on to give accurate health-related information is deeply flawed and troubling. It's one thing to use these things for finding patterns in data, for entertainment purposes, and for producing nonsensical code a human has to fix, but entirely different to rely on them for health diagnosis or advice. I shudder at the thought that a medical practitioner I go to will parrot whatever an LLM told them.

This insanity needs to be regulated yesterday.

View on HN

Can you state more clearly why it's deeply flawed?

Because while LLMs obviously have massive limitations, so do humans, and it's not entirely clear to me that some synthesis of the two can't produce much better results than either on its own.

> ... some synthesis of the two can't produce much better results than either on its own.

In theory, I agree with you. The world "some" is doing a lot of heavy lifting there though. I only hope that whatever definition of some emerges, it's not a horribly flawed one.

That's my hope as well. A responsible doctor that uses technology critically and knows its limitations is fine with me. My concern is that it's more likely that whatever AI says will be taken for granted, overruling any human judgment, because they're supposed to be expert systems that scored highly on some made-up benchmark produced by the same company that trained the model.

We see this today with AI-generated content on the web, and a flood of sloppily put together software produced by people who swear that AI is making them more productive. There's little interest in judging the output, and a lot of interest in lazy cash grabs. There are no guardrails in place in the healthcare industry AFAIA to prevent the same happening there, which is a scary thought.

You would not hire or go to a a mentally unstable doctor. If a doc hallucinates or makes a mistake, it will be rare. They can self introspect and work on it if they realise there mistake even. AI is NOT same. It has no self reflection, hallucination isn't rare. AI is not mentally stable in any sense.

Whether you (or anyone else) actually understands something is opaque to me. For all I know, your comment was left by an AI. There is no magic wand we can wave to separate out the people who actually understand something (whatever that means), from the people who just appear to.

The best we can do is test a human's mastery of a subject to estimate how well they actually know and understand that topic. Which is exactly what OpenAI is doing here.

What I care about is the results. If the "grade" is 10%, then I don't want to rely on it, whether it's a human or an AI. If it's 95%, then I feel fine about relying on it. Especially since I suspect that very soon, most doctors would not score as well on a benchmark like this as the SOTA models.

> What I care about is the results. If the "grade" is 10%, then I don't want to rely on it, whether it's a human or an AI.

So you judge mastery of a subject by some arbitrary grade? Produced, incidentally, by the same company that trained the model. But I'm sure they wouldn't cheat...

I also care about the results, but I judge it based on direct experience. IME the current SOTA models can't be relied on to accurately produce working code most of the time, yet I'm expected to trust they can produce accurate healthcare diagnosis and advice? Give me a break.

AI companies have sold us a fantasy that a neat computer trick, which can still be very useful for society, will somehow lead to AGI. Now that we're a decade into this hype cycle and investors are getting antsy, they're doubling down by anthropomorphizing the technology and selling us "chain of thought" and "reasoning", as if making everyone say these things will somehow magically produce intelligence.

What's really irresponsible is that the public and governments are buying into it, which is pushing the tech into industries that can actually harm people. Safety is not just about trusting some arbitrary grade.

IME the current SOTA models can't be relied on to accurately produce working code most of the time

My experience is the opposite.

>Now that we're a decade into this hype cycle and investors are getting antsy, they're doubling down by anthropomorphizing the technology and selling us "chain of thought" and "reasoning", as if making everyone say these things will somehow magically produce intelligence.

The Transformer, nevermind GPT-3 did not exist a decade ago. I guess LLMs aren't the only things that hallucinate and spout confident nonsense.

>I also care about the results, but I judge it based on direct experience. IME the current SOTA models can't be relied on to accurately produce working code most of the time, yet I'm expected to trust they can produce accurate healthcare diagnosis and advice? Give me a break.

You don't see the fallacy of forcing your experience in one domain onto a completely unrelated one regardless of any evidence to the contrary (even if you don't trust open ai, this is hardly the only paper trialing SOTA LLMS for diagnosis)? What does code have to do with diagnosis ? And while the current SOTA is by no means perfect, if you can't get them to produce working code, that's a you problem. At the very least, many users would disagree.

Yeah benchmarks aren't perfect. Doesn't mean they aren't useful. Certainly a lot more useful than your approach.

> The Transformer, nevermind GPT-3 did not exist a decade ago.

I'm attributing the start of the current AI hype cycle to the resurgence of CNNs using GPUs, roughly around the time of AlexNet and AlphaGo, not to LLMs and the Transformer architecture. Though if we're being really pedantic, the original Transformer paper is from 2017, so almost a decade ago. But this is beside my point.

> You don't see the fallacy of forcing your experience in one domain onto a completely unrelated one

The machine has no concept of a "domain". Whether it's outputting code, poetry, images, or video, it's all data generated by probabilistic pattern matching and pseudo-randomness. The structure and accuracy of the generated data is meaningful only to humans, and it's the most important factor that is measured in all these benchmarks.

We might find it collectively amusing when an AI produces funny text and weird looking pictures and video. Some might find it acceptable when it produces buggy code that humans need to fix, or when it mimics an expert by confidently spouting nonsense, which is where I personally draw the line. But we should all be concerned when the same models are used in industries where human lives depend on critical thinking by experts.

We have been promised fully autonomous vehicles for more than a decade now, and only in the last couple of years have some parts of that promise begun to come true in very limited scenarios. We're all understandably reluctant to give control of a speeding 2-ton object to a machine, for obvious reasons. The process has been very gradual, with a lot of oversight and regulation, as there should be. All I'm saying is that there should be an equal amount of oversight in other industries as well, particularly healthcare. Arbitrary benchmarks don't make me trust these systems more, regardless of who produces them.

>I'm attributing the start of the current AI hype cycle to the resurgence of CNNs using GPUs, roughly around the time of AlexNet and AlphaGo, not to LLMs and the Transformer architecture.

The current hype cycle, the one fueling hundreds of billions in investment by multiple of the biggest tech companies in the world has little to do with Alex Net and AlphaGo and everything to do with LLMs and generative AI in general.

>Though if we're being really pedantic, the original Transformer paper is from 2017, so almost a decade ago. But this is beside my point.

The transformer paper did not start the generative AI craze. GPT-3 in 2020 did it for the research world, and the release of ChatGPT in Nov 22 did it for the general public.

>The machine has no concept of a "domain". Whether it's outputting code, poetry, images, or video, it's all data generated by probabilistic pattern matching and pseudo-randomness. The structure and accuracy of the generated data is meaningful only to humans, and it's the most important factor that is measured in all these benchmarks.

What are you on about? Of course it does. This conversations are getting tiring. Yes, LLMs model concepts directly independent of the text it is trained on. This has been demonstrated multiple times including very recently again by anthropic. There is nothing random about the predictions they make.

And even if the machine didn't model these things directly, the concept of domains would still be relevant to the humans testing it as 'data' is not equal.

SOTA LLMs are good for diagnosis. That was evident even before this benchmark. I'm not talking about some imagined future. I'm talking about right now. Sticking your head in the sand because of your 'experience' with coding is nonsensical. The benchmarks aren't arbitrary - In some cases they are directly testing the ability in question.

I'm not advocating for removing doctors from the picture entirely. It wouldn't even be possible even if I was, at least at the moment.

Good lord. The idea that a sleep‑deprived, bias‑prone mammal can be relied on to give accurate health‑related information is deeply flawed and troubling. It’s one thing to seek a human for empathy, small talk, or a routine vaccine, but entirely different to assume an overworked doctor will consistently out‑diagnose a modern data tool. I shudder at the thought that the practitioner I visit might simply repeat whatever a drug‑rep lunch pitch or outdated guideline left in their memory.

This insanity needs to be regulated yesterday.

Thankfully, it is. :)

Are there regulations regarding doctor's working hours similar to say, truck drivers? I found this [1] article which states doctors are limited to 80 hours a week whereas truck drivers are limited to 70 hours in a 7 day period. [2]

[1] https://psnet.ahrq.gov/primer/duty-hours-and-patient-safety

[2] https://www.fmcsa.dot.gov/sites/fmcsa.dot.gov/files/docs/Dri...

Those restrictions are for residents (trainees).

They were also found not to improve patient outcomes (possibly due to increased number of handoffs, which are highly error prone).

Do you think it makes sense that doctors can work longer hours than truck drivers?

If the available evidence does not show an improvement in patient outcomes with shorter hours, and if longer hours are worked willingly and enable them to treat more patients (iso widespread physician shortages), yes I do.

Turning the question around, if mandating shorter hours for physicians were found to result in higher rates of significant patient harm, would you be in favor of those restrictions?

Doctors are in high demand, and don’t willingly work at places the demand long hours. Primary care doctors generally aren’t residents.

Reply on news.ycombinator.com