Hacker News Reader

Related similar thing when I sent my dog's recent bloodwork to an LLM, including dates, tests, and values. The model suggested that an advancement in her kidney values (all still within normal range) were likely evidence of chronic kidney disease in its early stage. Naturally this caused some concern for my wife.

But, I work in healthcare and have enough knowledge of health to know that CKD almost certainly could not advance fast enough to be the cause of the kidney value changes in the labs that were only 6 weeks apart. I asked the LLM if that's the best explanation for these values given they're only 6 weeks apart, and it adjusted its answer to say CKD is likely not the explanation as progression would happen typically over 6+ months to a year at this stage, and more likely explanations were nephrotoxins (recent NSAID use), temporary dehydration, or recent infection.

We then spoke to our vet who confirmed that CKD would be unlikely to explain a shift in values like this between two tests that were just 6 weeks apart.

That would almost certainly throw off someone with less knowledge about this, however. If the tests were 4-6 months apart, CKD could explain the change. It's not an implausible explanation, but it skipped over a critical piece of information (the time between tests) before originally coming to that answer.

The internet, and now LLMs have always been bad at diagnosing medical problems. I think it comes from the data source. For instance, few articles would be linked to / popular if a given set of symptoms were just associated with not getting enough sleep. No, the articles stand out are the ones where the symptoms are associated with some rare / horrible condition. This is our LLM training data which are often missing the entire middle part of the bell curve.

For what it's worth this statement is actually not entirely correct anymore. Top-end models today are on par with diagnostic capabilities of physicians on average (across many specialties), and, in some cases, can outperform them when RAG'd in with vetted clinical guidelines (like NIH data, UpToDate, etc)

However, they do have particular types of failure modes that they're more prone to, and this is one of them. So they're imperfect.

This is ChatGPT's self assessment. Perhaps you mean a specialized agent with RAG + evals however.

ChatGPT is not reliable for medical diagnosis.

While it can summarize symptoms, explain conditions, or clarify test results using public medical knowledge, it: • Is not a doctor and lacks clinical judgment • May miss serious red flags or hallucinate diagnoses • Doesn’t have access to your medical history, labs, or physical exams • Can’t ask follow-up questions like a real doctor would

Sorry, I should have clarified, but no this is not ChatGPT's self assessment.

I am suggesting that today's best in class models (Gemini 2.5 Pro and o3, for example), when given the same context that a physician has access to (labs, prior notes, medication history, diagnosis history, etc), and given an appropriate eval loop, can achieve similar diagnostic accuracy.

I am not suggesting that patients turn to ChatGPT for medical diagnosis, or that these tools are made available to patients to self diagnose, or that physicians can or should be replaced by an LLM.

But there absolutely is a role for an LLM to play in diagnostic workflows to support physicians and care teams.

I very often also get better programming results than less experienced engineers, even though I'm not remotely doing any kind of "prompt engineering".

Also how you ask matters a lot. Sometimes it just wants to make you happy with whatever answer, if you go along without skepticism it will definitely make garbage.

Fun story: at a previous job a Product Manager made someone work a full week on a QR-Code standard that doesn't exist, except in ChatGPT's mind. It produced test cases and examples, but since nobody had a way to test

When it was sent to a bank in Sweden to test, the customer was just "wait this feature doesn't exist in Sweden" and a heated discussion ensued until the PM admitted using ChatGPT to create the requirements.

This same phenomenon is true for classic search engines as well. Whenever I am becoming informed on a new topic my first searches are always very naive and targeted just at discovering the relevant jargon that will let me make better searches. It turns out that many disciplines contain analogue concepts, just with different words being used to describe them. Understanding the domain specific language used is more than half the battle.

Yeah I reminder that being the trick to get Google to provide good results was to find some key industry or area terms for what you were looking for. This doesn’t work anymore because Google search has gotten so bad

If I’m dealing with an unfamiliar domain, my next step is always wikipedia or an introductory book. Just to collect a sets of keywords and references to narrow my future searches. I don’t think I’ve ever asked google a question shaped query.

The difference is that in the search engine output one can somewhat judge if a site is trustworthy. E. g. if I research a legal question I would trust a guide on a government website more than an answer on Reddit which contradicts an official source. With LLM I don’t know where weight for a particular answer came from. You can ask LLM for sources but its text answer can still misrepresent these sources.

This anecdote corroborates my theory that it will still be critical to become an expert in your field. Everyone is treating AI like it's a zero-sum game with regards to jobs being "lost" to AI, but the reality is that the best results will come from experts in the field who have the vocabulary and knowledge to get the best answers.

My fear is that people treat AI like an oracle when they should be treating it just like any other human being.

What percentage of people are actually experts at their jobs though?

People treat certain humans, or humans in certain roles, as oracles too.

This is just bad design. Or a faulty tool. Why should the job market shift to accommodate this gap in the functioning of LLMs? This is a bug that needs to be fixed.

I have a personal gripe about this bringing an unfinished tool to market and then prophetizing about its usefulness. And how we all better get ready for it. This seems very hand-wavey and is looking more and more like vaporware.

It's like trying to quickly build a house on an unfinished foundation. Why are we rushing to build? Can't we get the foundational things right first?

Maybe try a 2-step approach: first ask the LLM to translate your question into expert-language, then ask that question :)

The expert's answer when asked in expert language is "it's complicated". :-)

I had a similar experience. I did some back of the envelope math and my wife suggested I run it through ChatGPT. After actually doing the math, I felt a lot better about my understanding of the problem and I just... don't trust an LLM to understand algebra. Yeah, they're awesome most of the time, but I don't want to trust it with something important, be wrong, and then have to explain to someone that I trusted the opinion of a couple of big matrices, over my own knowledge and experience, on a high school word problem.

I've noticed Grok struggles with dates and relative time, especially when referring to things it "remembers" from earlier conversations. Even a phrase like "last night" will be wrongly interpreted sometimes. So although I've had it research numbers and create estimates for me, I wouldn't just assume the numbers are right without checking everything.

Since LLMs are basically linear algebra all the way down, this is vaguely reminiscent of how human brains also have a very hard time understanding neural circuitry despite literally being made from it.

For the sake of discussion I want to play devil's advocate concerning your point

> It was an interesting result to me because it shows that experts in a field are not only more likely to recognize when a model is giving incorrect answers but they're also more likely to get correct answers because they are able to tap into a set of weights that are populated by text that knew what it was talking about. Lay people trying to use an LLM to understand an unfamiliar field are vulnerable to accidentally tapping into the "amateur" weights and ending up with an answer learned from random Reddit threads or SEO marketing blog posts, whereas experts can use jargon correctly in order to tap into answers learned from other experts.

Couldn't it be the case that people who (in this case recognizable to the AI by their choice of wording) are knowledgeable in the topic need different advise than people who know less about the topic?

To give one specific examples from finance: if you know a lot about finance, getting some deep analysis and advice about what is the best way to trade some exotic options is likely sound advice. On the other hand, for people who are not deeply into finance the best advice is likely rather "don't do it!".

In some cases, sure, but not here—neither option had more risk associated with it than the other, it was just an optimization problem. The first answer that the model gave to my wife was just wrong about the math, with no room for subjectivity.

> Couldn't it be the case [...] need different advise than people who know less about the topic?

> for people who are not deeply into finance the best advice is likely rather "don't do it!".

Oh boy, more nanny software. This future blows.

> Oh boy, more nanny software. This future blows.

I think this topic is a little bit more complicated: this is rather a balancing of the model between

1. "giving the best possible advice to the respective person given their circumstances" vs

2. "giving the most precise answer to the query to the user"

(if you ask me: the best decision would in my opinion be to give the user a choice for this, but this would be overtaxing to many users)

- Freedom-loving people will hate it if they don't get 2

- On the other hand, many people would like to actually get the advice that is most helpful to them (i.e. 1), and not the one that may answer their question exactly, but is likely a bad idea for them

Everything can always be more complicated, of course. For example:

1. The AI will never know the user well enough to predict what will be best for them. It will resort to treating everybody like children. In fact, many of the crude ways LLMs currently steer and censor are already infantilizing.

2. The users' benefit vs. "for your own good" as defined by a product vendor's financial interest is a scam that vendors have perpetrated for ages. Even the unsubtle version of it has a bunch of stooges and chumps that defend it. Things will not improve in the users' favor when it's harder to notice, and easier to pretend they're not malicious.

3. A bunch of Californians using the next wave of tech to spread cultural imperialism is better than China doing it, I guess. But why are those my options?

This is my experience with prompting as well, but I struggle to describe it adequately. Something like the "direction" of the prompt, if it's too open ended you're likely to get mixed results, but if you give it a kind of "running start" it performs much better.

What I do is to always set the context by giving the my "background" and some papers as reading material such that I've conditioned the model for whatever topic that will be discussed as the first step.

A couple more alternative explanations: 1. random - sample size is too small to claim otherwise 2. phrasing in both cases had a leaning in it, and as a "yes man" LLM gave corresponding biased response.

Interesting observation. I find myself using ChatGPT to find the proper words for something. I describe my problem or algorithm in a naive way, and usually ChapGPT will present some naive python solution. But when pressed ChatGPT will tell you this problem is actually say Graph Theory. But it will still fall back to the naive python solution.

So then I just start a new "conversation" and ask which graph algorithm applies to my problem and instead of a naive solution I am pointed to an optimised algorithm and a usable library that implements it.

Using LLM's in any unfamiliar context is dangerous. When asking them anything, always follow up with: "Are you sure?".

Far too often it'll cheerily apologise and correct their own answer.

Not necessarily correct the answer! Sometimes it just changes it, insists that it is correct - but it might still be wrong (perhaps now in a new way)...

First, these tips and tricks that ”works for me” aren’t universal, it may or may not work, there’s literally no way to tell other than to run large scale empirical experiments, otherwise it’s just mythbuilding.

Secondly, it can be even worse. I’ve been ”gaslighted” when pressing on answers I knew were incorrect (in this case, cryptography). It comes up with extremely plausibly sounding arguments, specifically addressing my counterpoints, and even chain of reasoning, yet still isn’t correct. You’d have to be a domain expert to tell it’s wrong, at which point it makes no sense to use LLMs in the first place.

Oh for sure, asking "Are you sure?" isn't a trick to improve accuracy, it's a trick to show people the danger of asking it in the first place.

It just leaves you with two contradictory statements, much like the man with two watches who never knows the correct time.

Ah, I think I misunderstood the sentiment. Thanks for clarifying

Often it'll even apologise and correct it's own answer despite the original answer being correct, because you just primed it's model to believe it was wrong.

I asked Grok for advice on a personal decision I was making. It suggested I do A. I said I was leaning toward not-A, and explained why. Then it said I should do not-A. I asked why, if not-A was the best choice, it hadn't said that in the first place. Was it unable to think of not-A before I mentioned it? It said no, it had considered not-A from the start, but thought A was the best choice for me until I gave it more context.

That reminded me how important it is to give it the full parameters and context of my question, including things you could assume another human being would just get. It also has a sort of puppy-dog's eagerness to please that I've had to tell it not to let get in the way of objective analysis. Sometimes the "It's awesome that you asked about that" stuff verges on a Hitchhiker's Guide joke. Maybe that's what they were going for.

It should have told you it has no concept of A or not-A, doesn't perform anything close to thinking, and was just picking the most likely words to follow the prompt and context window both times. The people who program it weren't "going for" anything as they couldn't "go for" any specific response. And the model has no concept of self, jokes, or even perception by an outside entity. But it will pick phrases that mimic its training set and those might happen to be about any one of those topics.

I wonder how you are to tell when it does give you the correct answer, and one asks "Are you sure?" and it cheerily apologizes and give you a new incorrect answer.

... But does it even actually try to evaluate the veracity of what it just output? Or is it merely modelling the idea that being asked "Are you sure?" is a reason for self-doubt?

Do LLMs "evaluate the veracity" of anything? That's not really how they work.

LLM users are highly susceptible to the confirmation bias, by putting their expectations into the prompt.

So LLMs need an ELI5 mode and detect when to use it. This would use the non-technical terminology but pull concepts from the more jargony model.

This may or may not be easily possible by tweaking current training techniques. But it shows the many edge cases still needed to be addressed by AI models.

I'd like to see the before and after questions because it seems possible that the layman's version was less exact and therefore interpreted differently, even if the intention was the same. Which, can happen with humans too.

Given the topic I'm unfortunately not comfortable sharing the details in a public space like this, but the answer that it gave was not just a misinterpretation of the question, it was actually entirely wrong on the merits of its own interpretation.

And even if it were a misinterpretation the result is still largely the same: if you don't know how to ask good questions you won't get good answers, which makes it dangerous to rely on the tools for things that you're not already an expert in. This is in contrast to all to people who claim to be using them for learning about important concepts (including lots of people who claim to be using them as financial advisors!).

If you don't know how to ask a human doctor a good question you can't expect to get a good answer either.

The difference is that a human doctor probably has a lot of context about you and the situation you're in, so that they probably guess what your intention behind the question is, and adjust their answer appropriately. When you talk to an LLM, it has none of that context. So the comparison isn't really fair.

Has your mom ever asked you a computer question? Half of the time the question makes no sense and explaining to her why would take hours, and then she still wouldn't get it. So the best you can do is guess what she wants based on the context you have.

Doesn’t matter. The LLM’s job should be to deliver results in the way that is intended, regardless of the skill of the prompter. If it can’t do that then it’s really no better than a Google search.

That is a pretty high bar. Humans aren't any better than a Google search based on this criteria.

An expert human can give you the answer you need with the same layman prompt, without errors.

You would first have to find the expert however, which might not be trivial. Anyway, I think there is value in the space between a basic google search and a human expert. If you don't think so that is fine.

Can you do a quick experiment and ask ChatGPT to craft a prompt to answer “laymen question “ as a financial expert? Having ChatGPT craft its own prompts is usually pretty successful for me.

That probably would work most of the time, but that's also an example of the phenomenon that TFA is talking about: you can't safely just use these tools without becoming an expert at least in the tool. The way they're currently being sold as totally accessible to everyone is dangerous.

> without becoming an expert at least in the tool

Yeah, we're basically repeating the "search engine/query" problem but slightly differently. Using a search engine the right way always been a skill you needed to learn, and the ones who didn't always got poor results, and many times took those results at face value. Then Google started giving "answers" so if your query is shit, the "answer" most likely is too.

Point is, I don't think this phenomenon is new, it's just way less subtle today with LLMs, at least for people who have expertise in the subjects.

> On two occasions I have been asked, ’Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?’ I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.

Understanding the terminology can help a lot. The key is to start the LLM conversation like this, ask it what terminology is used in a particular problem domain and then frame your actual question based on this.

very interesting - could you share your wife's and yours prompt to provide more concrete context?

I considered it, but given the personal nature of the topic I want to keep the details off the public internet. I realize that makes my observation less specific than it could be, but I think the result should be relatively unsurprising to anyone familiar with the tools—this is just a natural consequence of the way the stats fall out.

Also see the Gell-Mann amnesia effect: [0]

[0] https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect