Study: ChatGPT outperforms physicians in quality, empathetic answers to patients

  -Made to wait 45-60 minutes past their appointment time bombarded with pharmaceutical advertisements
  -Spend another 20 minutes sitting in the examination room staring at pharmaceutical company sponsored ads
  -Nurse takes your history with a bunch of redundant questions they already have the answers to
  -Finally the physician arrives
  -Assesses the patient without laying their hands on them
  -Ignores everything you say
  -Does no tests and cultures no pathogens
  -"take some antibiotics and I'll bill your insurance company $500, see ya"

My experience with most doctors is that they are among the least empathic people I have ever dealt with. I think that using AskDocs actually gives doctors an unrealistic _advantage_ in the study.

It's apparent, at least in the US, that a lot of GPs can be unhelpful but believing that accurate diagnosis via text without providing evidence, or going through some type, of physical examination is feasible by Reddit expert docs, only demonstrates lack of critical thought.

So we have here "ChatGPT performs better than doctors *"but* we sampled on an unpaid internet forum"

The other one thats doing the rounds where noone is verifying the source is "ChatGPT develops lethal chemical compounds *but*" It would be too unethical to verify with an actual chemist if any of these compounds it dreamt up actually do anything".

ChatGPT is showing how bad the economics of journalism are for understanding a topic outside of taking press releases at face value*

Some times this presenting of problem to it means I spend anywhere from 5-10 mins actually writing the points down that describes the requirement - which would result in a working component/module (UI/backend).

We have been trialing GPT4 in my company and unfortunately almost everyone's experience is more on the lines of yours than mine. I know it shouldn't, but honestly it frustrates me a lot when I see people complain that it doesn't work :). It definitely works but it depends on the problem domain and inputs. Often people forget that it has no other context about the problem than just the input you are providing. It pays to be descriptive.

I'll also note that when there was hype around Stable Diffusion, one of the images shared around was that of an astronaut riding a horse. If you actually run Stable Diffusion with its default tuning and ask for that prompt, you will get 6 images, of which 5 of them are outright disasters (horses with 6 legs and going downhill from there), and then the 6th image, the only one which could possibly pass as a decent result, is the one that everyone shared and reshared and hyped. Usually other prompts give even more terrible results where there are 0 passable images without extensive tuning. Stable Diffusion now is acknowledged to actually be crap --despite the hype-- and I supposedly need to try the next best thing, whatever that is. But I find myself facing the same situation with ChatGPT 3.5, and now with ChatGPT 4, despite the fact there is no "next best thing", and I don't even know how they could even possible try to fix the problem of it being just wrong.

If, on the other hand you actually put together a prompt which tells it what you expect, the results are very different.

E.g. I've experimented with "co-writing" specs for small projects with it, and I'll start with a prompt of the type "As a software architect you will read the following spec. If anything is unclear you will ask for clarification. You will also offer suggestions for how to improve. If I've left "TODO" notes in the text you will suggest what to put there." and a lot more steps, but the key element is to 1) tell it what role it should assume - you wouldn't hire someone without telling them what their job is, 2) tell it what you expect in return, and what format you want it in if applicable, 3) if you want it to ask for clarifications, either ask for it and/or tell it to follow a back and forth conversational model instead of dumping a large / full answer on you.

The precise type of prompt you should use will depend greatly on the type of conversation you want to be able to have.

> if it doesn't know well a person it simply makes up things

Asking it for factual information about a subject can be a bit hit/miss depending on the subject. Better to use bing chat, because it will use info from the web to inform the response

> if I ask code examples or scripts, most of the time they are wrong, I need to fix them, they contain obsolete syntax etc...

How wrong? More wrong than having a junior or mid level developer contributing code?

Think about it a different way: you just gained an assistant developer that writes mostly correct code in seconds. Big time saver.

Also: if you want it to use a particular code style etc, give it few shot examples.

> if I'm asking a question, I'm expecting being asked for more context if the subject is not clear

Then you need to tell it that in you prompt: "if the subject isn't clear, ask me some clarifying questions. Don't respond with your answer until I have answered your clarifying questions first". Or: "ask me 3 clarifying questions before answering" to force it to "consider" how well it "knows" the subject first.

ChatGPT isn't an AI in the sci fi sense of the word. It's a language model that needs to be prompted the right way to get the results you want. You will get a feel for that the more you use it.

It seems like you could maybe automate that. Let it spit out its first draft of an answer, have the framework tell it "please correct the errors" and then let it have another go and only present the second attempt to the user.

I find it very helpful to ask a series of questions and see a number of examples to get a primer on what to expect with something. The main benefit over Google or going straight to the docs is I can start with my specific requirements. I then dig into the documentation to deepen my understanding. I can typically move forward with ChatGPT generating some code as a starting point.

It can be incorrect or out of date but combined with my experience I find myself being more productive with it.

A weakness I see is complex code requirements. It knows what it knows.

I note that you seem a little frustrated with vague or incorrect responses. It helps to tell ChatGPT the role it should play. It helps as well to instruct it to ask questions of you to improve the response. Personally I prefer to tell it keep its answers brief, I get less walls of text and I can narrow in on the specific answer I am after more quickly.

After reading the article, what you wrote doesn't seem to make much sense.

- write a contract for the sale of my motorcycle: put all details, names and numbers with labels on a spreadsheet, paste on the chat and ask for a contract, then edit.

- learn french: I told gpt "when I write wrong stuff in french, always let me know and teach me the correct ways". Then, after a few weeks I asked for a .csv with the stuff that he corrected so I could import into Anki, which actually worked.

- coding on a daily basis: I am learning Rust on my new job, so I ask it things all the time, it helps me a lot.

If you ask it for boilerplate or for something that's a basic combination of things its seen before, it can give you something decent, possibly even useable as-is. But as soon as you step into more novel territory, forget it.

There was one case where I wanted it to add an async method to an interface as a way of seeing if it "understood" the limitations of covariant type parameters in C# with regards to Task<T>. It did not. I replied explaining the issue and it actually did come back with a solution, but it wasn't a good solution. I told it very specifically that I wanted it to instead create a second interface for holding the async method. It did that but made the original mistake despite my message about covariance still being within the context fed back in for generating this response. I corrected it again, but the output from that ended up being so stupid I stopped trying.

And at no point was it actually doing something that's very important when given tasks that are not precisely specified: ask me questions back. This seems equally likely to be a problem for one of these language models replacing a doctor. It doesn't request more context to better answer questions so the only way to know it needs more is if you already know enough to be able to recognize that the output doesn't make sense. It basically ends up working like a search engine that can't actually give you sources.

E.g., lots of HN users claim to use it for dev or learning new programming languages. Given the frequency of hallucination and their Dunning-Kruger complexes in full-effect, they don't know when it's teaching bad information or functions that don't exist.

It's an LLM. Not an AGI.

Combined with the prestige of the profession, many turn into egoistic know-it-alls whose real competence is equivalent to a car mechanic who tells you you're just driving your car wrong when you come in with anything that's not obvious from a 15s inspection because they get paid anyway.

I would be surprised if most of the doctors I've seen could outperform even GPT-2.

There are good doctors out there but there are a lot of bad. I always advocate people if they're unsure to get a second opinion. Just like you would if a plumber said "you need to replace the whole system." If it doesn't seem right or you don't feel like you got the proper attention, go somewhere else and see.

Medicine isn't an exact science for much of us. It's a lucky thing when it's a simple infection that antibiotics can cure. Most of our problems aren't so easy. Just be slightly skeptical. Don't go "Fruit will cure my pancreatic cancer" crazy either.

https://xkcd.com/937/ comes to mind. It's not implausible at all to me that ChatGPT could outperform Reddit in detail/manners for health advice (and honestly, even for actual doctors I've heard some horror stories about bedside manner and refusing to actually believe/consider symptoms), but if the study isn't actually checking that, if they're just checking if the chatbot was more polite/empathetic... that's a huge qualification that should be up-front and center.

It could also be that knowing the correct diagnosis changes your description to highlight things in a way that suggests the correct outcome. I believe doctors are also susceptible to that effect.

I'm not trying to imply that doctors should not do a better job diagnosing. People should not have to "learn" to find the right doctor, or how to operate them. It's a crying shame that people like you have to go on years-long journeys to get correct help, and I'm sorry that you and your wife had to go through that.

Just saying it might be slightly more apples-to-apples to compare ChatGPT's performance to the last couple of physicians you saw, and not the whole lot of them. But again, that's still a very favorable comparison for the non-human.

Medicine, and in particular diagnosis, is particularly difficult due to the width of knowledge required due to the span of possible diseases.

It completely makes sense that GPT, or similar, would simply be better than doctors at diagnosis in time, and it is very plausible that the time is now.

This is fantastic news for humanity. Humanity gets better diagnosis and we don't put high IQ people thought a grinder that is medical school and residency to do a job which is not suited to human cognition.

It's an amazing win.

Pride goeth before the fall. I wonder how many arrogant jerks will be humbled to see that they too are now inferior to a computer ("soon"). Humans will always be better at being human though, perhaps they will learn empathy is more important than they thought.

If I ask GPT4 about some arcane math concept it’ll wax lyrical about how it has connections to 20 other areas of math. But it fails at simple arithmetic.

Contact me at Drgpt@altmails.com

I recall in a chat a person getting rather exasperated with a coworker and the person using ChatGPT to generate a friendly/business professional "I don't have time for this right now."

GPT could be used in a similar manner - "Here is the information that needs to be sent to the patient. Generate an email describing the following course of treatment: 1. ... Stress that the prescription needs to be taken twice a day."

The response will likely be more personal than the clinical (we even use that word as an adjective) response that a doctor is likely to give.

That guy would have failed against ChatGPT, and I loved the way he told things. Anythong else would have just driven me crazy, maybe to the point of looking for a different doctor.

So I giess, what passes as good bed side manners for doctors largely depends on the patient. By the way, the dentist I have since is in the same category as my, luckily former, oncologist. A visit with hom usually takes no more than 5 minutes if he's chatty, less if not. Up to 10 when treatment is required, anuthing longer than thaf is a different appointment.

Nope, it's a sad reflection of the study construction.

Physicians' empathy was evaluated by their 52 word responses on Reddit. Unsurprisingly, a chatbot optimised for politeness and waffle outperformed responses of people volunteering answers in a different format optimised for brevity...