ChatGPT 4 saved my dog’s life
twitter.com
twitter.com
Of course, you could argue this is lulling you in a false sense of security. But the same thing (arguably worse) happens when you go to a real doctor! Half the time they barely look at you and just kind of shrug off concerns/anxiety.
Edit: I got curious and ran an experiment. Typically when you are anxiously googling you are worried about the worst case (even if it's not rational).
Google: "headache brain tumor"
First result, before links even, is a huge out-of-context info box from the Mayo Clinic on brain tumors that highlights the words "Or a brain tumor can cause swelling in the brain that increases pressure in the head and leads to a headache." Jesus Christ!
ChatGPT 3.5: "I have a headache, could it be a brain tumor?"
First two sentences: "A headache can have many different causes, and while a brain tumor is one possible cause, it is not the most common cause. Most headaches are not caused by brain tumors and are usually due to other factors such as tension, sinus issues, or migraine." It then goes on to list tumor-specific headaches symptoms (like changes in vision or hearing) and calls out to see a doctor if you're getting those.
Which do you think is more likely to calm you down? Or is more legitimately helpful and going to provide good outcomes?
Clearly you've never been debilitated by health anxiety.
Every story like this comes with ten more about someone obsessively ruminating about symptoms that turn out to be nothing, not least specifically because of horror stories like yours.
There is no one right answer for all situations. But as someone who struggles with health anxiety, it really fucking sucks and yes, there are damn good reasons to seek relief from that sense of constant and unrelenting dread.
> Here's the reassuring truth: Headache, by itself, is rarely caused by a tumor. According to a neurosurgeon at Johns Hopkins' Comprehensive Brain Tumor Center, the chance that your headache is a sign of a brain tumor is very remote.
After reinforcement learning on the other hand it copied the human fallacy: it's very sure when it's very sure, but after that it's stuck at "yeah, it could be true".
An essay on this topic: "WebMD, And The Tragedy Of Legible Expertise" https://astralcodexten.substack.com/p/webmd-and-the-tragedy-...
Went to a Vet. After the treatment didn't work, asked AI.
Considered AI's opinion.
Went to another Vet for Opinion.
Got Results.
Problem arises when People will blindly follow the diagnosis by GPT.
Got me thinking.
The first Vet might not have considered the issue due to a blind spot in judgement.
An interesting use of GPT for me will be to give it a scenario and try to come up with new ways of looking at the problem.
A way to detect blindspots in my judgement.
If we continued staying with the first vet without getting a second opinion, she definitely would have died. On the other hand, if the first vet had access to specialized AI medical diagnostics tools, she wouldn't have died. I think that's the key takeway here.
I think it's also worth pointing out this was less about GPT placing a certain diagnosis, and more about it swaying me in the right direction (i.e. away from listening to bad doctor's advice, and towards looking for a second/opinion trying something else). When the first vet said "welp, don't know what else to do, we gave her the treatment and there's nothing else it can possibly be, so we're just going to monitor her and see how things go", I stayed up all night asking GP4 various things, and no matter which way I sliced it, it was obvious something didn't add up
Historically (1800s), doctors did not feel the need to wash their hands between patients, leading to poor outcomes and death. It had to be mandated, leading to improved outcomes. Same thing.
https://www.nationalgeographic.com/history/article/handwashi...
Some are bad, some burn out, some are over worked, just like all professions.
In this case, the problem would have been blindly following the doctor's advice – not ChatGPT's. The medical professional was wrong. Lethally wrong. If there is a lesson here about the trustworthiness of AI, then there is also a lesson about the trustworthiness of the average doctor.
GPT-4 is better than the average human professional today. This isn't some hypothetical future we are talking about.
Not better at being a useful doctor able to intake a patient and get to the bottom of an unspecified disease. Both doctors and AI are generally poor at doctoring. Only a small fraction of doctors are good at being doctors. No LLM is good at being a doctor.
Someone is going to blindly follow GPT-4's medical advice and meet their end.
At some point you can't help people who lack common sense.
1. https://www.pharmacypracticenews.com/Covid-19/Article/03-20/...
Do not compare GPT-4 or other AI products against the ideal. Compare it to the base rate.
There's some scope for improvement there.
One anecdote is not even close for this to be reliable for life and death advice, especially with the high risk of hallucination and sophistry. This is survivorship bias at its finest.
The pedestals that our society has put professionals on are going to start tumbling very soon now.
And surely one anecdote makes a black box AI very trustworthy and reliable against human doctors. /s
> The pedestals that our society has put professionals on are going to start tumbling very soon now.
Nope. Not even close. The OP did not treat the patient by themselves, they reviewed the output with a human doctor.
I don't think even you yourself would begin to follow ChatGPT's advice and treat and inject yourself without input from an experienced human doctor based on ONE statistic of it saving someone's life especially them going to a second human doctor.
It’s only been available in beta to a significant number of people for about 18 months, hasn’t it?
It's a very good thing. Especially regarding GPT-4. Better than people fully trusting it and following its advice.
It's the core of techbroism.
LLMs have proven to be an excellent tool for data recall in this information age - but tech worshippers alwant to sell them as decision makers as well and that will cause more evil than even what Zuckerberg and co. accidentally brought to humanity.
> Someone is going to blindly follow medical reference book’s medical advice and meet their end.
> It's incredible, but I think we're going to witness the first person who gets killed by internet in about 3~6 months.
> Someone is going to blindly follow medical internet’s medical advice and meet their end.
Maybe we need to reconsider our approach to this and AI can certainly play a vital role in improving the situation.
Our life expectancy may even rise, because it's much more accessible.
Every now and then there is an article about scientists in some field rediscover a principle or technique that was known over a hundred years ago already but didn't have any practical applications so was forgotten again, but could then solve some problem we were having today for the past two decades. Because nobody remembered that obscure paper by some dude who didn't achieve anything remarkable during his lifetime. So we just spent 20 years reinventing the wheel.
This is the kind of task I expect AI to excel at. It might at least currently be on the cognitive level of an overconfident 6yo in many regards, but a 6yo with more knowledge than any single person alive. You will still need humans for a good while to sanity check the output and for the bigger picture though.
Did you read the story? We need AIs to "sanity check" human output more than the other way round.
And while this is an anecdote, it is by no means an isolated incident. Human professionals are being outperformed on trivial tasks (note that this was a common complication of a common disease) by chatbots. I'm far more scared by human incompetence than by AI making mistakes.
On the contrary; we need AIs to "completeness check" the experts, because experts are prone to forget things or have missing knowledge; we need humans to "sanity check" the AIs because current LLMs are insane: mixed in with brilliant pieces of insight are super-obvious mistakes and outright hallucinations.
Already with the current generation of AIs, and their strange mistakes and so-called hallucinations, I'd be hard pressed to say that they are overall worse than human professionals, considering what I've seen from those.
I specifically started my comment with "Maybe not in this specific case", but reading your other comments here you unfortunately seem to be mostly focused on ranting about medical professionals.
Also a short reminder of the HN guidelines, especially
Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
I'm not sure I agree entirely with this - It's very rare to see 6 year olds to be capable of writing valid code, summarising academic literature, or doing diagnostics on a dogs bloodwork.
Even if I gave a 6 year old access to all the information and literature, they just wouldn't really be able to do these things. Maybe if we trained the 6 year old for 10 years then they could do these things, but then they aren't 6 anymore they are 16.
At age 6 the developmental milestone is really "Can count to and understand the concept of "10."
I tested chatgpt a while ago by giving it some real world bugs I encountered in code in the past. It was really hit or miss, sometimes it confidently reasoned in hilariously nonsensical directions, other times it almost nailed it on first try. What's even more interesting, and what I don't see mentioned that often when people show impressive results, is that often times if you just regenerate an answer with the exact same input, you get a completely dumb and wrong reply the next time, or vice versa. So consistency is another problem.
Because GPT4 would destroy a 6 year old at a verbal reasoning, quantative reasoning or an abstract reasoning test.
In the worst case they prescribed that for a brain tumour. So yeah I'll take google and ChatGPT and upload my MRIs there before I trust some random person that I can't even tell if they are good at their job.
ChatGPT at least doesn't speak matter of factly and peer pressure me into doing whatever thing takes less time to get me out the door and the next person in.
The bar of competence set by most doctors is so hilariously low that any smart person with a search engine has been able to outdo the typical GP for a while now. All the accumulated knowledge these professionals have is worthless if they don't take more than 3 minutes to look at the patient.
Imagine if, as an engineering consultant, you took just 3 minutes in total for your customer, including the customer describing their problem. They would literally be better off just googling it instead. There's no reason to expect this to be different for doctors, and indeed it isn't.
So, as a HN user, how many times did that not happen to you?
That people (are forced to) accept this kind of behavior from medical professionals is nothing short of insane.
I mean, have you seriously never had a "knowing where to put the chalk mark" moment, neither in the medical nor in the engineering fields, neither as a customer nor as the professional, aside that it is not a great presentation to precede a fat invoice?
The images have their signal boosted by AI, and resolution doubled by AI. I described the image findings to GPT and asked for a differential.
The radiologist thought the results reasonable.
It was AI all the way down.
It's a known stereotype of Dutch GP's among expats in The Netherlands: no matter what limb(s) are missing, no matter how long you've been bleeding: the first time you go you go see the GP you'll get "take paracetamol and let me know if you still have issues in 2 weeks". lol
Enhance not replace! The map is not the territory, so you’ll always want to pair a native with the expert (and the territory here is reality).
The problem is that our current workflows do not allow medical professionals to offload responsibility.
I think the best scenario is to include ML models in the diagnosis workflow instead of making humans obsolete.
…
Fixed Dog
I think it will be a long while before that ellipsis is, well nothing.
I'm guessing about 1000x faster than using ChatGPT.
I think the real story here is how vulnerable many are to hype, and maybe even more so, how many are willing to ignore the obvious to cash in on the attention.
We saw it with crypto, now it's AI's turn. Honestly I can't tell which had more annoying grifters.
Sure, if you know exactly what you want to google for, you can google it. But that requires you to interpret all the information and piece together a theory, whereas with GPT I just stated all the facts and received accurate (in this case) info back
I have seen some good use cases for GPT but this one in particular (the Tweet) is not a good example.
1) She was already correctly diagnosed with the first issue (babesiosis), which causes anemia.
2) As a result of the babesiosis, she developed a secondary complication (immune disorder), which worsened the anemia. Note that this can also occur as a standalone disease, which is actually the google result you're getting.
3) You would have had to google "secondary complications causing anemia as a result of canine babesiosis", and at this point, google stops helping you. Not that I would have known to google that anyway.
https://vcahospitals.com/know-your-pet/anemia-in-dogs
Right there on the page: IMHA. If you add just a couple keywords from the large input, you likewise get more direct IMHA results.
As a guy who knows nothing about dogs, Chatgpt seemed to zero in on this 10x better than that very long VCA page.
Also this ignores my original comment where I just typed the keywords from the vet notes and got IMHA right on the first result.
That'd be about 100x faster using Google.
Here [1] it’s actually the author who rules that out himself based on prior knowledge and the fact that the situation didn’t get better after the first diagnosis and subsequent treatment. Looking for a secondary cause/explanation is what drives him to ask the question in the first place. GPT says here are some “general information on…”.
[1] https://twitter.com/peakcooper/status/1639716836911489025?s=...
The first Google hit goes to a page that lists IMHA right away.
I am sure LLMs already are helpful to do medical research. But this case seems not to be a great example to show their superiority to old fashioned googling.
Are we gonna celebrate the millions of lives saved by books next?
This is still very impressive for a computer program, but not as mind-blowing as I first thought when reading the thread. ChatGPT didn't find some obscure disease like in a medical TV show. Rather, it correctly read the low blood cell count, and pulled up the differentials for anemia from a reference book.
On a side note, considering how often ChatGPT will lie with full confidence, personally I can't imagine using it for anything medically related.
When benchmarking AI vs. humans, it's important to take into account how garbage humans can be.
Such a strong girl. Go Sassy!
> badgering their doctors/vets
You're saying it like being aware of possible diseases is a bad thing. One just has to avoid saying "you're wrong" to a doctor and instead just ask for a second opinion.
Efficiency and accessibility.
A bit like the image generation AIs that are trained on tons of ugly pictures, understand the common concepts, and only output beautiful pictures thanks to the fewer beautiful pictures from the training datasets.
I'm more or less an optimist. My default attitude is that things will be fine even if they aren't perfect just right now. I enjoy utopian sci fi. I know intellectually it is a utopia and not realistic. But I still like to imagine how great humanity could be if we got rid of that before mentioned attitude. Chat GPT is the most concrete thing in my life time that gets us close to how people interacted with such classic AI characters as c3po, twiki (Buck rogers), kitt in Knight Rider, or the nameless "computer" in Star Trek. I grew up in the eighties (obviously).
My first computer was a commodore 64, that wasn't quite that smart. All that went from being science fiction to being science fact in the last few months. We now have conversational AIs that we can discuss all sorts of topics with. Like C3PO it jumps to the wrong conclusions some times, can be wrong in very entertaining ways, and is scarily good when things go right. I had some debate with Bing as to what to eat and then debugged some code with chat gpt 3 and it pointed out some mistakes that I made.
Is AI going to replace me in everything I do? No, I don't think so. If only because it is in my interest to find ways to keep myself busy. But I sure am going to be using it a lot to do what I do a little bit faster. Which means I get to do more interesting things. Sounds good to me. I like doing interesting things.
GPT 4 apparently has the ability to use tools. Where GPT-3 struggles to do math, GPT-4 can use a calculator and learn to use other tools. This is going to be very disruptive for me. Because learning how to use all sorts of weird and obscure tools is a big part of what I need to do. Often what I do conceptually (I'm a startup CTO) is pretty easy to grasp. Except I then need to figure out a whole bunch of tools to get the work done. Which is actually somewhat tedious. I hate it when an idea pops in my head and I then have to grind at figuring out stupid tool issues for the next few days/weeks/months to get things done. I often don't have time for this so most of my brain farts don't get very far. I actually have to be very economical about what I pursue even or I won't get anything done at all. For most of the ideas I have I don't even have enough bandwidth to validate if they are even good ideas.
This is frustrating to me. And fundamentally, this frustration is what drives creativity. You get stuck on some problem, grind away at it, and then you find a solution and your brain rewards you with a little endorphin rush. We're not very complicated. That incidentally is also the business model behind social media: AI driven endorphin rushes in the form of an addictive feed of stuff. And that AI is mind numbingly stupid in comparison.
So, I don't see AI as a threat but as a massive enabler for me that I can delegate to, cross check ideas with, ask to provide me with some inspiration, explore new concepts with, etc. GPT-3 is already quite good in a limited way but from what I've read about GPT-4, I've seen nothing yet. And I'm sure we'll have GPT 5,6,7 and so on leap frog what is possible in a relatively short time frame. Not to mention the countless other companies that are working on competing AIs.
So, yes, AI is going to change lots of things and I think that's great. Can't wait. Can I fast forward ten years or so?