Deterministic Quoting: Making LLMs safer for healthcare
mattyyeung.github.io
mattyyeung.github.io
The trick is, healthcare users would complain to no end about determinism. But, these are “below-the-line” user - aka, folks who don’t write checks and the AI is better than them. (I am a pharmacist by training, and plain vanilla GPT4-turbo is better than me).
Don’t really worry about them. The folks who are interested and willing to pay for AI has more practical concerns - like what is my ROI and the implementation like.
Also - folks should be building Baymax from big hero 6 by now (the medical capabilities, not the rocket arm stuff). That’s the next leg up.
But I want this feature and I’ll look for software that has it.
This presumes "chunks" are the source. But it's not easy to identify the propositions that form the source of some knowledge. In the best case, you are looking for an association and find it in a sentence you've semantically parsed, but that's rarely the case, particularly for medical histories.
That said, deterministic accuracy might not matter if you can provide enough context, particularly for further exploration. But that's not really "chunks".
So it's unclear to me that tracing probability clouds back to chunks of text will work better than semantic search.
It's all grey isn't it? Vanilla RAG is a big step along the spectrum from LLM towards search, DQ is perhaps another small step. I'm no expert in search but I've read that those systems coming from the other direction, perhaps they'll meet in the middle.
There are three "lookups" in a system with DQ: (1) The original top-k chunk extraction (in the minimalist implementation, that's unchanged from vanilla RAG, just a vector embeddings match) (2) the LLM call, which takes its pick from 1, and (3) the call-back deterministic lookup after the LLM has written its answer.
(3) is much more bounded, because it's only working with those top-k, at least for today's context constrained systems.
In any case, another way to think of DQ is a "band-aid" that can sit on top of that, essentially a "UX feature", until the underlying systems improve enough.
I also agree about the importance of chunk-size. It has "non-linear" effects on UX.
Imagine having ~10-100 different LLMs, maybe some are medical, maybe some are general, some are from a different language. Have them all run it, rank the answers.
Now I believe this can further be amplified by having another prompt ask to confirm the previous answer. This could get a bit insane computationally with 100 original answers, but I believe the original paper I read was that by doing this prompt processing ~4 times, they got to some 95% accuracy.
So 100 LLMs give an answer, each time we process it 4 times, can we beat a 64 year old doctor?
Even with such a system, which will still have some hallucination rate, adding Deterministic Quoting on top will still help.
It feels to me we are a long way off LLM systems with trivial rates of hallucination
I believe I read doctors are only at like 30%...
I'd thought about experimenting with automatically validating that the quoted text does indeed 100% match the original source, but should even a tweak to punctuation count as a failure there?
The proposed deterministic quoting mechanism feels like a much simpler and more reliable way to achieve the same effect.
This I think is scarier. A healthcare-oriented (or any industry) RAG retrieving a bad, but highly linguistically similar answer.
Which is a much harder problem to solve outside few highly standardized niches/ industries.
I think synthetic data generation as a mean to guide LLMs over a larger than optimal search space is going to be quite interesting.
However, if your models distribution is wrong, you’re basically going to have an even more skewed distribution in models trained using the synthetic data.
To me, it seems like the architecture is the next place for improvements. If you can’t synthesise the entirety of human knowledge using transformers, there’s an issue there.
The smell that points me in that direction is the fact that up until recently, you could quantise models heavily with little drop in performance, but recent Llama3 research shows that’s not the case anymore
RAG + deterministic quoting = search engine that displays real excerpts from pages.
For example:
- Authoritative reference: https://www.montana.edu/rmaher/ee417/Authoritative%20Referen...
- Authoritative source: https://piedmont.libanswers.com/faq/135714
Deepmind published something similar recently for claim validation and hallucination management and got excellent results.
So far, we haven't found extractive QA (or any other technique) to significantly improve overall answer quality when compared to matching sub-string similarity. (I'd be interested to hear if you have different experience!)
There aren't a lot of applications can purely be solved with substrings of source documentation, so having both LLM prose and quotations in the answer provides benefit (eg ability to quote multiple passages). Now, we can modify the constrained generation side of things to allow for these but that gets complicated. Or, it can be done with recursive calls to the LLM, but that again requires some kind of DQ check on top.
Ultimately, both styles seem to perform similarly - and suffer from the same downsides (choosing the wrong quote and occasionally omitting useful quotes).
(Good writeup by the way, I've forwarded it to my team, thanks!)
Except with LLMs, you can run like 10 different models. With a human, you owe $120 and are taking medicine.
Well there are protocols, procedures and a bunch of checks and balances.
The problem with the LLM is that there isn't any, its you vs one shot retrieval.
Step 2: Have your physician dad get you a job at a hospital
Step 3: Have your physician dad's physician friend write a letter of recommendation
Step 4: Get into medical school
Step 5: Have your physician dad reach out to friends at various residencies.
Step 6: Get influenced by big pharma, create addictions, make big money.
Although presumably if a company is making decisions using an LLM, and the LLM makes a mistake, the company would still be held liable ... probably.
If there's no "damage" from the mistake then it doesn't matter either way.
If it hallucinates an incorrect (but valid) reference ID then hopefully your users can spot that the quoted text has no relevance to their question.
(1) if the <title> contents (unique reference string) doesn't match, then it's trivially detected. Typically the query is re-run (non-determinism comes in handy sometimes) or if problems persist we show an error message to the doctor
(2) if a valid <title> is hallucinated, then the wrong quote is indeed displayed on the blue background. It's still a verbatim quote, but it is up to the user to handle this.
In testing when we have maliciously shown the wrong quote, users seem to be easily able to identify. It seems "Irrelevant" is easier than "wrong" to detect.