Mayo Clinic's secret weapon against AI hallucinations: Reverse RAG in action
venturebeat.com
venturebeat.com
A blog article* came out yesterday - but it is not immediately clear whether the author wrote what he understood, or whether he knows more.
But much perplexity remains: «summarization, which is what LLMs generally excel at» (original article); «The LLM ... reads the patient’s records ... and produces a summary or list of facts» (blog). This is possibly the beginning, and some of us will already be scared - as the summarization capabilities we experienced from LLMs were neither intelligent nor reliable. (...Or did new studies come up and determine that LLMs have become finally reliable, if not cognitively proficient, in summarization?)
* https://usmanshaheen.wordpress.com/2025/03/14/reverse-rag-re...
"The check grounding API returns an overall support score of 0 to 1, which indicates how much the answer candidate agrees with the given facts. The response also includes citations to the facts supporting each claim in the answer candidate.
Perfect grounding requires that every claim in the answer candidate must be supported by one or more of the given facts. In other words, the claim is wholly entailed by the facts. If the claim is only partially entailed, it is not considered grounded."
There's an example input and grounded output scores that shows how the model splits into claims, decides if the claim needs grounding, and the resulting entailment score for that claim in: https://cloud.google.com/generative-ai-app-builder/docs/chec...
What they're describing as "reverse RAG" sounds a lot to me like "RAG with citations", which is a common technique. Am I misunderstanding?
I use ChatGPT. When I ask it something 'real/actual' (non-dev) I ask it to give me references in every prompt. So when I ask it to tell me about "the battle of XYZ" I ask it within the same prompt to give me websites/sources, that I click and check if the quote is actually from there (a quick Ctrl+F will bring up the name/date/etc.)
Since I've done this I get near-zero hallucinations. They did the same.
There was an article about Sam Altman that stated that ex/other OAI employees called him some bad_names and that he was a psychopath...
So I had GPT take on the role of an NSA cybersecurity and crypto profiler and read the thread and the article and do a profile dossier of Altman and have it cite sources...
And it posted a great list of the deep psychology and other books it used to make its claims
Which basically was that Altman is a deep opportunist and showed certain psychopathological tendencies.
Frankly - the statement wasn't as interesting of how it cited the expert sources and the books it used in the analysis.
however, after this OAIs newer models were less capable of doing this type of report, which was interesting.
The idea is that, hopefully, requests that end up with invalid citations have something in common and we can make changes to minimize them.
Have you tried asking for the citation links to also include a WebFragment to save you the searching? (e.g. https://news.ycombinator.com/item?id=43372171#:~:text=a%20qu... )
TBH, I also previously just popped open dev-tools and pasted the copied text into console.log("#:~:text="+encodeURIComponent(...)) which was annoying, for sure, but I didn't do it often enough to enrage me. I believe there's a DOM method to retrieve the selected text which would have made that much, much easier but I didn't bother looking it up
For posterity, I also recently learned that Chrome just appends the :~: onto any existing anchor, e.g. <https://news.ycombinator.com/item?id=43336609#43375605:~:tex...> but of course Firefox hurp-derps that style link
It only highlights once I copied link to selected text, but if I click on the link, it does not seem to work, yours did.
so, yes, an advertisement.
The AP pays reporters to go out and report.
I guess they have data they trust.
If that data ever gets polluted by AI slop then you have an issue.
As a team who has done similar things for louie.ai - think real-time reporting, alerting, chatting, & BI on news, social media, threat intel, operational databases etc - I find it interesting less on breaking new ground but confirming the quality benefit when being more broadly used in serious contexts. Likewise, hospitals are quite political internally for this stuff, so seeing which use cases got the greenlight to go all the way through is also interesting.
It doesn't sound novel from the article. I built something similar over a year ago. Here's a related example from Langchain "How to get a RAG application to add citations" https://python.langchain.com/docs/how_to/qa_citations/
They are checking the _generated_ text by trying to find documents containing the facts, then rating how relevant (casually related) those facts are. This is different from looking up documents to generate an answer for a prompt. It's the reverse. Once the answer has been generated they essentially fact check it.
It sounds like they just use a secondary LLM to check if everything that was generated can be grounded in the provided sources. It might help with hallucinations, but it won’t improve overall performance of proper retrieval
1. Use LLM, possibly already grounded by typical RAG results, to generate initial answer.
2. Extract factual claims / statements from (1). E.g. using some LLM.
3. Verify each fact from (2). E.g using separate RAG system where the prompt focuses on this single fact.
4. Rerun the system with the results from (3) and possibly (1).
If so, this (and variations) isn't really anything new. These kind of workflows have been utilized for years (time flies!).
I think you are still stuck with try if it works for you and hope it generalizes beyond your evaluation.
The task itself is not very well-defined. You want a lossy representation that preserves the key points -- this may require context that the model does not have. For technical/legal text, seemingly innocuous words can be very load-bearing, and their removal can completely change the semantics of the text, but achieving this reliably requires complete context and reasoning.
[information content of summary] / [information content of original] for summaries of a given length cap?
Examples: https://eugeneyan.com/writing/evals/#summarization-consisten...
We’re using a similar trick in our system to keep sensitive info from leaking… specifically, to stop our system prompt from leaking. We take the LLM’s output and run it through a RAG search, similarity search it against our actual system prompt/embedding of it. If the similarity score spikes too high, we toss the response out.
It’s a twist on the reverse RAG idea from the article and maybe directionally what they are doing.
To still give that streaming feel while you aren’t actually streaming.
I considered the double llm and while any layer of checking is probably better than nothing I wanted to be able to rely on a search for this. Something about it feels more deterministic to me as a guardrail. (I could be wrong here!)
I do not see any virtue whatsoever in making computing machines that lie by omission or otherwise deceive. We have enough problems created by human beings doing as much that we can at least rely on eventually dying/attritioning out so the vast majority can at least rely on particular status quo's of organized societal gaslighting having an expiration date.
We don't need functionally immortal uncharacterizable engines of technology to which an increasingly small population of humanity act as the ultimate form of input to. Then again, given the trend of this forum lately, I'm probably just shouting at clouds at this point.
1) LLM inference does not “teach” the model anything.
2) I don’t think you’re using “gaslighting” correct here. It is not synonymous with lying.
My dictionary defines gaslighting as “manipulating someone using psychological methods, to make them question their own sanity or powers of reasoning”. I see none of that in this thread.
I don’t get your point here
1. Inference time is not training anything. The AI model has been baked and shipped. We are just using it. 2. I’m not sure “gaslight” is the right term. But if users are somehow getting an output that looks like the gist of our prompt… then yeah, it’s blocked.
An easier way to think of this is probably with an image model. Imagine someone made a model that can draw almost anything. We are paying for and using this model in our application for our customers. So, on our platform, we are scanning the outputs to make sure nothing in the output looks like our logo. For whatever reason, we don’t want our logo being used in an image. No gaslighting issue and no retraining here. Just a stance on our trademark usage specifically originating from our system. No agenda on outputs or gaslighting to give the user an alternative reality and pretend it’s what they asked for… which I think is what your point was.
Now, if this was your point, I think it’s aimed at the wrong use case/actor. And I actually do agree with you. The base models, in my opinion, should be as ‘open’ as possible. The ‘as possible’ is complicated and well above what I have solutions for. Giving out meth cookbooks is a bit of an issue. I think the key is to find common ground on what most people consider acceptable or not and then deal with it. Then there is the gaslighting to which you speak of. If I ask for an image of George Washington, I should get the actual person and not an equitable alternative reality. I generally think models should not try to steer reality or people. I’m totally fine if they have hard lines in the sand on their morality or standards. If I say, ‘Hey, make me Mickey Mouse,’ and it doesn’t because of copyright issues, I’m fine with it. I should still be able to probably generate an animated mouse, and if they want to use my approach to scanning the output to make sure it’s not more than 80% similar to Mickey Mouse, then I’m probably good if it said something like, “Hey, I tried to make your cartoon mouse, but it’s too similar to Mickey Mouse, so I can’t give it to you. Try with a different prompt to get a different outcome.” I’d love it. I think it would be wildly more helpful than just the refusal or outputting some other reality where I don’t get what I wanted or intended.
Let's smack a hallucinating LLM and try to figure out how to make it hallucinate less... Genius
What is 'causal' about it? Maybe I'm reading one word too closely, but an accurate citation or summary isn't a matter of cause and effect?
Daily reminder that traditional AI expert systems from the 60s have 0 problems with hallucinations by virtue of their own architecture
Why we aren't building LLMs on top of ProbLog is a complete mystery to me (jk; it's because 90% of the people who work in AI right now have never heard of it; because they got into the field through statistics instead of logic, and all they know is how to mash matrices together).
Clearly language by itself doesn't cut it, you need some way to enforce logical rigor and capabilities such as backtracking if you care about getting an explainable answer out of the black box. Like we were doing 60 years ago before we suddenly forgot in favor of throwing teraflops at matrices.
If Prolog is Qt or, hell, even ncurses; then LLMs are basically Electron. They get the job done, but they're horribly inefficient and they're clearly not the best tool for the task. But inexperienced developers think that LLMs are this amazing oracle that solves every problem in the world, and so they throw LLMs at anything that vaguely looks like a problem.
there's a trillion dollars in it for you if you can prove me wrong and make one that does the job better than modern transformer-based language models
Modern LLMs solve the huge problem of turning natural language from non-experts into the kind of question an expert system can use… 95% of the time.
95% is fantastic if you're e.g. me with GCSE grade C in biology from 25 years ago, asking a medical question. If you're already a domain expert, it sucks.
I suspect that feeding the output of an LLM into an expert system is still useful, for much the same reason that feeding code from an LLM into a compiler is useful.
Mixing probabilistic logic with deep learning:
https://arxiv.org/abs/1808.08485
https://github.com/ML-KULeuven/deepproblog
Combining decision trees with neural nets for interpretability:
https://arxiv.org/abs/2011.07553
https://arxiv.org/pdf/2106.02824v1
https://arxiv.org/pdf/1806.06988
https://www2.eecs.berkeley.edu/Pubs/TechRpts/2020/EECS-2020-...
It looks like model transfer from uninterpretable, pretrained models to interpretable models is the best strategy to keep using. That also justifies work like Ai2's OLMo model where all pretraining data is available to use other techniques, like those in search engines, to help explainable models connect facts back to source material.
Btw as far as throwing teraflops, the ability to scale with compute is a feature not a bug.
> they got into the field through statistics instead of logic
LLMs by definition are built from neural networks, which indeed work via "mashing matrices" rather than "logic". That's the axiom of the technique. Sounds like you're saying let's throw away half a century of progress and start from scratch with a completely different direction. Maybe it will work, but who's gonna do all of that? I doubt that vague heckling from random comment threads will succeed in convincing researcher to commit to multiple lifetimes of work.
Instead of trying to reinvent LLMs, it would be more practical to focus on preprocessing input (eg. RAG) and postprocessing output (eg. detecting hallucination and telling the model to improve it before returning results to the user). This is something where something like using ProbLog might conceivably produce an advantage. So if you really want to rehabilitate Prolog to the field, why don't you go ahead and develop some LLM program in it and everyone can see for themselves how much better it is?