This is in line with my own personal experience with LLMs and non-trivial questions. They’re excellent when answering questions on topics you know nothing about, and somehow embarrassingly wrong when you actually know the answer yourself…
It’s not clear to me why we’re still trying to encode all of human knowledge in a single model, instead of teaching the model how to look for answers from an external source (e.g. RAG).
"The benchmark includes questions where at least one LLM confabulated, in order to minimize the number of questions requiring human assessment. Because of this, and since the questions are intentionally adversarial, the absolute percentage should not be used to infer that LLMs frequently confabulate. This leaderboard does not reflect a "typical" hallucination rate."
> instead of teaching the model how to look for answers from an external source (e.g. RAG)
My benchmark specifically focuses on the RAG use case. Even with provided texts, current models still hallucinate.
I disagreed then but it's unfortunately convenient to use this approach for some benchmarks.
This just makes me think LLMs must be wrong far more often than we realize. How well can you judge the quality of a response on a topic you know nothing about?
You cannot, but those pushing Gen AI wave away the probability and liability for obvious reasons. Sell the magic, it'll be someone else's problem when it doesn't deliver.
I stopped playing with larger models and have been pushing smaller models with this improvised system prompt and getting good results. It seems like it forces the model to do multiple passes before giving you any response.
My smaller local models give me less hallucinations than Meta.ai, for example, which generally spits out pleasing answers almost immediately (which are often hallucinations, since I don’t think it is system prompted to be adversarial to the user, or itself). I don’t have the same hallucination issue with Llama3 - 8b locally because of custom system prompts.
The model has all the correct information, so it almost needs to do RAG on itself. Multiple passes on itself seems like a way to do it.
This is probably the truth behind the black magic I’m imagining. You could have it explicitly spit out this process, in which case you would see it’s first rough draft, followed by a “My first paragraph is probably wrong”, followed by a third paragraph where it attempts to fix the first paragraph. There is no outside RAG in this process.
The mumbo jumbo part of all this is that I’ve told it to “hide” this process from the user where it doesn’t explicitly output anything but its final answer, and the accuracy has been just as good (for my use case at least).
:Shrugs:
By asking it to reconsider half its generated response, aren’t I essentially asking it to formulate the second half of its response from the first half internally? I’m bypassing the manual process of feeding in the extra prompt.
We are constantly having to tell the LLM close, but no cigar, iterate again, more or less.
That's how the chain-of-thought approach works. You can make the model do it inline, or you can run the loop yourself, possibly summarising the progress as you go. (Although with prompt caching it's not that important anymore) You can encourage it to print out assumptions/steps/ideas as it goes.
(Disclosure: I have not tried your prompt)
There is no logical reasoning happening, it has no concept of right and wrong, let alone that it can force a specific percentage of wrongness.
You tend to get catered responses to whatever role you assign in the prompt. This is well documented. Here's a quick example from search results
https://www.ssw.com.au/rules/give-chatgpt-a-role/
"You are wrong 80% of the time" could be misconstrued as an expected role/command, rather than a mere observation.
> let alone that it can force a specific percentage of wrongness.
Ah, I see what you're saying here. I agree. Maybe I should have said that given the prompt, I'm surprised it doesn't give intentionally incorrect answers (full stop)
Your prompt might give the model context that gives it better token quality much in the same way that asking “How to swim?” is worse than “I’d like to learn the proper technique for the freestyle swimming stroke, and you’re an expert swimming coach.”
There’s no guarantee your prompt isn’t giving less factual answers to be honest. I wouldn’t go about telling the model that it’s often wrong, as it’s not useful context and might skew results.
We basically want it to enter double-checking mode on it’s own from its initial rough draft (its original response, first half of its response, first paragraph, however you are formatting the output). Otherwise the model will output whatever it outputs, and we will have to manually tell it to reconsider facts, locations, and events.
I agree that there’s no guarantee, but this was a suggestion for those who are getting very wrong answers for simple things.
To be fair: we tried a primordial version of that with venue-weighted citation-based ranking and it worked INCREDIBLY well for 10+ years. Then myopic profit motive poisoned the well. Ever since then we've been searching for solutions.
We do so by allocating resources in a way that primarily leverages a scientific credit assignment system that fetishizes... checks notes... venue-weighted citation-based ranking.
Jokes aside: I remain convinced that the O.G. google search appliance on prop data and then simply ignoring all academics remains the best knowledge retrieval (or whatever) tool available.
I forgot the name of this phenomenon with humans, described it to o1 and it gave the correct answer - Gell-Mann Amnesia Effect [1]
"Briefly stated, the Gell-Mann Amnesia effect is as follows. You open the newspaper to an article on some subject you know well. In Murray's case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward—reversing cause and effect. I call these the "wet streets cause rain" stories. Paper's full of them.
In any case, you read with exasperation or amusement the multiple errors in a story, and then turn the page to national or international affairs, and read as if the rest of the newspaper was somehow more accurate about Palestine than the baloney you just read. You turn the page, and forget what you know."
– Michael Crichton (1942-2008)
Sounds like Gell-Mann Amnesia. Maybe LLMs should replace reporters.
> At least one of the four completions must be incorrect for the trainer to continue with that question; otherwise, the trainer was instructed to create a new question.