Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? | Hacker News Reader

Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?

arxiv.org

36 pointsJimmc41417 comments

Does anyone have real life experience (preferably verified in production environment) of fine-tuning actually adding new knowledge to the existing LLM in a reliable and consistent manner? I've seen claims that fine-tuning only adapt the "forms" but can't adding new knowledge, while some claim otherwise. I couldn't convince myself either way with my limited adhoc/anecdotal experiments.

I’ve taught LLMs imaginary words and their meanings with minute amounts of data (two or three examples) via full fine-tuning, LoRA and QLoRA.

I have no idea where the myth of ‘can’t add new knowledge via fine-tuning’ came from. It’s a sticky meme that makes no sense.

Pretraining obviously adds knowledge to a model. The difference between pretraining and fine-tuning is the number of tokens and learning rate. That’s it.

It seems like few shot prompting and providing some examples to LLMs with large context windows vastly out performs any amount of rag, or fine tuning.

Aren't rag and fine tuning fundamentally flawed, because they only play at the surface of the model? Like sprinkles on the top of the cake, expecting them to completely change the flavor. I know LoRA is supposed to appropriately weight the data, but the results say that's not the solution.

Also anecdotal, but way less work!

Long context windows get confused, so shorter is better, and they cannot fit everything in general. I'm not sure where you are seeing results that say otherwise.

RAG is effectively prompt context optimization, so categorically rejecting doing that doesn't make sense to me. Maybe if models internalized that or scaled... But they don't.

Totally agree. Every decision on what context to put in a context window is “RAG”. Somehow the term was co-opted to refer to “context selected by vector similarity”, so presumably when people say “is RAG hanging around”, what they mean is “are vectors a complete solution”, to which the answer is obviously “no”. But you still need some sort of _relevance function_ to pick your context - even if it’s pin-the-tail-on-the-donkey. That’s “RAG”.

Doesn’t make sense to ask “will we still have to curate our context?” The answer is of course you will.

RAG and fine-tuning are very different. Few-shot prompting and RAG are both variants of in-context learning.

That's definitely my experience as well, sufficiently large context window with a capable enough general purpose LLM solves lots if not all of the problems rag/fine tuning claim to solve.

I've also found (anecdotal) significant success in just throwing in available context before prompting. I've written multiple automations in this way as well.

I asked this on Twitter a few weeks ago and didn't manage to dig out any examples: https://twitter.com/simonw/status/1786163920388177950

Afaict gorilla, as in that thread ;-)

Nexusflow probably too, as it also does function calling and would need to bake in, or explicit fine-tuning for RAG use, which I don't recall seeing

I haven't look recently, but there is also a cool category of models that provide GIS inferencing via LLM

This blog post I saw recently might be relevant: https://refact.ai/blog/2024/fine-tuning-on-htmlx-making-web-...

Yeah... So looks like at least it's still an open question. I guess until we can definitively know how "knowledge" is collectively represented among the weights, it's hard to say either way. The other part of the question is how to evaluate the existence of "knowledge" in an LLM. TFA suggests a way, but still not 100% convinced that's THE way...

TFA says you can teach it new facts, but it's very slow and makes the model hallucinate more.

A new dark age incomming

Ice age

Not really answering your question, but all the "alignment" of the big models is done through a combination of supervised fine tuning and RLHF. So all the chat and censorship and other specific behaviors are at least in part fine tuned in. Maybe that is closer to forms rather than actually knowing more...

The term "hallucinations" incorrectly anthropomorphizes LLMs and falsely assumes that there is a difference between "correct" and "incorrect" output. It's all the same to the model. LLMs are probalistic and have no concept of facts or truth.

Yes, the output can contain facts, but that's a function of the training data and prompt, not something you can ever guarantee.

However, if you're looking for job security, convincing management that "it just needs a bit more fine-tuning" will guarantee a never-ending stream of work. :-/

Reply on news.ycombinator.com