Hacker News Reader

Comment by rafaelero | Hacker News Reader

I tried finetuning the 13b LLaMa model to insert the knowledge from my own documents, but my experiments weren't successful. My conclusion is that you need billions of tokens to make a LLM reason based on your own dataset. And even if they did acquire these reasoning skills, they probably wouldn't beat GPT-4. And we are not even getting into the costs of self-hosting these LLM's. So why bother? Just use an API from these companies with powerful models and tweak it to deal with your own need.

View on HN

https://github.com/khoj-ai/khoj

This is the easiest I found, on here too.

The readme says it’s offline and doesn’t mention any LLMs

Does it not use any LLMs, or does it have its own, or how how does it work?

Thank you

You can set it up so that it downloads Llama for offline use in the UI. You do not have to do it manually.

@rafaelero I'm working on a blogpost (https://colinharman.substack.com/) to demonstrate this fact since I get a lot of tiresome questions like "why don't you just train instead of retrieving"

Do you have any scripts you could share for the training/eval process? Would love to credit you in the post

Did you try a Retrieval Augmented Generation (RAG) approach? If so, what made you give it up in favor of fine-tuning?

This is the approach I am using right now. My intuition for trying to finetune was that for complex questions it would be better if the model could naturally deal with those intrincacies instead of reading documents with concepts that are connected but not very explicitly so. There is also the problem with context window limit; sometimes I have to truncate the relevant documents, limiting the capacity of the model to offer a good answer.

But I am very impressed with GPT-4's abilities in finding the correct answer just by reading those documents, so I think the only problem still enduring is the context window size.

> My intuition for trying to finetune was that for complex questions it would be better if the model could naturally deal with those intrincacies instead of reading documents with concepts that are connected but not very explicitly so.

That was my impression as well, which is why your comment was so interesting to me. Have you found tools/projects for the DAG approach that you'd recommend?

Reply on news.ycombinator.com