I've used https://github.com/PromtEngineer/localGPT for this and thought it was nice. So I packaged it in a docker container for easy use.

docker run -itd --gpus all -p $(PORT):5111 --name llm-local-wizardlm-7b obald/llm-launcher:0.0.2

just use localhost:port in the browser and upload docs then ask questions in the gui.

Really nice for easy lookup of rules in boardgames and such. As it provides the relevant text from the docs in addition to the query answer.

https://gitlab.com/PeterHedman/llm-local

I explored privateGPT today and gave it a source of documents in txt format. It worked well for some queries but for some it gave responses which were not part of the source documents. Any tips on fixing this? There is an open issue for this problem - https://github.com/imartinez/privateGPT/issues/517