PrivateGPT
github.com
github.com
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
sentry-sdk 1.22.2 requires urllib3<2.0.0, but you have urllib3 2.0.2 which is incompatible
…many many lines
raise ValueError("%r is not a directory" % (package_path,))
ValueError: 'build/py3k/scipy' is not a directory
Converting to Python3 via 2to3... /tmp/pip-install-hkb_4lh7/scipy_088b20410aca4f0cbcddeac86ac7b7b1/build/py3k/scipy/signal/fir_filter_design.py
[end of output]
note: This error originates from a subprocess, and is
likely not a problem with pip.
error: metadata-generation-failed
Someone is thinking "this is part of learning the language," but I think it's just bad design.
python3 -m venv venv
. venv/bin/activate
Also, `python` is usually Python 2.7. If it is, I advise removing it from your system unless you have a strong reason to keep it.
(base) vid@kk:~/D/ai/tortoise-tts$ python3 -m venv venv
(base) vid@kk:~/D/ai/tortoise-tts$ . venv/bin/activate
(venv) (base) vid@kk:~/D/ai/tortoise-tts$ python -m pip install -r ./requirements.txt
Collecting tqdm
Using cached tqdm-4.65.0-py3-none-any.whl (77 kB)
Collecting rotary_embedding_torch
Using cached rotary_embedding_torch-0.2.3-py3-none-any.whl (4.5 kB)
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [8 lines of output]
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "/tmp/pip-install-i7ubxxkc/scipy_4d5af4f3e2094adca3313ccb41a6d5ff/setup.py", line 196, in <module>
setup_package()
File "/tmp/pip-install-i7ubxxkc/scipy_4d5af4f3e2094adca3313ccb41a6d5ff/setup.py", line 147, in setup_package
from numpy.distutils.core import setup
ModuleNotFoundError: No module named 'numpy'
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
[notice] A new release of pip available: 22.3.1 -> 23.1.2
[notice] To update, run: pip install --upgrade pip
(venv) (base) vid@kk:~/D/ai/tortoise-tts$
This isn't the first time venv didn't work for me, then there's anaconda, miniconda, and a bunch of other things that add env and directories. I don't really know what any of them do, and -I don't want to- I'm not an expert on every app on my system, but I can use nearly all of them without pain. (remember this is about ease of use)
Oh yeah, and python 2 vs python 3. <rolls eyes>
It's very much the "works for me" experience from the old days. There's no good learning from it, except dependencies suck and python systems aren't good at them.
I think when releasing anything that includes dependencies that span the operating system, it's just good engineering to use a container approach. Otherwise you're just causing a lot of discomfort in the world for no good reason.
It's funny because chatgpt would give me an answer to this in a few moments, but I'm locked out for a week because it can't figure out where I am.
Now I'm spending my Sunday morning setting up a dockerfile for tortoise-tts. At least I will learn something reusable from that. I guess I will create a PR for it, though it seems the author isn't tending the repo anymore.
That said, this tortoise-tts project might be a particularly bad example. It somehow locks to scipy 0.10.1 from 2012 [1] (during the Python 3.2 release cycle, when Python 3 was heavily in flux) in requirements.txt [2]. Probably not terribly surprising it doesn't work. I didn't bother to look into why they lock to that.
[1] https://pypi.org/project/scipy/0.10.1/
[2] https://github.com/neonbjb/tortoise-tts/blob/0ea829d37aa6528...
this is exactly why I am learning Nix, to help contain chaotically-designed dependency garbage like this to 1 project directory
Ironically, the poo of things like python multitenant dependency management will likely push Nix adoption forward (and unfortunately also Docker)
This is bad, `rotary_embedding_torch` as a package is not in a high enough quality to put as a requirement.
The good news is Pip 23.1+ is forcing the issue, `rotary_embedding_torch` will fail even if you have `numpy` installed because builds by default take place in an isolated environment and you *must* define any build requirements you have. This should force the quality of packages in the Python ecosystem to improve and no longer have this error.
All that nonsense vs docker pull / docker run
Edit: Not saying offering it as an option is bad. But your pip install should work regardless, scipy=0.10.1 is bad whether you offer a Docker image or not.
Arguably, it's a pretty reasonable requirement. Widely used, mature, easy to set up.
I don't remember when I switched to running all my dev envs in Docker, but I wouldn't go back.
From a linux perspective, I wouldn't blindly suggest the average reader to purge Python 2.7 from their system, as it might drag core parts of the WM with it. Consider aliasing, or better yet, relying on modern venv tools such as Conda instead.
I don’t know about desktop Linux though.
Python is very fragile to deploy and run on your own machine.
It's too bad the ecosystem seems to be so messy, because Python seems like the best language for general utilities.
The comment does not really make sense. It sounds like pyenv wasn't setup correctly on your system (needs to be added to your bash_profile etc). The typical setup is to put pyenv first in PATH so it takes precedence.
It may be easier to use pipx which will auto manage the virtual envs of end user apps for you.
Say what you will about the old school way of manually building and copying shit around, at least when something breaks I don't have to spend a couple hours keelhauling a bloated toolchain in a debugger for mutiny.
You should not be seeing these dependency conflict issues if you install each project in its own virtual environment.
If you just want them to be easily installed you can just use pipx (`pipx install my-package`) which will manage the virtual environment automatically.
Making a full blown Docker image for it is overkill 99% of the time. Virtual environments serve the same purpose while being much faster and lighter weight.
While consumers are happy to get their data mined to avoid paying, businesses are the opposite: willing to pay a lot to avoid feeding data to MSFT/GOOG/META.
They may give assurances on data protection (even here GitHub copilot TOS has sketchy language around saving down derived data), but can’t get around fundamental problem that their products need user interactions to work well.
So it seems with BigTechLLM there’s inherent tension between product competitiveness and data privacy, which makes them incompatible with enterprise.
Biz ideas along these lines: - Help enterprises set up, train, maintain own customized LLMs - Security, compliance, monitoring tools - Help AI startups get compliant with enterprise security - Fine tuning service
It has worked before like in case of Linux and can work again.
Linux is different. It doesn't require huge investments in server farms.
[0]: https://www.semianalysis.com/p/google-we-have-no-moat-and-ne...
The big benefits are AAD auth and the ability to put a proxy (APIM, etc.) on the OpenAI endpoint to do quality control, metering, logging, moderation, etc. all within Azure.
I can see a self-hosted LLM being akin to a company’s ship-mind. Anyone can ask questions, order analyses, etc, so long as you are a member of the company. No two LLM’s will be exactly the same - and that’s ok.
https://fractalverse.net/explore-to-sleep-in-a-sea-of-stars/...
Right now, you can't pay a lot and get a local LLM with similar performance to GPT-4.
Anything you can run on-site isn't really even close in terms of performance.
The ability to finetune to your workplaces terminology and document set is certainly a benefit, but for many usecases that doesn't outweigh the performance difference.
https://lmsys.org/blog/2023-03-30-vicuna/
https://www.semianalysis.com/p/google-we-have-no-moat-and-ne...
In this case it appears to be using RetrievalQA from LangChain, which I think is this prompt here: https://github.com/hwchase17/langchain/blob/v0.0.176/langcha...
Use the following pieces of context to answer the question at the end. If you don't
know the answer, just say that you don't know, don't try to make up an answer.
{context}
Question: {question}
Helpful Answer:The LLM will absolutely lie if it doesn't know and you haven't made it perfectly clear that you'd rather it did not do that.
LLMs seem to be trying to give answers that make you happy. A good lie will make you happy. Unless it understands that you will not be happy with a lie.
Is this anthropomorphizing? Yep. But that's the best way I've found to reason about them.
Generally, you want some external way of verifying that you have something useful. Sometimes that happens naturally. Ask a chatbot to recommend a paper to read and then search for it, and you’ll find out pretty quick if it doesn’t exist.
The common case we see publicized today is people poking around with prompts, but isn't it more likely, or at least a risk, that mass adoption will look more like AI running as longlived processes talked with managing done system on their own?
If by “AI” you mean “bare GPT-style LLMs”, no, they can’t do that.
If you mean “systems consisting of LLMs being called in a loop by software which uses a prompt structure carefully designed and tested for the operating domain, and which has other safeguards on behavior, sure, that’s more probable.
One way to think about it, though, is that many important processes have a non-zero error rate. Particular those involving people. If you can put bounds on the error rate and recover from most errors, maybe you can live with it?
An assumption that error rates will remain stable is often pretty dubious, though.
Think about it: 100 people ask Jeeves who won the space race. They would all get the same results.
100 people ask Google who won the space race. They'll all get the same results, but in different orders.
100 people ask ChatGPT who won the space race. All 100 get a different result.
The LLM itself just emulates the collective opinions of everyone in a bar, so it's not a credible source (and cannot be cited anyway). Any two of these people arguing their respective GPT-sourced opinions at trivia night are going to be forced to go to a more authoritative source to settle the dispute. This is no different than the status quo...
The purpose is to serve as a component of a system which also includes features, such as the prompt structure upthread, that mitigates the undesired behavior while keeping the useful behaviors.
Even if LLMs never get any more reliable than your average human, they're still valuable because they know much more than any single human ever could, run faster, only eat electricity, and can be scaled up without all kinds of nasty social and political problems. That's huge on its own.
Or, put another way, LLMs are kind of an concentrated digital extract of human cognitive capacity, without consciousness or personhood.
Hopefully, for the former.
Be a bit terrifying if it turns out "attention is all you need" for that too.
I assure you, those still exist in AI. AI follows whatever political dogma it is trained on, regardless of if you point out how logically flawed it is.
If it is trained to say 1+1=3, then no matter what proofs you provide, it will not budge.
How LLMs are able to give convincing wrong answers: they “can predict the correct ‘shape’ of an answer” (parent).
Why LLMs are able to give convincing wrong answers is a little more complicated, but basically it’s because the model is tuned by human feedback. The reinforcement learning from human feedback (RLHF) that is used to tune LLM products like ChatGPT is a system based on human ranking. It’s a matter of getting exactly what you ask for.
If you tune a model by having humans rank the outputs, despite your best efforts to instruct the humans to be dispassionate and select which outputs are most convincing/best/most informative, I think what you’ll get is a bias towards answers humans like. Not every human will know every answer, so sometimes they’ll select one that’s wrong but likable. And that’s what’s used to tune the model.
You might be able to improve this with curated training data (maybe something a little more robust than having graders grade each other). I don’t know if it’s entirely fixable though.
The brilliant thing about the parent’s comment about the “shape” of the answer is that it reveals how much humans have (uh, historically, now, I guess) relied on the shape of information to convey its trustworthiness. Expand the notion of “shape” a bit to include the medium. If somebody bothered to take the time to correctly shape an answer, we take that as a sign of trustworthiness, like how you might trust something written in a carefully-typeset book more than this comment.
Surely no one would take the time to write a whole book on a topic they know nothing about. Implies books are trustworthy. Look at all the effort that went in. Proof of effort. When perfectly-shaped answers in exactly the form you expected are presented in a friendly way and commercial context, they certainly read as trustworthy as Campbell’s soup cans. But LLMs can generate books worth of nonsense in exactly the right shapes without effort, so we as readers can no longer use the shape of an answer to hint at its trustworthiness.
So maybe the answer is just to train on books only, because they are the highest quality source of training data. And carefully select and accredit the tuning data, so the model only knows the truth. It’s a data problem, not a model problem
> The brilliant thing about the parent’s comment about the “shape” of the answer is that it reveals how much humans have (uh, historically, now, I guess) relied on the shape of information to convey its trustworthiness.
This is the basis of Rumor. If you tell a story about someone that is entirely false but sounds like something they're already suspected of or known to do, people will generally believe it without verification since the "shape" of the story fits people's expectations of the subject.
To date I've decried the choice of "hallucination" instead of "lies" for false LLM output, but it now seems clear to me that LLMs are a literal rumor mill.
By telling it not to lie to you, you're biasing it toward a particular output in the event that its confidence is low. Otherwise, low confidence results just fall out somewhere mostly random.
This is something I really don't understand about LLMs. I think I understand how the generative side of them work, but "asking" it to not lie baffles me. LLMs require a massive corpus of text to train the model, how much of that text contains tokens that translate to "don't lie to me", and scores well enough to make its way into the output?
My take? It's like a high-schooler being asked a question by the teacher and having to answer on the spot. If they studied the material well, they'll give a good and correct answer. If they (like me, more often than I'd care to admit) only half-listened to the lectures and maaaaybe skimmed some cliff's notes before class, they will give an answer too - one strung together out of few remembered (or misremembered) facts, an overall feel for the problem space (e.g. writing style, historical period, how people behave), with lots and lots of interpolation in between. Delivered confidently, it has more chance of avoiding a bad mark (or even scoring a good one) than flat-out saying, "I don't know".
Add to that some usual mistakes out of carelessness and... whatever it is that makes you forget a minus sign and realize it half a page of equations later - and you get GPT-4. It's giving answers like a person who just blurts out whatever thoughts pop into their head, without making a conscious attempt at shaping or interrogating them.
I think it might be more accurate to say, "LLMs are writing a novel in which a very smart AI answers everyone's questions." If you were writing a sci fi novel with a brilliant AI, and you knew the answer to some question or other, you'd put in the right answer. But if you didn't know, you'd just make up something that sounded plausible.
Alternately, you can think of the problem as the AI taking an exam. If you get an exam question you're a bit fuzzy on, you don't just write "I don't know". You come up with the best answer you can given the scraps of information you do know. Maybe you'll guess right, and in any case you'll get some partial credit.
The first one ("writing a novel") is useful I think in contextualizing emotions expressed by LLMs. If you're writing a novel where some character expresses an emotion, you aren't experiencing that emotion. Nor is the LLM when they express emotions: they're just trying to complete the text -- i.e., write a good novel.
What is taught indirectly though is level of certainty, so if you get LLM's to rationalise their answers you tend to get more reliable evidence based answers.
Bottom line, teaching a monolithic model what it means to not know something with certainty, is difficult and not currently done. You'll likely get a lot of false negatives.
- [GitHub - e-johnstonn/BriefGPT: Locally hosted tool that connects documents to LLMs for summarization and querying, with a simple GUI.](https://github.com/e-johnstonn/BriefGPT)
- [GitHub - go-skynet/LocalAI: Self-hosted, community-driven, local OpenAI-compatible API. Drop-in replacement for OpenAI running LLMs on consumer-grade hardware. No GPU required. LocalAI is a RESTful API to run ggml compatible models: llama.cpp, alpaca.cpp, gpt4all.cpp, rwkv.cpp, whisper.cpp, vicuna, koala, gpt4all-j, cerebras and many others!](https://github.com/go-skynet/LocalAI)
- [GitHub - paulpierre/RasaGPT: RasaGPT is the first headless LLM chatbot platform built on top of Rasa and Langchain. Built w/ Rasa, FastAPI, Langchain, LlamaIndex, SQLModel, pgvector, ngrok, telegram](https://github.com/paulpierre/RasaGPT)
- [GitHub - imartinez/privateGPT: Interact privately with your documents using the power of GPT, 100% privately, no data leaks](https://github.com/imartinez/privateGPT)
- [GitHub - reworkd/AgentGPT: Assemble, configure, and deploy autonomous AI Agents in your browser.](https://github.com/reworkd/AgentGPT)
- [GitHub - deepset-ai/haystack: Haystack is an open source NLP framework to interact with your data using Transformer models and LLMs (GPT-4, ChatGPT and alike). Haystack offers production-ready tools to quickly build complex question answering, semantic search, text generation applications, and more.](https://github.com/deepset-ai/haystack)
- [PocketLLM « ThirdAi](https://www.thirdai.com/pocketllm/)
- [GitHub - imClumsyPanda/langchain-ChatGLM: langchain-ChatGLM, local knowledge based ChatGLM with langchain ｜ 基于本地知识库的 ChatGLM 问答](https://github.com/imClumsyPanda/langchain-ChatGLM)
"ggml_new_tensor_impl: not enough space in the context's memory pool (needed 18296202768, available 18217606000)"
Using embedded DuckDB with persistence: data will be stored in: db [1] 8281 illegal hardware instruction python privateGPT.py
I want to have the memory part of langchain down, vector store + local database + client to chat with an LLM (gpt4all model can be swapped with OpenAI api just switching the base URL)
https://github.com/aldarisbm/memory
It's still got ways to go, if someone wants to help let me know :)
The process roughly is:
Ingestion:
- Process embeddings for your documents (from text to array of numbers)
- Store your documents in a Vector DB
Query time:
- Process embeddings for the query
- Find documents similar to the query using distance from other docs in the Vector db
- Construct prompt with format:
""" Answer question using this context: {DOCUMENTS RETRIEVED}
Question: {question} Answer: """
Is that correct? Now, my question is, can the models be swapped easily? Or that requires a complete recalculation of the embedding (and new ingestion)?
This project could help me create a personal AI which answers any questions to my life, finances or knowledge...
https://www.youtube.com/watch?v=A3F5riM5BNE
Also has a suggestion of a few alternative models to use.
> Name Quant method Bits Size RAM required Use case
WizardLM-7B.GGML.q4_0.bin q4_0 4bit 4.2GB 6GB 4bit.
WizardLM-7B.GGML.q4_1.bin q4_0 4bit 4.63GB 6GB 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models.
WizardLM-7B.GGML.q5_0.bin q5_0 5bit 4.63GB 7GB 5-bit. Higher accuracy, higher resource usage and slower inference.
WizardLM-7B.GGML.q5_1.bin q5_1 5bit 5.0GB 7GB 5-bit. Even higher accuracy, and higher resource usage and slower inference.
WizardLM-7B.GGML.q8_0.bin q8_0 8bit 8GB 10GB 8-bit. Almost indistinguishable from float16. Huge resource use and slow. Not recommended for normal use.
> Name Quant method Bits Size RAM required Use case
wizard-vicuna-13B.ggmlv3.q4_0.bin q4_0 4bit 8.14GB 10.5GB 4-bit.
wizard-vicuna-13B.ggmlv3.q4_1.bin q4_1 4bit 8.95GB 11.0GB 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models.
wizard-vicuna-13B.ggmlv3.q5_0.bin q5_0 5bit 8.95GB 11.0GB 5-bit. Higher accuracy, higher resource usage and slower inference.
wizard-vicuna-13B.ggmlv3.q5_1.bin q5_1 5bit 9.76GB 12.25GB 5-bit. Even higher accuracy, and higher resource usage and slower inference.
wizard-vicuna-13B.ggmlv3.q8_0.bin q5_1 5bit 16GB 18GB 8-bit. Almost indistinguishable from float16. Huge resource use and slow. Not recommended for normal use.
> Name Quant method Bits Size RAM required Use case
VicUnlocked-30B-LoRA.ggmlv3.q4_0.bin q4_0 4bit 20.3GB 23GB 4-bit.
VicUnlocked-30B-LoRA.ggmlv3.q4_1.bin q4_1 5bit 24.4GB 27GB 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models.
VicUnlocked-30B-LoRA.ggmlv3.q5_0.bin q5_0 5bit 22.4GB 25GB 5-bit. Higher accuracy, higher resource usage and slower inference.
VicUnlocked-30B-LoRA.ggmlv3.q5_1.bin q5_1 5bit 24.4GB 27GB 5-bit. Even higher accuracy, and higher resource usage and slower inference.
VicUnlocked-30B-LoRA.ggmlv3.q8_0.bin q8_0 8bit 36.6GB 39GB 8-bit. Almost indistinguishable from float16. Huge resource use and slow. Not recommended for normal use.
Copied of some of The-Bloke's model descriptions on huggingface. With 16G you can run practically all 7B and 13B versions. With shared GPU+CPU inference, one can also offload some layers onto a GPU (not sure if that makes the initial RAM requirement smaller), but you do need CUDA of course.
It looks like you can ask a question and the model will use its combined knowledge of all your documents to figure out the answer. It looks like it isn't fine-tuned or trained on all the documents, is that right? How is each document turned into an embedding, and then how does the model figure out which documents to consult to answer the question?
Projects like this for using with your documents datasets are invaluable, but everything I've tried so far is hallucinating, so not practical. What's the state of the art of the LLM without hallucination at the moment?
I assume that for questions like "What year was Bill Gates born in?", it should never return a wrong answer, if the answer was in the training data. If it was not, it should respond that it doesn't know.
Also, the local model used makes a big difference. Right now wizard-mega and manticore are the best ones to use. I run the 16b ggml versions in an M2 Pro and it takes about 30 seconds to “warm up” and produce some quality responses.
What would be the difference in user experience or information retrieval performance between the two?
My impression is it saves work on the dataset transformation and compute for fine tuning, so it must be less performant. Is there a reason to prefer the strategy here other than ease of setup?
In general, Qdrant is a real DB, not a library and that's a huge difference.
Why? Why can't I define any directory (my existing Obsidian vault, for example) as the source directory?
weekday and time of day have an impact. Thousands of entries are posted each day (see https://news.ycombinator.com/newest) most never get a comment or upvote.
They say that people who need external validation don't have their own values and need to seek approval from others...