Do you need a vector database?

This is generally very context/use case specific. In general, if a document is a `Dict[str, Any]`, then you either have to have one (or multiple) vector(s) per field, unless you want to combine vectors across fields (it's not self-evident how you'd best do that). In saying that, specific reason's to do this (or why I've done it in the past).

1. Chunking long text fields in documents so as to get a better semantic vector for them (also you can only fit so much into an LLM). 2. Differently to 1. chunking long text fields (or even chunking images, audio, etc), is one way to perform highlighting. It helps to answer the question, for example, for a given document what about it was the reason it was returned? You can then point to the area in the image/text/audio that was most relevant. 3. You may want to run different LLMs on different fields (perhaps a separate multi-modal LLM vs a standard text LLM), or like another comment said have different transforms/representations of the same field.

Perhaps 100 vectors is non-standard, but definitely not unseen.

I’m not a data scientist but I think I know why one document could lead to many vectors.

(Happy to be corrected and/or schooled.)

A vector is a list of numbers each of which represents weight accorded to a certain word along a certain dimension.

Let’s take an example.

Is an “apple” a “positive” or a “negative” thing? Most people would associate positivity with apples. So, for the general population, the vector for “apple” along the 0-1 continuum where 0 represents negative sentiment and 1 represents positive sentiment would be something like [0.8].

Let’s add one more dimension. Is an apple associated with computers (1) or not (0)? For the majority of the world where Windows has a massive market share, “apple” would recall a fruit, not a sleek laptop. Therefore, the vector for apple along the computer/non-computer dimension is probably [0.3].

Taking this together, apple = [0.8, 0.3] where positionally, 0.8 is the value for positive/negative sentiment while 0.3 is computer/non-computer.

Agree?

(Hoping you do)

But that [0.8, 0.3] vector is for the general population.

Would a bible literalist who publishes blogs on bible stories feel the same way?

For someone like that, the notion of the original sin could taint their sentiments towards the apple. So they might weight an apple at 0.2 on the positive/negative line. Since they’re bloggers, it’s more likely they associate apple with computers so they might call it 0.5. Therefore, their apple vector is [0.2, 0.5].

Extend this to more content and you’ll see why there are more than one vector.

At least that’s how I understood it. Happy to be corrected and/or schooled.

If you look at something like LangChain[0], it supports/recommends splitting larger documents into smaller chunks. In this way, when doing something like semantic search you can get the specific paragraph/section that holds the closest relevance, rather than having to read the entire document again (think of a 100 page PDF).

https://python.langchain.com/en/latest/modules/indexes/text_...

One thing others didn't mention is that "document" is a general term but in some cases (e.g., question answering) the typical document can be a very short paragraph and take much less memory than the vector. Also note that with some ML architecture the vector is very large (e.g., an entire very layer output)

Storing one document as one embedding is like making a movie poster the average of all frames in the film.

I think you can and it has some benefits. One interesting thing that can help is to store representations from transformations over the document and then "fuse" the vectors (i.e. average them) at indexing time. You are effectively able to do run-time augmentation but without any extra inference overhead at query time and without increased memory. An easy way to think of this is for similarity measures that are linear (i.e. dot product) you are now scoring the document over a weighted sum of the transformations of the document. Test-time augmentation is a very well known method in ML generally for improving performance and is applicable here. You can do the same for queries as well - akin to query expansion.

I'm interested if anyone has some hard data on the "best" size of the document "fragments" that are used for embedding into a dense vector.

Obviously, embedding single words probably aren't particularly useful for reassembling portions of a document for submission to an LLM in the prompt. I'm currently pondering on what size of string is best for embedding, and considering a variable size might be one option.

Testing with strings around 512 characters seem to do pretty well, but it may be storing multiple lengths of similar runs in the document might be a better way to do it.

On your last point: I guess recalculating 10 million embeddings takes much longer than the 20-30 mins to re-index?

Or perhaps we care because calculating the embeddings can be done in parallel with no limit, but the indexing is somehow constrained?

I've been playing with Hachi this afternoon, as it's a more developed version of something I've been hacking around with on and off for a while, and there's little point is duplicating the effort. Thanks for the release - first impressions are very positive!

There are a couple of things I'd like to discuss/suggest, and also FWIW I get "Face Recognition not available" when I try that feature.

Should I use Github, or is there a better way to contact you?

Thank you for building and sharing this!

Have you tested this on a large photo/video collection? How does it perform with (say) 20k images?

Excited to try this out for my own needs, as long as it can handle real-world usage.

This flexibility is a big draw. I can experiment with in memory, launch with cloud, and move to my own infra if I’m lucky enough to need that kind of scale.

> Each p1 pod has enough capacity for 1M vectors with 768 dimensions.

> Each s1 pod has enough capacity for 5M vectors with 768 dimensions.

I think that’s what I was remembering. I guess I just don’t understand what that means.

It really boils down to which stack you are familiar with / which stack integrates better with the rest of your infrastructure / use case.

If you're already ingesting a lot of data to ES and you want to vectorize it, it has a good support for cosine similarity in its indexes.

If you store data in a PG database and you want to make it searchable by similarity, then pgvector is a good choice. It's especially powerful coupled with the ease of use of the supabase platform. You can make a document-based chatbot very very quickly.

In both cases it is more of a datatype and a lot of your logic will still reside in your application layer.

In my case I was already ingesting data into elastic, so I just added a dense_vector property to my index, and a vectorization step in my external code by calling the openAI api and saving the result into dense_vector.

In the future, I'm planning to build an AI powered webapp and my stack of choice will be supabase + pgvector because it's a better option as a public app backend.

Doesn't a faiss index have to fit in memory?

> The reason for this is: dedicated tools are always better, faster, and more feature-rich.

Depends. Polyglot persistence has the benefit of letting you use the "best tool for the job" for each job but that benefit can fall apart if you have cross-cutting concerns. If you need to query across different storages you often end up compromising on several of the initial benefits (e.g. performance from passing data between storages or resource use and consistency from duplicating critical data).

For example, you could store your graph data in Neo4J and your document data in MongoDB but good luck doing graph queries that need to access the document data. OR you could use something like Tiger or Arango that's a graph database that can also store data other than pure edges.

It could be that pgvector isn't good enough for serious use. It would be great to see some benchmarks. The one you site sounds like there's enough reason to try something more specialized.

Postgres is quite good as a JSON store btw.

In general I agree with you, but

> You can also store JSON into Postgres, but you should better use MongoDB for NoSQL purposes, right?

It's a common folklore at this point that Postgres is sometimes a better document store than most NoSQL databases (including Mongo), see for example this which is also at the front page of HN today, https://news.ycombinator.com/item?id=35544499

“Dedicated tools are always better, faster, and more feature-rich.”

Yes, most people are perfectly OK with just one car.

Specialists need special cars, but general public most of the time is ok with a family car.

PgVector might not be production ready, but it doesn't mean that for most of the situations Postgres's full-text, JSON, GIS, Graph-walking, Queues etc. isn't good enough with advantages of using just one database. There's a new category of problems when you let your data be in multiple places. On top of that, when you go full into Pg, with things like PostgREST, you can sometimes end up with a very minimalistic backend.

> The ANN index IVF implemented in pgvector has very poor performance, with only around 50% recall.

My understanding is that this mostly due to the default settings that pgvector uses (nprobes = 3) and not due to the usage of IVF. The recall would improve significantly with better defaults. This of course would also increase the latency of vector searches, but that is the trade-off of using IVF instead of HNSW (worse latency at high recall, but much lower storage/memory costs).

Shilling from the same people who ask to go upvote their posts in "new" on HN over private forums?

Yeah, right, dedicated tools my ass.

Having many point solutions is problematic from a cost and complexity standpoint. General purpose solutions that would work for 80% of your use cases would be better long term. Having said that, I don’t think Postgres can be used as a general purpose solution to cover vector search, full text search and NoSQL use cases. The best general purpose solution would expose a unified API but under the hood use different storage engines to support these diverse vector search and full text search use cases.

This is fascinating, thanks for explaining this use case. It sounds like it's a fancy way to get a generic LLM to understand your domain by including the relevant context in a prompt.

What about specializing the LLM in the first place? Could you fine-tune the model by training it on those internal documents instead? Then you don't need to mess about with the prompt, right, since the LLM has "learned" the data?

Thanks for mentioning Haystack :)