thirdtrigger

Might be interesting to add an optional embedded Weaviate [1] with a flat-index [2] to the project. It wouldn't use external services and is fully disk-based. Would allow you to search the whole filesystem (about 1.5kb per file (384 dimensions) which would be added to the metadata as well).

1. https://weaviate.io/developers/weaviate/installation/embedde... 2. https://weaviate.io/developers/academy/py/vector_index/flat

TL;DR (from the article)

How the HammerBot Works

We also want to share details about how HammerBot works on the backend. Right now, our developers are using OpenAI’s text-davinci-003 model, otherwise known as GPT-3, trained on a custom dataset of our articles. The data is stored in a vector database from Weaviate and the bot is coded primarily with Python, using LangChain, a framework that makes it easy to customize AI output.

When you enter a prompt, the server queries against the dataset that’s stored in Weaviate to get the search results. Those are then sent to the LLM to help it develop a consistent response to your question.

A lot of chatbots will talk to you about anything on Earth. But don’t ask HammerBot for a knitting pattern! It’s designed to have limits on what it will talk about; it focuses on the expertise you can only get from Tom’s Hardware, so it may say it doesn’t know or can’t answer if prompts that fall outside of its training.

Agreed – one can also use Weaviate which comes with an OOTB OpenAI module leveraging the embeddings end-point https://weaviate.io/developers/weaviate/current/retriever-ve...

There is also an integration with Weaviate.

* Docs: https://weaviate.io/developers/weaviate/current/retriever-ve... * Post: https://weaviate.io/blog/2022/12/Cohere-multilingual-with-we...

Thanks for your kind words about Weaviate (I’m affiliated).

What you propose is certainly possible, you could do that with a vector database like Weaviate (https://weaviate.io). Import the data, vectorize it, and connect it to a person with cross-references

+ Weaviate for vector based search. Has a BSD-3 license. https://weaviate.io/developers/weaviate/current/

> At this point vector search is almost 'solved'

Well, there is is still a lot to do. Going from memory to disk, etc. But I see your point.

What might be a bit confusing is the mix of vector search libraries with vector search engines. It's a bit like comparing an inverted index library with SOLR :)

But we (I.e., the wider vector search ecosystem) are working on this

Agreed with your observations about the difference between the libraries and databases. Maybe for a follow-up article? It's nice to see some independent research on the topic, tho.

Agreed with fzliu, you can also use https://weaviate.io (disclaimer, I'm affiliated with Weaviate). You might also like this article which describes why one might want to use a vector search engine: https://db-engines.com/en/blog_post/87

Im affiliated with a company that makes one. We create a vector search engine called Weaviate [1] we also publish content on how it's done and the search engine itself is open source which might be helpful for you too.

[1] https://weaviate.io

Always when Ted Nelson shows up, this point is made one way or the other. I even once heard Vint Cerf make these points at a conference.

1. Xanadu being vaporware 2. His ego

What I really don't get is what Ted Nelson does to trigger this anger in people.

E.g., his book computer lib/dream machines inspired so many, and I find him super self-aware. His appearance in Werner Herzog’s movie about the internet was (IMHO) so refreshing.

I love Ted's work, and it's a huge inspiration to me in my daily work. This video just adds to that.

I'm affiliated with Weaviate, so maybe nice to get this out here for the record :)

We call Weaviate a "vector search engine" (i.e., we prefer "vector search engine" because it describes the type of database) since around Aug, 2020

Github: https://github.com/semi-technologies/weaviate/tree/a3967aff5...

The reason was simple; our community started to say that the mixed vector and scalar filter search capabilities were what they liked most.

Also, our benchmarks are available for quite some time here: https://weaviate.io/developers/weaviate/current/benchmarks/a...

They are based on ann-benchmarks.com but adjusted for full databases.

Interesting writeup. I'm of the opinion that the problem of the naming issue (how to call "things"?) sits in the idea that going from structured documents to structured data is one abstraction level too deep (i.e., people don't agree on how to call "things"). I believe this can be solved by similarity search; if we can approximate the data and represent the structure in embeddings. Hopefully, this might be a step in the 2nd try, as mentioned in the MD :)

> It would be like wikipedia, but even more all encompassing, and far more transformational.

You might like to see this (https://weaviate.io/developers/weaviate/current/tutorials/se...) as a step in this direction because it contains the structured Wikipedia data and the embeddings to target individual nodes in the graph.

MESHCode is a theory about how information might be stored in the brain. Very interesting and potentially has other implications for information storing as well. Check "The Mechanical Basis of Memory – the MeshCODE Theory": https://www.frontiersin.org/articles/10.3389/fnmol.2021.5929...

For Weaviate, you can check: https://weaviate.io/developers/weaviate/current/benchmarks/a... This is not a benchmark against other OSS solutions, but it gives you an understanding of what performance you can expect.

For production this might be helpful: https://weaviate.io/developers/weaviate/current/getting-star...

Great article! Thanks for sharing

Thanks, that's actually helpful :)

There is also a podcast where the results are discussed: https://youtu.be/kG3ji89AFyQ

Agreed! You can check out Weaviate too. It includes out of the box vectorization modules

Super cool article, we work on OSS vector search technology (Weaviate) as well and I can confirm that we see many similar use cases especially with multi-model models and rerankers.

Thanks and couldn't agree more! Its great to see how these models work in harmony with a vector search engine like Weaviate.

Did you see this blog? https://db-engines.com/en/blog_post/87

It's an interesting point tho. Maybe it's good to add this to the docs as well

> I am still waiting to see an end to end API doing all this

That’s kinda the idea of Weaviate. You might like the Wikipedia demo dataset that contains all this. You indeed need to run this demo on your own infra but the whole setup (from vector DB to ML models) is containerized https://github.com/semi-technologies/semantic-search-through...

Not true – Weaviate is open source: https://github.com/semi-technologies/weaviate

We are actually discussing this on the Weaviate Slack :-) https://weaviate.slack.com/archives/D02JM9D3HND/p16347312830...

That's a fair question – but I'm going to assume the open-source nature is being meant with this.

Jup – this is an example from the demo dataset in the docs: https://link.semi.technology/3DPcphe

These are all great!

There is also this video about modern search engines and Weaviate on the AI Coffee Break YT channel: https://www.youtube.com/watch?v=YkK5IKgxp-c

Interesting – we are working on an open source vector search engine called Weaviate and did the same for the complete Wikipedia and Wikidata.

[1] Docs: https://www.semi.technology/developers/weaviate/current/

[2] Github: https://github.com/semi-technologies/weaviate

[3] Wikipedia demo dataset: https://github.com/semi-technologies/semantic-search-through...

[4] Wikidata dataset: https://github.com/semi-technologies/biggraph-wikidata-searc...

Last week there was also a feature on Techcrunch about vector search and Weaviate: https://techcrunch.com/2021/12/11/2246180/