Weaviate is an open-source search engine powered by ML, vectors, and GraphQL

>"For van Luijt, this was an "Aha" moment. Like everyone else working in technology,

he had to deal with lots of unstructured data.

In his words, relating data is a problem. Data integration is hard to do, even for structured data. When you have unstructured data from different sources, it becomes extremely challenging.

Van Luijt read up on RankBrain and figured it uses word vectorization to infer relations in the queries and then try to present results. Vectors are how machine learning models understand the world. Where people see images, for example, machine learning models see image representations, in the form of vectors.

A vector is a very long list of numbers, which can be thought of as coordinates in a geometrical space. Three-dimensional vectors -- i.e. vectors of the form (X, Y, Z) -- correspond to a space humans are familiar with. But multi-dimensional vectors also exist, and this complicates things:

"There are many dimensions, but to paint a mental picture, you can say there's just three dimensions. The problem now is, it's great that you can use a vector to recognize a pattern in a photo and then say, yes, it's a cat, or no, it's not a cat. But then, what if you want to do that for one hundred thousand photos or for a million photos? Then you need a different solution, you need to have a way to look into the space and find similar things."

This is what Google did with RankBrain for text. Van Luijt was intrigued. He started experimenting with Natural Language Processing (NLP) models. He even got to ask Google's people directly: Were they going to build a B2B search engine solution? Since their reply was "no," he set out to do that with Weaviate.

Searching the document space with vectors

NLP machine learning models output vectors: They place individual words in a vector space. The idea behind Weaviate was:

What if we take a document -- an email, a product, a post, whatever -- look at all the individual words that describe it and calculate a vector for those words.

This will be where the document sits in the vector space. And then, if you ask, for example: What publications are most related to fashion? The search engine should look into the vector space, and find publications like Vogue, as being close to "fashion" in this space.

This is at the core of what Weaviate does. In addition, data in Weaviate are stored in a graph format. When nodes in the graph are located, users can traverse further and find other nodes in the graph.

It's not that it isn't possible to store vectors in traditional databases. It is, and people do that. But after a certain point, it becomes impractical. Besides performance, complexity is also a barrier. For example, van Luijt mentioned, in most cases, people are not privy to the details of how vectorization happens.

Weaviate comes with a number of built-in vectorizers. Some are general-purpose, some are tailored to specific domains such as cybersecurity or healthcare.

A modular structure enables people to plugin their own vectorizers, too."

PDS: In the above context,

"Vectorizer" = "Domain Specific Search Engine"