Will keyword (BM25, TD-IDF) be replaced for search by Neural Search?

We, humans, are preconditioned to be linear in our extrapolation (as opposed to exponential) thanks to our hunter ancestors (and FPS games!). It is very clear that the rate of advancement of Large Language Models is super-linear, if not exponential.

Hence, I indeed predict that keyword search will be completely supplanted in the next 5 years as a mechanism for search.

Of course we will still need to do lookups for ISBNs and generic ids, but that isn't keyword search, that is index lookup functionality.

Case in point: take a look at Meta Research's Contriever model (https://github.com/facebookresearch/contriever), which already matches keyword techniques in efficacy without any supervision.

This is only the beginning, come build the future with us, we see it very clearly :)

Amr Awadallah, cofounder of Cloudera, is arguing that Keyword Search will be replaced with Neural Search within the next five years. He’s not alone; a number of companies have emerged in this space recently (e.g. Deepset Haystack, Hebbia, etc.). Amr launched his new startup Vectara today. They claim their Neural Search as a service is as easy to use as Algolia, as scalable as Elastic, but "neural-first" leading to much higher semantic relevance (they are free for 15k queries per month). My question: do you really see neural search replacing keyword search, or do you simply see it as an extension?

There is a place for both:

If you are searching for a specific ID/ISBN some random token, keyword search will be always useful and easy to implement.

If the goal of the search is more semantically ambiguous and can not be expressed by a unique phrase, then neural search will be the way to go.

Most of the interesting applications of search will be semantically driven and therefore neural search has a big role to play.

Interesting! I wonder if it can help improve search for images/products/code and other artifacts.

People have been predicting the death of keyword search for at least as long as I've been working in the field (23 years and counting!). We've seen outrageous marketing claims, buzzwords like "concept search", "insight engines", many new companies promising a step change in search quality, some of which are still in business but many who shone briefly then vanished. The concept of an easy to use, fire-and-forget, scalable search engine that gives you great relevance out of the box isn't new. Yet the bag-of-words model remains the standard across the sector, it's well understood with many powerful and scalable open & closed source implementation options.

To make search work in practice however is hard. It's as much about process and people as it is about technology: many companies aren't even measuring search quality, recording search issues correctly or have an active search team (bigger than one poor overworked search person). No matter how clever the tech, these problems aren't going away: they're compounded by bad source data quality, misunderstandings of user search intent and bad search UX. Martin White, author of many books on search, describes search as a 'wicked problem'. Getting all these parts working in harmony so you can truly own your search is what we do here at OSC and it takes time, investment and commitment.

I think Vectara is very interesting and the people involved have impressive track records (there's also some other great engines like Vespa, Pinecone, Qdrant, Weaviate...). However I think the future of search is hybrid - we'll see keyword search still there for many use cases but enhanced by vector/neural approaches (the most widely used search engine Lucene recently gained vector features and work is happening on how to combine these with keyword ranking). No one approach will solve everyone's search problems, cope with special cases like part number search, or the specialised language used in some sectors, or always understanding the searcher's intent, magically without considering the human factors above or without extra tuning/training.

That said, with all these exciting new approaches, tools and companies, it's a very interesting time in the search world!

Further reading/viewing: at the Haystack EU search conference a couple of weeks ago www.haystackconf.com Dmitry Kan, host of the Vector Podcast (he featured the Vectara team a while ago) gave a great keynote describing the current state of vector search - I wasn't going to release the video until Monday but you can get an early look here https://youtu.be/2o8-dX__EgU . You can also read the joint article we wrote for The Search Network on vector search here https://opensourceconnections.com/wp-content/uploads/2022/05... (aimed at executives and others needing to understand the field).