“Lucene HNSW is the future and (pure) vector DB vendors are in trouble.”

twitter.com

6 points·softwaredoug··3 comments

I hope people on Twitter realize how many of us refuse to jump through Elon's hoops. Case in point, I see a single bare, uninteresting tweet while signed out with no indication of how to view what I assume is a thread.

I'm trying to keep my judgement in check but it's getting harder and harder.

Don't worry it's not that interesting. The author argues that Lucene has more types of data formats, indices etc. that the mix is more useful than each one feature individually. Of course what we actually see with databases, including also graph databases and SQL databases, document databases, is that all databases sooner or later end up as hybrid databases. So this category of pure vector databases it's simply a strawman, if not now then in the mid term

I just saw Jimmy Lin’s Twitter post a little while ago and have been looking at this. His work in the past has been hyper-useful to me (e.g., his book with Chris Dyer on map reduce algorithms) so this automatically gets looked at.

He is comparing to text embedding vector data stores, like people use with LangChain and LlamaIndex for LLM applications. I have been basing my work for 7 or 8 months almost exclusively on text embedding vector data stores - to be honest I am in the mood to dig into something else for a while.

Reply on news.ycombinator.com