RIP, vector database | Hacker News Reader

RIP, vector database

turbopuffer.com

391 pointsrazin116 comments

> This write amplification is large enough that our efforts to tune indexing throughput have started to hit diminishing returns.

> don't key on the ANN address. That is precisely the change turbopuffer v3 makes. As you can imagine, it is not a trivial change.

This is a direct parallel to how Postgres and Mysql built indexes.

Your design choice went from a Postgres design pattern to a Mysql one. The difference is the reindexing cost vs the lookup cost - Postgres optimized for lookup and Mysql does for indexing on writes. Or more accurately, Postgres was better with good schema design using joins & mysql was optimized for a bad design with less normalization where many indexes exist for the same table.

Postgres always points an index to a row-id within postgres which is an arbitrary value which changes on each update.

Mysql, always assuming the storage engine is pluggable, points to the primary index entry and adds an extra indirection to the lookup.

This means that you point the mysql index to a stable id, so unless you go update the primary key for a row, you won't have to update the indexes for all the attribute lookups you might have made to data.

I don't do databases any more that much, but the design for NIMBLE file format has a lot of quirks which are relevant to this specific idea (wide tables).

But the old Uber post about switching from Postgres to Mysql to prevent index amplification[1] is a direct mirror to this post.

[1] - https://www.uber.com/us/en/blog/postgres-to-mysql-migration/

> mysql was optimized for a bad design

TIL I should have been using mysql the whole time

Richard Gabriel's "Worse is better" vibes.

I literally had opening slide about Worse is Better when adding mysql support to peerdb. Lots of things in mysql have v1 as stupidest thing that works, then v2 fixes issues

There's a bunch of internal types like decimal vs newdecimal, binlog started out statement based until they realized uuid generation is random so added data replication on top. CDC offset started as filepos before GTID was made so offset could survive failover

There were aspects of the design I appreciated (logical slots in postgres have a bunch of drawbacks avoided by just appending to 2nd serial log which has an expiry date instead of tracking clients' offsets), but developing against protocol you learn to not try build a consistent mental model

MySQL had worse-is-better dominance over postgres until, like, 2016 or so? I'll take the worse solution with better performance and a better replication story any day.

I hear you, but there's just so many knobs on postgres that at our correct scale we couldn't amortize the time and effort to become proficient at postges.

MySQL was often just much easier to start with for a greater number of people for the average complexity and need for the vast majority of projects.

This in no way made Postgres any less amazing and cool quietly all those years - if anything it's what really let it step into the forefront the past few years.

As someone who has worked with more than a few databases and MySQL a lot, I'm quietly content learning and beginning with postgres every chance I can get now, to see where and how long "Postgres for everything" can work in a project, simply from there being fewer pieces to build, maintain, integrate, and let the bottlenecks reveal themselves instead of prematurely optimizing for them.

I find it amusing people started quoting LLM output and are responding to it. Hopefully the original authors end up having the LLM respond back.

Many of the commenters on this site are also obviously LLMs. I'd imagine that quite a few of the entities quoting aren't necessarily people. Keep an eye on where they slide mentions of other products that a marketing team would like to promote.

Personally I haven't seen it too often; the other aspect is that HN is a forum for startups to pitch shit to each other, so this has been happening with or without marketers (f.e. "i'm working on a similar thing").

That LLMs are taking over the comments section is something that was already flagged, and Lobste.rs and others have started solving it by having gated registrations. HN should do this but it is unlikely to until it is too late.

lobste.rs has always been invite-only.

Sorry, how does a gated registration stop someone creating an account, then handing it over to an LLM?

Serious question, am I misreading what's being said?

It stops the automation part. One LLM spammer can be dealt with.

One can focus on generating 10 other accounts before it's flagged. Now you have 10 others to deal with.

But that does not work on lobsters (or anywhere else with gated registration), because you cannot just create 10 accounts. You can create 1 account after 1 invite. If you abuse, you loose the invite (and maybe the person inviting you the right to invite others).

I think, if i have understood the answers correctly that it's stopping a quick proliferation - but not a long slow infiltration?

The spam bots I regularly deal with on reddit are using accounts that are several years old. I'm not sure if they are hacked, purchased, or something else.

I don’t get it though… what’s the point of using an LLM just to comment here? Like what does it gain the person doing it

Marketing and influcence.

For example, consider this prompt -- "Find topics that would be relevant to people interested in <company> and post a topical post that mentions <product>".

Also, influencing public opinion on certain topics, such as Israel or Palestine, the current administration, the democrats, the various wars that are ongoing, or AI itself.

Some bots are trying to influence conversations on some topic, and they comment on neutral topics to build up karma or whatever.

Also, some people are weirdos.

Disproportionate number of the "weirdos" category on this particular site vs astroturf marketing bots on Reddit/etc.

I've lost count of how many times I've clicked into a bio from a flagged, obviously LLM written comment to find some variation of "Building new AI tools for agentic devops"...

There are HN accounts that have been shadow-banned for years, and yet they keep posting despite few people ever seeing their posts. They're not LLMs AFAICT, but they just keep posting into the void.

So why use an LLM for commenting? See above, people are weird.

Steelmaning it: perhaps people who cannot write to save their life are tired of grammar Nazis correcting them? (Admittedly since the llms I have noticed less and less people being pedantic about these things, perhaps it is like the ill fitting cupboard door signaling that this is handmade)

> Like what does it gain the person doing it

Time / quantity.

Pre-LLMs you needed a whole "marketing" agency to astroturf on a meaningful scale. Post-LLMs a single person can run multiple astroturfing campaigns in parallel.

LLM's when leaned on too heavily, have a tendency to destroy critical thought, or perhaps more accurately replace it.

It is similar to a spell checker, while spell checkers improve spelling in general, they do not improve a persons spelling ability. Instead acting as a crutch. No need to spell well when the machine will do it for you.

Some people really like expanding their thoughts via LLM prompt. Some so much it acts like a big crutch, no need to think coherently, the machine will do it for you. So they use the LLM for everything.

As a related tangent something is messed up in my web browser spell checker, it gives the red squiggles indicating a misspelling, but refuses to give suggested corrections. I would fix it but... My spelling ability has never been better than it is right now.

Build up karma to expand the reach of future promotional posts.

How do you prove you're not an llm? The only thing that would work is to add sexist and racist terms lmao.

> How do you prove you're not an llm?

I feel like we've already reached the point where HN users have discovered 100 of the last 5 LLM commenters. It's the new way to disagree by not having to engage with the argument at all. Just say that a certain sentence structure or word means it's an LLM and move on.

One of the things HN has been missing for a very long time is an etiquette policy around this very thing. Accusing people of being "russian bots" and such is downright poorly mannered, and dang's policy of "less is more" succumbs to mob mentality. The proliferation of mob mentality is one of the things that has eroded the quality of HN over the last... I want to say 10 years, but it might be going too far back.

However, such a policy requires enforcing otherwise its like the rest of guidelines - vapid shit. Maybe now that HN is infused with LLM-Powered Moderation™, dang can do a bit better.

> Your design choice went from a Postgres design pattern to a Mysql one. The difference is the reindexing cost vs the lookup cost - Postgres optimized for lookup and Mysql does for indexing on writes. Or more accurately, Postgres was better with good schema design using joins & mysql was optimized for a bad design with less normalization where many indexes exist for the same table

You are right that MySQL does better when you have lots of indexes, but I don't think the tradeoff is that the overall Postgres architecture is better with good schema design.

Having secondary indexes point the primary key enables things like undo logging, which obviates the need for vacuums - vacuums being the most painful part of Postgres. On top of that your primary key index will be mostly cached so the cost of the indirection is much smaller than it may first appear

I think OP is just alluding to the fact that Postgres needs to do less work to go from secondary index to table data, since the tid is a direct pointer to the exact page and slotted entry while MySQL needs a b-tree walk.

> primary key index will be mostly cached so the cost of the indirection is much smaller than it may first appear

Not sure I follow. If it's in-memory you save having to read from disk, but you still have to walk the b-tree to go from PK to data.

MySQL was generally (pre 8) optimized for point queries on primary keys. So rows are stored in the PK index, the PK index is a clustered index. Everything more or less falls out of this.

Around v2 myisam was not a clustered index. It changed with innodb.

Sure, I'm only taking about innodb. The join strategies supported before 8.0 (5.7 and lower) were basically nested loop and that's it. Which only really works well for a couple of query patterns: point queries with a handful of joined tables all using indexed FKs, or a big sorted scan over one single table and everything joined to it, with pagination through where clause and sorting all on the primary key, and possibly a straight_join to make sure it doesn't start the execution plan anywhere other than the spine table.

> only really works well for a couple of query patterns

To be fair this probably covered probably 9x% of production queries, like high-selectivity index-covered filters and sorts (with a limit). Even today most planners will still use an inner loop.

It's been interesting to watch how different planners have evolved over the years. SQL Server 7 in 1998 launched with features that MySQL and Postgres wouldn't catch up to until the late 2010s, but they had other features and quality-of-life improvements that made these edge-case optimizations hardly noticable.

> I think OP is just alluding to the fact that Postgres needs to do less work to go from secondary index to table data, since the tid is a direct pointer to the exact page and slotted entry while MySQL needs a b-tree walk.

Yes, this is true, but they framed this as "the Postgres approach is better when you have a good schema design", but that's not true. There are plenty of ways the MySQL approach is better even when you have a really good schema.

> Not sure I follow. If it's in-memory you save having to read from disk, but you still have to walk the b-tree to go from PK to data.

The point I was trying to make is that going to disk is going to be orders of magnitude slower than doing an in-memory B-tree traversal. Because of that, the cost of doing an extra b-tree traversal to find the page you're looking for is a relatively small cost compared to reading the page in the first place

.. and undo logs make rollbacks and crash recoveries slower. It's a trade off. MySQL storage engine architecture is nice, postgresql extension mechanism is nice.. and so on.

> Having secondary indexes point the primary key enables things like undo logging

That's neither here nor there. Heap-oriented tables can have undo logging, too; and index-oriented tables don't strictly require undo logging.

It's just that you're much more likely to want something like undo logging for index-oriented tables, because maintaining uniqueness for rows that are not in-place updated in the index becomes very expensive as more and more versions of the same key value may need to be checked for visibility and liveness, and removing old deleted versions becomes a maintenance hassle, too. It can be done without undo logging, but apparently that wasn't a sufficiently robust (or performant) design.

MSSQL (Clustered Indexes) and Oracle (Index Organized Tables) among others let you chose because there are advantages and disadvantages for different situations.

Not having true clustered indexes in PG is something I miss coming from MSSQL, it helps performance when the majority of access is always primary index avoid indirection from index lookup then tuple lookup and it also saves space if its the only index.

I feel the same way.

Vector databases were always more about retrieval than either vectors or data storage. But the term stuck all too well and companies held on to it a tad too long. Sorry :)

That's just a search engine, but then you're competing with traditional players like Elasticsearch and Vespa who all have built-in vector support by now, and you have to compete on attributes like price, performance, features, and who can mention 'AI' the most times on their web page.

I’m developing a local “code graph mcp tool” (not yet published) and followed a similar path, though I may have been able to go further since I have fewer vectors in my database (even on projects with 50M LOC).

At first, I tried all those popular vector databases and was disappointed with their performance. In the end, the best and fastest solution turned out to be building a multi-database system on SQLite, compiled with everything related to multi-client operations removed. Only exclusive mode was left. Everything is as binary as possible. The index is completely separate — an IVF with pre-training — and is built on the GPU (250K vectors are built, processed, and saved in 4 seconds). Right now, my biggest problem is frequent data changes, and I need to implement optimizations to reduce recalculations.

So far, I haven’t seen any vector database implementations that are heading in the right direction. Maybe only Lancedb looks promising, but it’s too heavy for my needs.

I have a deja-vu with the NoSQL databases, gladly to keep using classical SQL databases for data, while they get the features that matter after the dust settles.

AI has some of the craziest up and down cycles of tech I've ever seen

Yeah, that's true

I never bet on vector database tech (and I don't like RAG in general). Reasons:

- You create a new problem to solve an already tough problem (information retrieval) - You bill people for storing vectors and searching when they are already paying for information storage and search - You overcomplicate a retrieval problem that has well-defined solutions - All the BS HYPE

The LLM wiki is one good option. At WAIC, I saw a company change the filesystem to store the document index as you save documents so they never go out of sync.

Think harder boys.

I've really liked lancedb for similar use cases. Not just that it is OSS. But Lance treats ANN as a secondary index similar to what turbopuffer v3 does. Rows sit in fragments, and the vector index never moves them.

Dashboard was last updated on September 7, it started on September 5. Bug? Or no progress? It's linked in the blog post so would expect it to work: https://turbopuffer.com/v3

It is working! Check back soon for progress updates. The graphs will update once we have the companion dev log entry explaining the perf optimization.

As you can see from the dates we're a few weeks behind our publishing schedule. Hard to pull ourselves away from the perf hacking to write the dev log entries.

I see, you're updating the graph with 3 weeks delay, that's a bit confusing. Now there are 4 days at least.

They are not killing the vector database or vector search. They are killing the vector-primary storage layout.

The post itself says ANN becomes “just another” secondary index, and that v3 should make vector search faster along with text and regex.

“RIP, vector-primary index” would have been accurate. “RIP, vector database” is marketing.

Im not full read up on RAG pipelines, but has anyone ever tried to make the database a neural net itself? I.e get rid of any sort of traditional databases, and then you basically just have some sort of autoencoder?

In some sense there's probably a database compression scheme that does something similar. Usually people care too much about fidelity

There's been quite a bit of research into this over the past 3-4 years under the name "generative retrieval." The general approach is to use a transformer and treat the weights as the index. You input the query, and then used constrained decoding to generate the document ID.

Honestly don’t understand the hype over pure vector databases. Vector is more useful in a pre-existing standard like SQL. MariaDB server for example has vector through InnoDB and TidesDB and the ability to utilize both engines so you get read. write and space optimized engines with FTS and more. Seems way more effective to be able to have all the server jazz, predicates, sharding, replication, backups and many different kinds of engines to handle different workloads and indexing.

Soon they’ll just sell you markdown.

> The problem with a vector primary index

We've realized this a long time ago at TopK and built a flexible serverless search engine from scratch. Supports dense/sparse vectors, late interaction, lexical search, indexed regex, filtering, and custom scoring in one query.

- https://www.topk.io/blog/vector-dbs-are-the-wrong-abstractio... - https://www.topk.io/blog/topk-embed-v1

I'd want to see p99 at 1k+ QPS on the same scale

the multi-vector duplication thing makes sense, copying every attribute once per vector explodes quickly. what's the new primary index?

An automatically generated internal ID: (segment ID, doc ID). The user-provided primary key (the field called `id` in the document) turns into a secondary index at the storage layer.

Waitint for the CEO of Qdrant to step in

One more extra DB that is yet to to put to rest is Graph only DBs .

I find very little reason to use a pure vector database for enterprise retrieval. We built an enterprise retrieval engine on top of a SQL database with native vector support, and the flexibility is something we cannot ignore. Vector similarity is just one query primitive alongside full text search, filters, joins, ordering and normal relational predicates. Tenant/app/collection isolation becomes part of the query itself. ACLs, document versions, categories, metadata constraints and temporal filters are ordinary predicates rather than something you have to bolt onto a vector store. SQL is already going to be part of almost any enterprise system. Adding a separate vector database introduces another moving part and syncing two system whenever you update your data is the most difficult thing to get right.

Which database did you use?

We use cratedb. but now, clickhouse, starrocks all hve vector.

MariaDB as well

Yes, but we were looking for a distributed sql for massive government data scale.

for the same reason we used ElasticSearch for vector search, because it's just one part of a query. Everything else we already had (filters etc), just stay and can be combined with vector search.

Indeed! Why servers like MariaDB are super useful.

This sounds like the Postgres vs. InnoDB argument 10 years later. Postings pointed at physical location (the ANN slot), so every SPFresh rebalance rewrote every index touching that doc. InnoDB solved this by pointing secondary indexes at the PK and eating an extra lookup on read. Curious what that extra lookup costs you when it's an S3 GET instead of a B-tree hop.

"Updating one vector can move hundreds of attributes and their indexes" is basically Uber's 2016 Postgres write amplification post, but for search. Same fix too: stop pointing indexes at where the row lives.

So ANN becomes a secondary index that points at a doc ID, and vector search now needs a hop to complete. Do clusters keep their own copy of the vectors so the search itself stays local, and only result fetch pays the indirection? Otherwise cold p99 seems like it gets worse.

I haven’t read it, but “ stop pointing indexes at where the row lives” sounded interesting. So if not the row, what does the index point to instead?

It points to the full primary key (which rarely changes).

Ah, thanks. But weird though because I would have thought that’s exactly what it was pointing to!

Postgres is a database server, InnoDB is a storage engine. Comparing apples to oranges. MySQL and MariaDB both can utilize different storage engines that bring different things to the table.

AI Slop. Will not read.

The article?

turbopuffer is founded by some of the smartest people I ever worked with in past jobs. I strongly doubt they used an LLM in the writing of this article.

confirmed - we still write by hand

This one sentence sounds very LLMish:

> Object storage as the source of truth gave the economics, and tiered NVMe SSD/memory caches gave the performance.

Yes, that construct is, like em dashes, something that humans have been writing for a long time. That's the problem with triggering on one isolated tic. The tic comes from human practice, it's not like they invented it!

I'm not sure I've ever read a sentence like this before LLMs. Maybe I didn't pay attention, possible. I should do a search of pre-LLM writing for this construction.

Are humans who use em dashes that intimidating to you?

Ad hominem attacks are against the rules on HN, but derision of bad faith actors is encouraged. :)

I kept asking myself "is this ai written" while reading it to. Nothing to do wit h the em dashes. Phrasing like: > RIP, primary vector index. > The solution to these problems is simple: don't key on the ANN address. That is precisely the change turbopuffer v3 makes. As you can imagine, it is not a trivial change."

Cute heading, followed by wordy opening sentence that feels like it's repeating stuff even when it's not

Sounds like you need to read more.

FYI, replying "Stupid. Will not read" is less effort.

It would be nice to have a page that actually loads. This one doesn't. RIP.

UPDATE: It loads now, but it didn't when it was first posted. Traffic load on the server does matter.

loads just fine on my $10k laptop with 10g internet here in NYC

Also loads fine on my beater in the sticks :)

Takes 11 seconds to load on Firefox on Linux with 3G-level throttling enabled in Dev Tools.

Do you actually think that 10G makes pages load faster than 1G or even 100M? It doesn't. The blocker was most likely on the source server, not on your side.

You're the reason the /s tag has to exist.

The onion is more valuable when there are people to eat it, we should be thanking him

Deep

lmao

Loads really fast for me. (MacBook Air, average internet)

If you still have issues, try https://web.archive.org/web/20261001100105/https://turbopuff...

It has a pagespeed insights score of 55 and noticeably sluggish on my m3 max.

And what's with the throwaway account for this one comment? Is this becoming reddit with throwaway shills now?

fucking shills, making helpful comments and promoting seemingly nothing, what's this place coming to?

Yes, I get big money from the Internet Archive to promote their services. It's the new scheme that shills like me go for.

The reason is that I have no account on HN and rarely comment. I create a new account a few times a year because I don't remember or care about my previous account.

I could have made an account named john2026 and you would not think twice. Instead, I let people know upfront what type of account this is. Quite the opposite of what a true shill would do.

I got a Lighthouse score of 99 in Chrome. Believe it or not, I won't spend more of our time on this. (relevant XKCD: https://xkcd.com/386/ )

First Contentful Paint 0.7 s

Largest Contentful Paint 0.9 s

Speed Index 0.7 s

It makes a lot of requests, and some are stopped by my ad blocker, but most of them don't seem to make an difference. It is almost instant from my point of view. I disabled the ad blocker and didn't notice any visual difference.

Is it time to kill the database and replace it with a LLM optimized compiled version that simply implements the required API directly in (Rust) code, without any dynamic overhead? It probably will still be based of off a base design or a base file format.

Ultimately this system will encompass the whole OS, of course, but the DB might be the best place to start.

You mean get rid of Postgres and build bespoke database-esque systems for every use case?

If so, then no. It is not time for that.

Yeah, I'm struggling to come up with a really good time for that.

The best I got is if you are trying to do an old-school style video game asset/save game storage. But even then, the value in just using sqlite or even parquet is really high.

There's so many really good data formats that deciding on a new one at this point seems pretty silly. Particularly because what you sign up for when you make a new one is losing any and all tools that could be used to work with and diagnose that data.

Unless you're tigerbeetle and want to handroll every single thing you do lol

everything code is going to be liquefied, maybe even including tigerbeetle. simpler systems are simply more reliable, and the effects compound (= the best part is no part). organizations will get rid of database professionals and buy their migration expertise externally, when they need it. it won't be from Oracle.

Is it time to get rid of hammers and replace them with swiss army knives?

We could build a new hammer for each nail!

Why build one hammer and use it forever when you could pay $100 a month to build a new hammer every time you need to hit a nail?

for the database it is actually the opposite, as it is a swiss army knife.

look at the sibling comment, people are already doing this. with time only more people will.

an era is ending. there is a time and a season for everything.

Databases, like filesystems, are one of those pieces of infrastructure you do not want to build yourself unless you have a really good reason. They're complex, and you will not catch all the bugs yourself.

I have seen this happening already at two different companies. And I'm also doing it as well. Particularly for search indexes where there's no risk of data loss.

Thanks for sharing.

Data loss is the obvious concern. Are you perhaps reusing parts of sqlite or postgres?

There'd be no benefit. They're good general purpose databases but for any specific workload you can do significantly better on your own. For search in particular even if you use Postgres you wouldn't want to use Postgres FTS you'd want to at least use Tiger Data's extension.

Instead of a database, the LLM will expose an api endpoint and build a database on demand?

That's interesting. Maybe to decrease latency the LLM could "cache" it's build of it's database and reuse in between instances. It could host this artifact on a "hub" of git trees and then any new use cases that come up, can be added to this git tree. Then it can possibly be reused in different use cases.

Yes you got it. Get rid of all the abstractions we've gotten so used to and effectively approach this as an embedded project, where every line of code has to be justified.

It tends to make sure you understand the system fully, as no foreign concepts need to be imported and deferred to. That should make your organization run better.

Just like Rust does, btw.

Thx.

SQLite already exists and some people use it

SQLite is sure to provide a good parts toolbox. one may even use it as is ;-

Reply on news.ycombinator.com