Hacker News Reader

Comment by andrenotgiant | Hacker News Reader

I think what we're seeing with every database company providing new full-text search capabilities is an example of AI coding productivity showing up in the real world.

It started with paradeDB and pg_search https://www.paradedb.com/blog/introducing-search

Timescale has pg_textsearch https://github.com/timescale/pg_textsearch

Neon and Databricks have Lakebase Search https://docs.databricks.com/aws/en/oltp/projects/lakebase-se...

Now PlanetScale.

AFAIK all of these are implementations of the BM25 algorithm. You can just tell an agent to read about BM25 and implement it in your system of choice. Cool to see. Seems like there's still a lot of juice to be squeezed out of how it's architected and integrated into each system, but you can't help but wonder if this will lead to aggressive commodification

View on HN

There is a lot of truth to this, but it's also very much down to domain experts being able to do this to move faster.

Planetscale (assuming they used a agentic development practice) will have pulled this off, to the level of performance that they have, because they have a team of very highly experienced Postgres developers. Their knowlage of Postgres internals will have given them the insights needed to steer the models to a plan that used the architecture as described in the post. That's not something a model can do on its own*

World experts + LLMs = moving mountains.

(* we're obviously seeing something a little different from inside the research teams in the labs. They are showing that the models, when you burn the level of tokens only they can, are able to do novel things from the models own insights.)

> There is a lot of truth to this, but it's also very much down to domain experts being able to do this to move faster.

Yeah, I don't think I could tell Qwen3.8 (my LLM of choice) to study up on bm25 and then implement full text search in the couchdb instances I maintain without studying both bm25 and couchdb internals myself.

Oh, you could.

Yeah, he totally could. Wherever the output is actually useable or a dumbsterfire would be opaque for him, however

Seems like a lot of this knowledge was encoded into the blog post. I wonder if given this post and access to a planet scale instance to compare with, how close an agentic agent could get.

The folks at Springbird are giving it a shot: https://github.com/TeamSpringbird/stannum

The easiest way to find that out is to TIAS

I think we can frame it as LLMs materializing existing potential. It seems like there needs to be an underlying potential to tap into, without which, the results could be slop.

THIS++

LLMs are becoming a world expert in everything.

I am exploring this exact area of search and analytics for vanilla postgres as replicas. Guess what, the LLM came up with this exact conclusion of using ctids as docids, all by itself. It was surreal for me to read the blog above , when I hit that paragraph about ctids.

I am no postgres internals expert.

>all by itself

Or it’s read countless articles on doing the same thing.

I mean, that goes without saying for LLMs. It is an approximation of human knowledge after all.

In the same way that a google search is .

LLMs are still playing word association - humans have something a little more complex going on where the word has meaning.

Was not able to find any mention of AI or LLM usage on the article. The article is very detailed and goes in depth about how they have been able to do it. If anything it just shows the database level expertise and understanding of the existing implementations to find the optimization opportunities.

Unless its explicitly mentioned lets not dilute the credit of the folks who worked on.

ParadeDB's implementation builds on the Tantivy crate, which predates AI coding.

BM25 is the easy part. It's probably a dozen lines of code, maybe two. The real work is in the design of the index that enables you to write that dead-simple function -- the in-memory data structures, the on-disk data structures, keeping them in sync, fault tolerance, batching, and a bunch of other things.

If you ask Claude to "implement BM25" you will not get what you want. I see a whole lot of this: people that don't know what they're doing get garbage results out of LLMs.

I don't completely disagree with your hypotheses but it feels like the hard part of his TIN stuff isn't BM25 (which has been around for donkeys years) it's all the hardcore storage engine work around it. And is an LLM particularly good at e.g. segment merging under a thousand updates a second? I've had a few situations where I've been told "we've hit the perf floor" by Claude only to have persisted myself and shaved substantial amounts off still.

More damning for the theory might be that I think paradedb's pg_search predates the agentic coding by a few years?

ParadeDB/pg_search developer here. Yes, we started the project before LLMs. Moreover, our project is now much more than just full-text search.

I do agree with the sentiment here that many of the other FTS-in-Postgres projects in recent years have been heavily enabled by vibe coding.

Or how to transform the open-source, no-vendor community PostgreSQL into a proprietary database, where each managed service comes with its own unique syntax, features, and benchmark claims. People go to PostgreSQL for freedom, not to bind their application to one extension with one owner and one maintainer.

Reply on news.ycombinator.com