Hacker News Reader

Comment by 0x008 | Hacker News Reader

You can use custom ranking functions (shown in the article) add weights to columns, etc.

What techniques are you talking about that cannot be implemented in Postgres?

The hundreds of features things like Solr and Elasticsearch/Opensearch have. These are complex products but mostly this is a form of necessary complexity.

To be blunt, the intersection of people that know what they are doing on this front and people that choose to use Postgres for this is pretty narrow. It does happen and I've seen a few nice things built with Postgres. But mostly it's just people using the wrong tool for the job.

I agree with you, but to be fair, that's true of virtually every technology I've encountered in my career.

This is partially true. You're right that the feature set is necessary, but it's now possible to do this in Postgres.

ParadeDB pg_search bundles Tantivy, a Lucene-inspired library inside Postgres, to give users the feature set (BM25 ranking, tokenizers, faceted search, etc.) while keeping the benefits of Postgres (keep your data normalized, avoid ETL, etc.)

Disclaimer: I work on ParadeDB

I’ve been using pg_search for a tranche of emails in 2+ GB PDFs converted to PDF-page-per-row. Along with some materialized views for regex extracted timestamps it has given us a means for data forensics that has been very fruitful.

Orchestrating this outside of a few Python scripts and a single instance of postgres would have taken much more work!

So happy to hear you like it. We have a lot more coming up for it. Stay tuned!

I used to have this same argument but apart from the few that I've used on solr, it is not trivial to have general search using it. Won't even comment on ES b/c they already target analytics better than search. I think it is worth exploring pg and other tools as all search cases are narrow/specific (ecomm, graphs, domain-specific documents etc), specially if you need facets and filtering. Also multilanguage ok to consider for a tool but products usually look for better recall at their original lang then to have same results in other languages.

Reply on news.ycombinator.com