Hacker News Reader

Comment by tudorg | Hacker News Reader

(disclaimer: author of the post)

I plan a follow up to compare it with Elasticsearch, however, I don't think I'm going to attempt benchmarking, because whatever realistic scenario I come up with, it will not necessarily be relevant to your use case.

I mostly agree with you and I probably wouldn't use this at large scale (say, more than a few million records). I was primarily interested how much of the functionality I can replicate. Because for small search use cases this has some clear advantages: less infra to maintain, strong consistency, joins, etc.

Also, at Xata we're thinking about having a smooth transition between using Postgres at small scale, then migrating to use Elasticsearch with minimal breaking changes.

View on HN

Using measured comparisons and making informed choices is certainly the best way.

If you can start with Postgres to have a relational database with the benefit of Full Text Search (i.e. avoid Elastisearch) as well as JSON fields (i.e. avoid MongoDB) then you end up simplifying initial hardware/software requirements while retaining the ability to migrate to those solutions when user demand requires it.

So many developers seem to build with the idea that they'll become the next FAANG when actual (or reasonably forecasted) user load doesn't remotely require such a complex software stack.

I think you’re being ungenerous in the reasoning. The initial setup to get something running is similar while a specific technology may scale better. From that perspective, you may be better off picking ES to avoid paying back technical debt if you do need to grow. Early strategic investments help - sure, you won’t get everything right, but getting enough things right can be the difference between the sales channel being bottlenecks on deal flow vs on engineering. The former can be a lot faster to resolve than the latter.

No failed company ever failed because of a lack of technical complexity, nor have successful companies been held back by not adopting complex architectures early.

> "It is hard for less experienced developers to appreciate how rarely architecting for future requirements / applications turns out net-positive."

— John Carmack

https://en.m.wikipedia.org/wiki/You_aren%27t_gonna_need_it

I think picking lucene/es for search is hardly "architecting for future requirements" if your task is to build search functionality. Using Postgres FTS for search feels much more like cutting corners for me and an under-appreciation of the complexity inherent in full-text search, a complexity that ES solves competently and Postgres barely even addresses.

I'm not against picking the right tool for the job.

My only point is that making an architecture decision because it will immediately reduce complexity is much more sensible than basing your choice on potential future needs.

Evaluating needs from a "complexity reduction" standpoint is safe and will net returns. Evaluating needs from a "potential risks" standpoint is a lot harder, and easy to do wrong; the true risk is not growing at all, so the heuristic for any new project should be to do the simplest possible thing that solves the problem and starts the scaling process (i.e. whatever produces a saleable product).

The other benefit to starting with uncomplicated architecture is that you leave yourself with more scaling vectors, so once you deeply understand the problem(s) you actually need to solve, you can pick the right tool.

For us, Postgres FTS covered 95% of our use-cases. If we had started by just using ElasticSearch, we would have had a lot more complexity to maintain, and we would never have discovered our current (surprisingly elegant) architecture.

You're talking about application complexity when the parent is talking about operational complexity. The difference between some app servers talking to a Postgres instance with a replica and that plus an ES cluster is jump. If you're paying for something managed then it probably doesn't matter much to you. In my experience it's actually quite rare that teams working on a product where search is a feature and not the product itself ever graduate from what Postgres offers so I tend to push back on ES when it isn't free.

You may be able to hedge your bets by separating the Postures search DB from the OLTP. Perhaps it could use FDW or similar to make them appear the same from the app layer.

Then if FTS won't scale to unpredictable future needs it should be easier to rip out and replace with ES or anything else. And one doesn't have to pay that cost if/until it's certainly a requirement.

I had done something similar recently. Goal was to take a huge Postgres database and make it searchable and usable.

It ended up that doing a offline batch job to boil down the much bigger dataset into a single pre-optimized table was the best approach for us. Once optimized and tsvectored it was fairly performant and not a huge gain with Elastic. Still keeping the elastic code around “in case”, but yeah, Postgres search can be “good enough” when you aren’t serving a ton of clients.

Sure, benchmarks won't prove that it works in ALL cases, but it can at least prove it works in SOME cases?

Reply on news.ycombinator.com