Hacker News Reader

Comment by saisrirampur | Hacker News Reader

Based on my experience working with Postgres users since a decade (ex-Citus/Microsoft), I don't think Postgres is there yet to support every possible workload - ex: medium to larger scale (ex:1TB+) workloads in Real Time analytics, Data Warehousing, Search etc. Sure at smaller scales, it is very versatile to support any workload. That is why it is super common for companies to complement Postgres with other data stores. Don't mean to say that Postgres will not get there, I think it will! But I see it to be more in the long term.

View on HN

We operate with 80 Tb of data ATM. It is laying in several nodes and meta nodes (this is our own terminology). All Postgres.

Recently we need to move data from one DB to another, about 600M records. It is not biggest chank of the data, but we need it on different server because we use FTS a lot. And don't want to interrupt other operations on previous server. It took 3 days and costs 0.

Thanks for context! Totally understand where you are coming from. Postgres can be moulded to work for many use-cases. However it could take good amount of effort to make it happen. For example in your case building and managing a sharded Postgres environment isn't straightforward. It requires quite a lot of time and expertise. Citus automated exactly this (sharding). However it wasn't a fit for every workload. https://docs.citusdata.com/en/v12.1/get_started/what_is_citu...

Tbh I never bothered by these mental restrictions. I can't say that we have effort that requires more than normal human brain can handle (and we aren't smartest people on the planet).

I used to work in one project where we process big part of all shop's cash receipts in one of the biggest european country. We don't use any of these products. And it was done by one person.

Only stupid idea we had was to use AWS. Learned helplessness push people to change best product on the market but without salesman who tickle your balls.

Postgres is one of the best product on the market. But so much FUD makes a new space for "problem solvers" for the problem never exist. I'm not about Citus, I'm about idea that it is require much effort to build something around Postgres.

Is your workload an analytical or transactional workload ?.

Both, we process all the data in several blockchains and make public reports available for our users. Some of data can be processed once, others need to be recalculated on the schedule. So we process every block, every log entity in transaction nature. And also have a lot of data that processed several ways 2nd/3rd/etc times.

We also slowly evolve our internal analytics/intelligence. It is not something that generates high load, but will be at some point. Imagine something like dune.com.

> I don't think Postgres is there yet to support every possible workload

No system targets every possible workload. If that’s your goal, change your goal to make it more achievable.

With work (e.g. a cluster of Postgres instances), you can do quite a lot with Postgres if you think it through. If you want an out of the box solution to a specific problem, it may make sense to move away from Postgres. Of course another option is just to change the overall design/deployment of your Postgres cluster.

At some point you’ll need to do some custom work. Unless you plan on moving away from your new system next year because it doesn’t cover every possible workload.

since when is 1TB+ medium to large scale ? That easily fits on a single computer running Postgres, can even run almost all of it on memory depending on the server..

Sure, the workload could be compute bound than memory. It also depends on what query workloads is the postgres planner and executor optimized for. TPC-H benchmark is an example where vanila Postgres is not as optimal as pure analytical stores.

Reply on news.ycombinator.com