32,067 karma · joined September 20, 2008
You can find more thoughts from me: https://www.craigkerstiens.com
[1] https://github.com/pganalyze/libpg_query/blob/15-latest/READ...
If you're designing from scratch and make it worth with Citus then (specifically for a multi-tenant/SaaS sharded app) it can make scaling seem a bit magical.
The good thing is this would at least shine a better light on all those that "claim" to be Postgres but really have little to nothing to do with it. Overwhelmingly people are supporting the wire protocol despite being a completely separate database because Postgres is already so universal and it'd be a huge investment to recreate that ecosystem of language drivers and everything else around it.
The reality is even "wire protocol" there are varying levels of support depending on what you're trying to do.
Then when you get down to functionality, it could be "we support the Postgres data types"... well except this one or that one. That's fine and good until a user is surprised 2 years into building an application.
Even the notion of we support all Postgres extensions, well all Postgres extensions don't work together some take hooks and change queries that other extensions want to modify for themselves.
Having worked with Postgres and managed Postgres for a very long time. Postgres is Postgres, there are extensions that modify Postgres, there are forked versions of Postgres, and things that are "Postgres" compatible simply aren't Postgres.
There is no 'standard' or 'spec' for what makes something Postgres wire compatible.
This feels like a strong overreach on the marketing front to leverage the love people have for Postgres to help boost what they've built. That is not to say there isn't hard and quality engineering in here, but slapping Postgres compatible on it feels lazy at best.
That isn't fully open source at this time but has been production grade for some time. This was one piece that makes getting to that easier for folks and felt a good standalone bit to open source and share with the broader community. We can also see where this by itself for certain use cases makes sense, as you sort of point out if you had time series partitioned data, leveraged partman for new partitions and pg_cron which this same set of people authored you could automatically archive old partitions to parquet but still have thing for analysis if needed.
We recently submitted our (Crunchy Bridge for Analytics-at most broad level based on same idea) benchmark for clickbench by clickhouse (https://benchmark.clickhouse.com/) which puts us at #6 overall amongst managed service providers and gives a real viable option for Postgres as an analytics database (at least per clickbench). Also of note there are a number of other Postgres variations such as ParadeDB that are definitely not 1000x slower than Clickhouse or DuckDB.
Justin Searls (fairly known in Ruby and Rails community) mostly quit a lot of various social channels though publishes on some of them one direction. He started a podcast that wasn't meant to be guests of some specific topic, it's just him updating you on things. What he's working on, what he's learning, random stories, etc. - https://justin.searls.co/casts/
Brandur who I've worked with at a couple of places (Heroku previously, and now Crunchy Data) who writes great technical pieces that often end up here also has more of a personal newsletter. While there are technical pieces in there at times he'll also talk about personal experiences my favorite one is some of the unique experiences hiking the Pacific Trail (https://brandur.org/nanoglyphs/039-trails).
Crunchy Postgres for Kubernetes is great if you're running Postgres inside Kubernetes, but is more of standard Postgres than something serverless. Citus also not really serverless at all, Citus is more focused on performance scaling where things are very co-located and you're outgrowing the bounds of a single node.
At this time it's not FOSS, we are going to consider opening some of the building blocks in time, but at the moment they have a pretty tight coupling on both the other extensions and on how Crunchy Bridge operates.
Marco and team worked were the architects behind the Citus extension for Postgres and have quite a bit of experience building advanced Postgres extensions. Marco gave a talk at PGConf EU on all the mistakes you can make when building extensions and best practices to follow–so in short quite a bit gone into the quality of this vs. a quick one off. Even in the standup with the team today it was remarked "we haven't even been able to make it segfault yet, which we could pull off quite quickly and commonly with Citus".
I recall seeing them crop up in the early days of building and running Heroku Postgres, they were a very very early managed service provider. To my knowledge they never seemed to grow to massive scale but were a steady business (though I don't know any of the details for sure). That they were still around for over a decade is a testament from a lot of others.
Single node Postgres (with a beefy machine) can definitely manage in the 100k transactions per second. When you're pushing the high 100k into millions read replicas is a common approach.
When we're talking transactions, question of is it simply basic queries, bigger aggregations, and is it writes or reads. Writes if you can manage to do any form of multi-line insert or batching with copy you can push basic Postgres really far... From some benchmarks Citus as mentioned can hit millions of records per second safely with those approaches, and even without Citus can get pretty high write throughput.
I'm not sure the full details on supabase as it's more recent.
This is a pretty good breakdown of various database providers and in particular a lot paired with Fly - https://dancroak.com/webstack/
My understanding of Fly Postgres is they put a lot into the tools to orchestrate, but there is not centralized monitoring and in the event of a failure it is up to you to realize and remediate.
Disclaimer: Was part of the team that built Heroku Postgres, and know the Fly team pretty well but don't personally use Fly Postgres so it's my understanding from the team. We've had a number of customers leverage Crunchy Bridge (build by a lot of the original Heroku Postgres team) use us for the managed Postgres connected to fly.io via Tailscale.
The shirts he's generally helped promote and publicize those companies so he has things to hand out to his students, TAs, graduate students.
There was sort of a fuzzy "major" contributors (https://www.postgresql.org/community/contributors/) which were people that contributed major features and then a list of other contributors. Depending on who you talk to this is either dated or a pretty close attempt at reflection of reality but not perfect. In recent years they expanded the contributors to include others that were contributing in non-code ways though it's still a decent place to find people contributing to major feature sets.
Of course this is not to be confused with the core team–which is more like a steering committee. But not so much steering committee of code and feature sets.
The bigger hurdle which Peter mentioned in another thread is simply building up enough expertise with the system and having the right level of domain expertise.
For several years there were no new committers at all. In recent years the team has tried to be a little more intentional about adding new ones and culling those no longer involved.
About 15 years there was a phase of letting a lot of younger people earn their commit bit. I can recall 3 people by name that all got a commit bit before the age of 25, and they may have actually all been under 22. One of those three shortly after moved on to work outside of the Postgres community, another quietly was busy on other things for over 10 years before coming back, and the third was actively involved going forward. I suspect there was some unease of folks getting a commit bit and then sort of falling off a cliff so it slowed for a few years on adding new folks. Edit - sounds like it was less age driven but maybe still slightly related to some folks falling off that there was a slow down in new committers – tldr - you're not getting a commit bit right out of college for Postgres.
What to me would be interesting but likely hard to gather is what age to people become a committer to Postgres. It wouldn't surprise me if the average age of getting a commit bit is closer to 45 than not. Many folks contributing come to Postgres after other systems work or just don't consider contributing to they're a bit more seasoned because it feels intimidating–I mean patches sent on a mailing list who does that any more? Postgres thats who.
I think a ton of the current set of "next generation" came to Postgres a good bit later. Even Tom himself will tell you he did some stuff with images for a few years which undersells himself (tiff, jpg, png - in some form involved in creation of each of those), then found this Postgres thing and started working on it.
Way more Rails apps used MySQL or other databases, it was largely Heroku winning Rails that led to the strong adoption amongst that community.