Neki – Sharded Postgres
planetscale.com
planetscale.com
- Intro: Explains problem but not what is Neki or what it's for.
- Why Neki: Explains alternatives and why they suck but not what is Neki or what it's for.
- How does Neki work: Describes the technical components but not what is Neki or what it's for.
- What you get beyond sharding: Explains how to run/deploy but not what is Neki or what it's for.
- What is a platform preview + Try Neki today: 140 words on the definition of "preview" and links to get started... but not what is Neki or what it's for.
Edit: The landing page has 100x more useful information up front -> https://neki.dev/
Edit #2: They've since added a "What is Neki" section. Good reaction time.
- short paragraph saying "we launched neki"
- short paragraph saying why neki was needed
- paragraphs about what neki is (with a header)
That seems like a very reasonable structure for a blog post like this one. Not being able to get to the third paragraph of a blog post seems like a "you" problem, not a blog problem.
Why do people post screenshots of an article on social media instead of linking to it?
Sometimes ignorance, but often because the web is such a hostile experience these days - paywalls and subscribe banners and cookie warnings and slow pages.
That's sad.
If you know that to be true and still add your links everywhere ("helping"), isn't this a spammer's confession?
Edit: your marketing site actually has excellent marketing information.
Edit #2: They've since replaced the blog post with a link to the marketing page. Good reaction time.
Does Neki solve for this, and if so how? My understanding of CAP theorem is that this basically requires some compromises around availability, but I'm curious as to what that looks like in practice here.
The usual way to run it is that you partition your db based on something like a user, so that single user gets a consistent DB, but anything cross-shard may not be.
I know when I worked at Block, Cashapp was using Vitess and getting cross-shard DB writes down and functioning correctly was one of the major blockers to adoption. (though I just did tls management for vitess and didn't write any workloads on top of it, so my impression might be a bit off)
If you have serializable on each shard and you chop transactions so you have no cross shard transactions, the application is technically still serializable, it just behaves like two independent deployments of the application for different shards
Sam, it feels like you're just playing games. It's hard to take anything you say or do seriously. Are you going to OSS this thing? Or have you decided to backpedal?
Congrats on the checks notes closed launch, I guess?
Open source being by far the more important of the two, since it means the user is more insulated by terrible corporate decisions thus making it more suitable for long term planning.
Commerical open source is fine. It's nobody else's fault if you raised too much to make it work sustainably.
> When we are ready we will release Neki as an open source project suitable for running the most demanding Postgres workloads.
Has this changed?
https://web.archive.org/web/20260516145550/https://planetsca...
https://web.archive.org/web/20260910204345/https://planetsca...
Sad to see this coming from a company that was itself built on the foundation of an open-source project like Vitess.
We maintain Vitess using our company funds. Why are you going after a small company for defending itself against companies like Amazon but not criticizing the reason we have to do this.
I would love to open source Neki and I have not said we won't.
Unlike you, probably, if we're slinging unfounded accusations.
Their CEO always immediately comes in and makes a fool out of himself. He is what you get when you mix narcissism and being a midwit.
A remarkable own goal given that if they'd literally held their tongue, I'd be at worst indifferent.
I want to let you know that I have customers that pay PlanetScale millions of dollars and have told me they came to the product because of my Twitter account. So you will never make me feel bad about this.
Just launch the product and let it speak for itself. The constant need to take shots at competitors comes across as pretty childish.
And now planetscale have made a Vitess like equivalent for Postgres which they have made proprietary
Google is big enough to survive open-sourcing Vitess, k8s, angular, etc. Planetscale isn't.
That Youtube built. I know at the time Youtube was already part of Google but I don't think of Vitess as a Google Stack.
I wonder how the performance will be, but I can just rewrite everything in Rust.
They claim to provide unlimited IOPS (I/O operations Per Second).
I would like to get some of these drives myself. Sounds like a technical miracle.
On the other hand claiming such miracles makes me wonder which other aspects in this are are actually a working miracle or rather a mythos/false marketing claim.
I think this is marketroid for “we don’t limit them” not a statement about physics
For example: point 08 says "Assign different tables or workloads to different shard groups" and point 02 says "Split hot shards as workloads grow". How do those interact ? Can a single table be split across multiple shards ? If so don't you need 2pc to enforce primary key constraints ?
The intro pages just read as AI slop.
Selfish doubts aside, congrats to Planetscale on the launch!
- PgDog is open source
- It runs anywhere, not just your cloud
- It's built for Postgres, not a MySQL pivot
Found this from neki.dev.
Apparently not.
However we have a whole team working to improve Postgres and we always prefer to contribute those efforts to the Postgres community.
> This adds some undesirable side effects, however. First, the coordinator becomes the bottleneck.
Apart from being able to use any node as a coordinator (and you can load balance them to avoid having "multiple connection strings), there's a new pattern which effectively allows you to have as many coordinators as you want. They are effectively "data-less" nodes. We have devised and implemented this pattern in StackGres [1].
> Adding a shard with more resources for a noisy tenant, or many small shards for a wide shard space requires substantial manual configuration of not only the database servers themselves, but wiring them up together with Citus.
Adding nodes (infrastructure) is what operators solve. In StackGres, adding new nodes means editing one/two characters from your YAML file: the integer number that represents the number of workers that you have.
Shard rebalancing is fully built-into Citus as a UDF, which you can call (manually or in an automated manner) over Postgres protocol.
> Managing backups is also external to Citus, so operators still need to build the proper infrastructure
Agreed, but it's also solved (see distributed backups in StackGres [2]).
> PgDog is a spiritual successor to PgCat, both of which improve on Citus's architecture substantially.
Unsubstantiated why. I assume it's because of the assumption that a proxy model is superior than Citus. To which I have to say that Citus model is also a proxy model, where the proxy just happens to be Postgres, which unsurprisingly, speaks Postgres protocol. Sure, there are nuances that we could debate in this area and we can say that Citus is not a "pure proxy", but that doesn't lead to concluding that a proxy model is better --it's arguably not.
[1]: https://stackgres.io/doc/latest/administration/sharded-clust...
[2]: https://stackgres.io/doc/latest/administration/sharded-clust...
I am looking for interdimensional and interuniversal scale. Which of you trust fund babies has a startup which is working on this problem?
Vitess today is not the Vitess of 2021. About 70% of the current codebase was written in the last five years, based on lessons we learned running it at scale.
Neki is built for Postgres from the ground up, by the engineers who did that work.
On the Multigres comparison, I'll admit we get a bit salty, mostly because "Vitess for Postgres" invites the comparison. But to date there's no public evidence of a working sharding implementation in Multigres. Their sharding design doc landed this week, and it lists cross-shard query planning and resharding execution as explicitly out of scope.
Neki does sharding today, including online resharding that switches traffic without a maintenance window. It isn't open source, but you can spin up a cluster and try it yourself.
No one accused you of dishonesty.
PS - Just some advice that I learned in sales 101 early in my career...focusing on shit talking the competition generally isn't a strategy that works well.
"You're not wrong Walter, you're just an asshole." - The Dude
Only petty people can hold that view.
(I'm a paying PS customer)
Never heard of the guy before today, but his dickish, edge-lord aura (on X and in this thread) makes me not want to meet him. If I had to choose between buying from him, or some other random dude, I'll hear out the random dude first.
If you have such luxuries, I bet you don't have enough scale to be really profitable for him.
I can think of at least 4 other ways of scaling horizontally without switching DB engines. I'd have to be mightily constrained to have this particular product as the only option.
Based on what I've read so far, with his shit-talking the competition, he seems like... a bit of an asshole.
They build good stuff and care.
The pitch is compelling. I wonder how many teams will be able to operate sharded database setups in production as a result of this.