Writing a Postgres logical replication system in Golang | Hacker News Reader

Writing a Postgres logical replication system in Golang

dolthub.com

91 pointsingve19 comments

If anyone from DoltHub team is reading this, please consider fixing [0] the RSS/Atom feed link [1]. I love your blog posts, but I am not able to subscribe to them in my feed reader. Your posts on SwissTable and Prolly Trees are my fav ones.

[0] - https://validator.w3.org/feed/check.cgi?url=https%3A%2F%2Fww...

[1] - https://www.dolthub.com/blog/rss.xml

Also the publication date of this article shows March 8, but we are on March 7 :)

We used to have a feature on the blog that would hide articles until their publication date even if we submitted them a bit early, but it was kind of buggy so we got rid of it. You guys weren't supposed to find the article until tomorrow, we never know what you're going to decide to send to the front page.

Thank you for drawing our attention to this! Our blog content surpassed the RSS limit awhile ago, we are working on paginating the feed so new content fits.

We just updated our RSS feed to only show the most recent 50 blogs so that it fits within the W3C validator size limits. The feed of all blogs is still available at https://www.dolthub.com/blog/rss-all.xml

I can confirm it works now! Thank you so much :)

If you're looking for something that's simple to operate (but not quite as robust yet), check out WalEx: https://github.com/cpursley/walex

> the world's first version-controlled SQL database

Isn't that what others like Supabase are already offering or am I missing a distinction here?

Whenever you read "first" just add "with the special meaning of A,B,C" in your head. There was the first person on the moon, or the first person to run 100m below 10s, but for mostly everything else "first" is basically never binary.

Pretty sure dolt actually is the first SQL db with branch, merge, and diff implemented natively.

Supabase offers version controlled database? To be clear, what dolt does is enabling you to branch like git and then merge your branch, rollback old "commits" of your database and similar stuff. It is much more advanced than audit logging

How does DoltgreSQL compare to Neon's branching?

In neon you can create different branches but there is no merge functionality. I think merge functionality is very hard to implement anyway. not sure if it is worth the effort.

To have the same state on two branches you would need to run the same queries that you ran on the other branch.

> not sure if it is worth the effort.

Exactly. Teams that are mature enough to use data branching in development likely already have a schema migration tool that handles the merge safely with all the necessary controls.

And how about those teams that are starting out and don't want to build all of that? It's worth the effort given that it's a common problem that teams need to solve over and over.

One of the biggest challenges for us has been getting through people's heads that "version controlled database" doesn't mean "schema migrations with extra steps". Some people get it immediately, but in general there's a real failure of imagination about what kind of use cases this technology enables.

Our customers are building workflows that are just impossible to achieve with other databases. There are a few common use cases that come up over and over:

* Game development, to version control game configuration data being edited by an entire team of developers at once * CMS-like tools, including online storefronts, so you can have pull request workflows and easy rollback * Collaborative data sourcing, where multiple independent sources get merged together into a single database

There's also the distributed use case, where people fork and clone each other's data sets, that we haven't really seen take off yet. Or maybe those people are just unlikely to buy hosting services or file support tickets.

Neon is basically a copy on write postgres backend. So you can create a branch at near zero cost and start making changes, but there's no way to merge those changes back to the original branch.

To do that, you need a storage system that can efficiently compute a diff to do a three-way merge. That's what dolt does. It means you can diff any two commits in the database with cost proportional to the size of the diff, not the size of the data. And it uses a commit graph just like git which enables it to find common ancestors for a three-way merge algorithm, again just like git.

March 08, 2024.

News from the future? Who cares about causality?

If it could replicate your database transaction before they've actually happened, that would be really handy.

Reply on news.ycombinator.com