Hacker News Reader

Comment by hinkley | Hacker News Reader

> Logical replication allows the real-time streaming of changes from a database into any system that can understand the PostgreSQL logical replication protocol.

My Christmas wish for 2024-2025 is that we have an industry standard format for replication. A simple transform (hopefully) of either WAL or logical replication, that we can feed into backup software, search, analytics, whatever.

There's something out there that fulfills the duties of Kafka and half a dozen other tools (at least two of which don't exist yet due to significant barriers to entry) with about a quarter of the code. We just need to carve it out of the block of marble.

View on HN

This would be amazing. Anybody who plugs into the PostgreSQL replication protocol is on my watchlist. I’ve been playing with https://github.com/turso-extended/pg_turso which lets you replicate PostgreSQL tables into SQLite on fly.io using Turso.

I’m still predicting we’re getting a Kafka clone in Rust at some point. No idea who or where from but I’m predicting/hoping.

The world is moving to events. I’d be surprised if 2024 was “the year of events” but I’m sure it’ll be that year sometime in the next 5 years or so.

I want to speak WAL everywhere I go. Let me talk WAL to my vector database, and my OLAP, I want to talk WAL to NoSQL, so what if the schema isn’t enforced, I still want a logical representation of it.

Also, sidenote although pg_turso is experimental I added text, varchar, uuid, timestamp, timestamptz, json and jsonb to the supported types ~1 month ago for anyone who saw it in the past but that was holding them back, don’t let it be what stops you from hacking on it.

I tested it on the chinook music db, I was able to fully replicate a table, although I admit it’s barely functional as a cache because it doesn’t handle relationships or primary/secondary keys and I don’t think it does anything with unique constraints either, but with that said it is functional. It will take a table from Postgres and put it into every region supported by Fly.io and with embedded replicas you can just push it through onto the users device if you want. I jumped out of my chair and was laughing like an idiot the first time I got it to stream the entire chinook music DB from a Postgres creation script straight into SQLite all over the world, all while running postgresql on localhost inside Docker. I can taste the future. XD

Do you mean debezium? It’s relatively easy to set that up, what is not trivial is translating data types between various databases.

Reply on news.ycombinator.com