Hacker News Reader

Comment by netcraft | Hacker News Reader

What I would love is a turnkey service I could set up with logical replication and a straightforward configuration that can listen for any changes on certain tables (ideally with optional filters) that will take those changes and put them in a queue / ESB / whatever. If youre designing a new application from scratch and you have only one place that certain logical changes happen you can do this from the start. But in any system not designed that way from the start, the database is the place where all things go. Using that as your event source allows everything to keep on working like they always have but allow you to now notify new processes when something changes.

View on HN

If you're using Elixir, check out https://github.com/cpursley/walex

I've actually been thinking about turning this idea into a product where you can just point it at your postgres database and select the tables you want to listen to (with filters, like you describe). And have that forwarded to a webhook (with possibility of other protocols like websockets).

I'd love to hear folks thoughts on that (and if it would be something people would pay for). And if anyone might want to partner up on this.

Yo :D This is what Supabase Realtime does!

https://github.com/supabase/realtime

Spin up a Supabase database and then subscribe to changes with WebSockets.

You can play with it here once you have a db: https://realtime.supabase.com/inspector/new

Yeah, Supabase is awesome. In fact, WalEx originally started out as stolen code from realtime so I could do stuff right in Elixir (biz logic).

What advantages does this have compared to Debezium?

You wouldn't have to host it yourself (not everyone knows how or wants to run a heavy Java app).

Add your database string, a couple extra migrations, your webhook endpoint - and you're off to the races.

Target customer would be low-code space (think admin dashboards on postgres, webhook integration tools).

I used to work at Yelp, which had something that I think it similar to what you are describing called Data Pipeline (https://engineeringblog.yelp.com/2019/12/cassandra-source-co...).

I remember it being pretty simple (like, run one or two bash commands) to get a source table streamed into a kafka topic, or get a kafka topic streamed into a sink datastore (S3, mysql, cassandra, redshift, etc). Kafka topics can also be filtered/transformed pretty easily.

E.g. in https://engineeringblog.yelp.com/2021/04/powering-messaging-... they run `datapipe datalake add-connection --namespace main --source message_enabledness`, which results in the `message_enabledness` table being streamed into a (daily?) parquet snapshot in S3, registered in AWS Glue.

It is open source but it's more of the "look at how we did this" open source VS the "it would be easy to stick this into your infra and use it" kind of open source :(

Something like Debezium? https://debezium.io

- https://hevodata.com/

- https://airbyte.com/

- https://www.stitchdata.com/

Unfortunately none of them are perfect as your data scales, each have up and downsides...

Reply on news.ycombinator.com