Hacker News Reader

Comment by odammit | Hacker News Reader

Looks cool. Seems like a good alternative if you can’t use listen/notify for some reason.

I’ve used listen/notify with triggers for a while with good success. The flexibility of my apps controlling what they want to listen for instead of having that config elsewhere.

Downside to l/n is that if you aren’t listening you aren’t getting changes. Seems like the same is true if the webhook fails here since there isn’t a retry.

If you want to check out another cool alternative see Debezium[1]. It’s uses PGs logical decoding feature to stream changes. Supports protobufs and JSON output.

http://debezium.io/docs/connectors/postgresql/

View on HN

I did use Debezium in a Kubernetes cluster to stream Postgres changes into Kafka and then through a custom application that can send them to clients via HTTP or WebSockets. But the entire system had multiple points of failure. So we then decided to write our own thing that connects to the LR slot and sends over WebSockets. Working on open sourcing that too!

can you talk more about the points of failure? and is there a place I could sign up for more information when you open source your websocket solution?

Hi. Sorry for the late reply.

Well the system had Postgres -> Debezium -> Kafka -> CustomApp -> Client. If data stopped flowing through the system, I would have to examine each component to figure the issue out. They'd mostly turn out to be Kubernetes network issues such as Kafka brokers not being able to talk to each other, or Debezium not being able to contact Kafka. This was too painful to run when all we wanted was notifications of changes on Postgres.

If you drop me a note with your email, I'll make sure to get in touch with you when our solution using Postgres LR exposed over WebSockets (and more) is out :)

I remember this old article about Debezium's now-unmaintained predecessor, Bottled Water. Its diagrams may still be useful for those wanting to understand how this might be used.

https://www.confluent.io/blog/bottled-water-real-time-integr...

Unfortunately, most of these LR programs don't work in AWS (RDS) land, since they require PG plugins, and RDS LR only supports the built-in test decoder.

I wrote a similar program that uses LR to stream changes to AWS's Kinesis[1], using a mini library that parses the output of the default test decoder[2] as a result, in case anyone else is in RDS and has the same limitation.

[1] https://github.com/nickelser/pg_kinesis [2] https://github.com/nickelser/parselogical

IIRC Debezium is RDS compatible since version 0.7.0

That's awesome -- wasn't aware that was the case (wasn't many moons ago :). Thanks for the heads up! But, we don't use Kafka internally so it's sadly a non-option still...

RDS supports wal2json.

Reply on news.ycombinator.com