Hacker News Reader

Comment by tacker2000 | Hacker News Reader

tacker2000·

Is Debezium the only good CDC tool out there? I have a fairly simple data stack and am looking at integrating a CDC solution but I really dont want to touch Kafka just for this. Are there any easier alternatives?

View on HN

You don't need Kafka in order to run Debezium. While that certainly is the most common deployment, you can use Debezium also with things like Kinesis, Google Cloud Pub/Sub, or even plain HTTP sinks, all via Debezium Server (https://debezium.io/documentation/reference/stable/operation...). You even can embed it as a library into your application for the highest degree of flexibility.

That said, streaming events into Kafka (or something like it), rather than pushing change events directly to destinations like Snowflake has big advantages when it comes to reprocessing data, setting up additional consumers, etc.

Redpanda Connect is a yaml file that has a cdc input and you can process the data and send it anywhere, no Kafka required.

I’ve seen postgres cdc directly being written to Snowflake.

https://docs.redpanda.com/redpanda-connect/components/inputs...

Conduit.io is where it's at. FAR more source and destination connectors, easier to deploy etc... Pair it with NATS to replace Kafka's mess

AWS has DMS with a serverless option which is great for CDC

Except when a new column is added upstream and DMS inserts the new column not at the end but before the dms timestamp fields, ruining the index based loads of DMS files

Reply on news.ycombinator.com