Hacker News Reader

Comment by polskibus | Hacker News Reader

Thanks. What happens when a DDL is run on the source table (add or remove column for instance) ? Does it break the entire flow?

View on HN

DDLs that alter schemas definitely cause things to catch on fire at the moment (Debezium writes them to a side-topic, which isn't obviously synchronized with the data topics). In principle `CREATE` and `DROP` statements are things we could pass through, but some work needs to get done to pull the right information out of Debezium at the right moments (and drop the right prefixes of the existing Kafka topics if you re-create a table with the same name).

To add a bit more context to Frank's answer: adding a column to a table will work just fine today, but the data in that column won't propagate through to Materialize. You'll continue to see the latest inserts, updates, and deletes, though. The new column can be incorporated by re-creating the source, though this does require re-reading the entire table from the beginning at the moment.

Removing a nullable column will also work fine along the same lines: that column will simply be filled in with NULL in Materialize for all new data. Removing a non-nullable column is backwards-incompatible, of course, and so Materialize will be unable to process any additional events for that topic, and so you'd have to recreate the source and restart from the beginning.

> Debezium writes them to a side-topic, which isn't obviously synchronized with the data topics

That's unfortunate. It seems like it would be ideal to feed schema changes through the same topic, so that the ordering is defined without the usual distributed-system hilarity.

Agreed! We plan to contribute to the Debezium project over the next months along these lines. We’ll have more to say on this shortly!

Nice, looking forward to your contributions. Let me know if you'd like to discuss anything upfront. On a general note, have you considered to rely on the schema of change events themselves (inline JSON schema, or Avro schema in a registry)? Not all connectors provide the separate schema history topic, you won't have to parse DDL and the synchronization issue is avoided at all. Happy to discuss in more detail.

Disclaimer: I'm the Debezium lead.

Reply on news.ycombinator.com