Hacker News Reader

Comment by cbdumas | Hacker News Reader

I'm not sure if it's a mistake or just some cheeky humor but the "Known Limitations" link on their Docs page returns a 404.

View on HN

It looks like a caching bug somewhere in the web pipeline. It isn't running on Materialize, sadly.

There are a pile of known limitations, and our goal is to be 100% clear about them.

Hi Frank! Congrats on the progress - I remember listening to you describing Materialized at SIGMOD'19 in the big hall - it sounded like magic back then, glad to see it come true :) I have a question now - can you subscribe to view updates somehow, instead of querying them on an ad-hoc basis?

Yes, the term we're using is "Sink", dual to a Source.

    https://materialize.io/docs/sql/create-sink/

At the moment you can create Kafka sinks, and we'll write the change log out to that topic. They are still baking though, and need some hardening for failure cases.

Thanks. What happens when a DDL is run on the source table (add or remove column for instance) ? Does it break the entire flow?

DDLs that alter schemas definitely cause things to catch on fire at the moment (Debezium writes them to a side-topic, which isn't obviously synchronized with the data topics). In principle `CREATE` and `DROP` statements are things we could pass through, but some work needs to get done to pull the right information out of Debezium at the right moments (and drop the right prefixes of the existing Kafka topics if you re-create a table with the same name).

To add a bit more context to Frank's answer: adding a column to a table will work just fine today, but the data in that column won't propagate through to Materialize. You'll continue to see the latest inserts, updates, and deletes, though. The new column can be incorporated by re-creating the source, though this does require re-reading the entire table from the beginning at the moment.

Removing a nullable column will also work fine along the same lines: that column will simply be filled in with NULL in Materialize for all new data. Removing a non-nullable column is backwards-incompatible, of course, and so Materialize will be unable to process any additional events for that topic, and so you'd have to recreate the source and restart from the beginning.

> Debezium writes them to a side-topic, which isn't obviously synchronized with the data topics

That's unfortunate. It seems like it would be ideal to feed schema changes through the same topic, so that the ordering is defined without the usual distributed-system hilarity.

Agreed! We plan to contribute to the Debezium project over the next months along these lines. We’ll have more to say on this shortly!

Nice, looking forward to your contributions. Let me know if you'd like to discuss anything upfront. On a general note, have you considered to rely on the schema of change events themselves (inline JSON schema, or Avro schema in a registry)? Not all connectors provide the separate schema history topic, you won't have to parse DDL and the synchronization issue is avoided at all. Happy to discuss in more detail.

Disclaimer: I'm the Debezium lead.

Reply on news.ycombinator.com