Hacker News Reader

What about the other big problem ignored here: does your streaming platform separate compute and storage?

Because GCP DataFlow does. Flink doesn't. DataFlow allows you to elastically scale the compute you need (Snowflake, Databricks). If you can't do that, materialized views will be a more niche feature for bigger 24x7 deployments with predictable workflows.

View on HN

As George points out above, we haven’t added our native persistence layer yet. Consistency guarantees are something we care a lot, so for many scenarios, we leverage the upstream datastore (often Kafka).

But to answer your question, yes, our intention is to support separate cloud-native storage layers.

My dim and distant recollection is that Beam and/or GCP Data Flow require someone to implement PCollections and PTransforms to get the benefit of that magic. That's not a trivial exercise, compared to writing SQL.

Reply on news.ycombinator.com