Hacker News Reader

Comment by sradman | Hacker News Reader

Materialize sounds like it has reinvented Michael Stonebraker's StreamSQL [1] and SAP's Continuous Computation Language (CCL) [2] which was created as part of a StreamSQL competitor named Coral8 and lives on in an enterprise product now named SAP HANA Smart Data Streaming. This space has gone by many names, Streaming SQL, Complex Event Processing (CEP), and Event Streaming.

I think the Continuous Computation Language (CCL) name captures the essence of these systems: data flows through the computation/query.

These systems have always had promise but none have found anything but niche adoption. The two most popular use cases seem to be ETL-like dataflows and OLAP style Window queries incrementally updated with streaming data (e.g. computations over stock tick data joined with multiple data sources).

[1] https://en.wikipedia.org/wiki/StreamSQL

[2] https://help.sap.com/doc/PRODUCTION/e1b391d2a3f3439fbab27ed8...

View on HN

The projects you've mentioned are attempts to address stream processing needs with a SQL-like language. That is fundamentally different from providing incremental view maintenance of actual SQL using streaming techniques (what Materialize does).

If you want to maintain the results of a SQL query with a correlated subquery, StreamSQL in Aurora did not do that (full query decorrelation is relatively recent, afaiu). I have no idea what TIBCO's current implementation does.

If you want to maintain the results of a SQL query containing a WITH RECURSIVE fragment, you can do this in differential dataflow today (and in time, Materialize). I'm pretty sure you have no chance of doing this in StreamSQL or CCL or CQL or BEAM or ...

The important difference is that lots of people do actually want to maintain their SQL queries, and are not satisfied with "SQL inspired" languages that are insert-only (Aurora), or require windows on joins, or only cover the easy cases.

> The projects you've mentioned are attempts to address stream processing needs with a SQL-like language. That is fundamentally different from providing incremental view maintenance of actual SQL using streaming techniques (what Materialize does).

With all due respect, CREATE SINK and CREATE SOURCE are SQL-like. I would argue that the pipeline created from the set of SINKs and SOURCEs is the key concept to grasp for developers new to your platform. The purity of the PostgreSQL CREATE MATERIALIZED VIEW syntax and other PG/SQL constructs seems like a minor selling point, in my (very narrowly informed) opinion. I hope I'm wrong.

Our difference of opinion involves marketing language and perceived differentiators. There are some important use cases for continuous SQL queries over Kafka-like data streams that remain unaddressed (as far as I know). I hope Materialize gains traction where others have failed to do so. If PG/SQL compatibility was the only thing holding back this style of solution then kudos to you and your team for recognizing it. Good luck (honestly).

Reply on news.ycombinator.com