I wonder if materialize.io could handle such workloads at this stage.
It's early days, but I can confirm that it's a bit hard to know out of the gates whether Materialize can handle such workloads. In particular, unless I missed it the query workload isn't really discussed in the post, and that is all that matters.
Druid makes a few compromises that Materialize isn't willing to make. For example AFAIK Druid doesn't support deletions or modifications in "realtime", which means they can track "min/max" style queries much more efficiently, but they lose out on some other features (connecting pipelines of these views where you might have to "retract" a prior min or max). This could certainly let it scale more, but also means that you learn a few months down the road that it doesn't do everything you hoped it would.
No joins in Druid is also reportedly a bit of a pain. Instead, you get to pre-denormalize your data. No need with Materialize (and hey, feel free to push down reductions through the joins, rather than denormalize and then reduce).
Short version, Materialize is definitely a "higher sophistication" data processing play. Whether that works out well remains to be seen! I'll slap up a blog post later today with a worked example.
Netflix's workload would likely exhaust the resources of even a vertically-scaled single node.
Because of that, you don’t really need to scale Materialize in a sharding sense. You can just have a bunch of “the same” Materialize node (i.e. every node just freestanding clone of a template node, with exactly the same sources and matviews) and then hit them with the parts of a map-reduce query launched by, say, Citus—where Citus was thinking it was talking to a bunch of Citus shard nodes each holding a table-shard named X, but was actually talking to Materialize nodes each holding a matview named X. As long as the query sent from the map-reduce job to each node is constrained in its WHERE clause to only the part of the data it expects to get from that node—rather than relying on the node to know what data it has—then the Materialize nodes would each just do the work required to supply that data (including only pulling in the parts of the configured sources required to compute that result.)
That’s just my intuition from how Materialize presents itself as PG-wire-protocol compatible, though; I haven’t tried this myself, and there might be some footguns in the path of anyone really trying to implement it.
And, of course, this is all irrelevant the moment you write a query that needs a pure reduce (e.g. the computation of a current finite-state-machine state over an event-stream source) rather than a map-reduce. Druid/Clickhouse/etc. can probably “scale” those, in at least the Hadoop “move the job around between each serial stage, so each stage has data-locality for the data of that stage” sense; while Materialize would give you no benefit at all in such a job over just querying a plain PG view defined on top of a Foreign Data Wrapper source.
This is absolutely correct!
> You can just have a bunch of “the same” Materialize node (i.e. every node just freestanding clone of a template node, with exactly the same sources and matviews) and then hit them with the parts of a map-reduce query
This should work, but we have been thinking about it/testing it differently internally. In general you should be able to create materialized views on different "shards" that have different `where` conditions, allowing you to control memory that way. This technique does require data that is actually partitionable in this way, same as it must be partitionable in mapreduce.
> this is all irrelevant the moment you write a query that needs a pure reduce
Of course, with materialize's sinks you can spin up a bunch of `materialized`s and connect them for a final reduce after data has gone through e.g. kafka or shared files. Being able to write joins and aggregates across heterogenous sources makes this kind of workload actually pretty pleasant.
For now. We have a pretty good idea of what needs to be done to shed state to disk, and have designed to be able to implement it. We expect it to "just" be a matter of putting in the engineering effort.