MapReduce is making a comeback

First let me say that I think Timely Dataflow and Materialize are both super cool. The two approaches are quite different, in part because they solve slightly different problems. Or maybe it's more fair to say that they think of the world in somewhat different ways. Probably most of the differences can be traced back to how Timely Dataflow relies on the expiration of timestamps in order to coordinate updates to its results. You can read the details on that in their docs (https://timelydataflow.github.io/timely-dataflow/chapter_5/c...).

I think a reasonable TLDR might be to say that continuous map reduce has a better fault-tolerance story, while timely dataflow is more efficient for things like reactive joins. They both have their purpose, though, and I imagine that both Flow and Materialize will go on to co-exists as successful products.

> $buzzword

Yeah, the article is an odd spin on what they're building.

> min/max/average as you add new data

Or a 2D kernel density estimate for your dashboards, a real-time view of 3-neighbors in a graph (nodes+edges definition) sized by log1p(request frequency), .... I find it way easier to write a few custom incremental primitives to piece together into that kind of algorithm than to write such an algorithm from scratch.

I'm not crazy about a general-purpose framework/product that tries to allow incremental updates of AllTheThings™ -- my experience thus far suggests that getting it to do what you want (or perform reasonably) on your own data will require enough kludges that you would have been far better off writing the WholeDamnedThing™ yourself.

If they do only support min/max/average and other simple transforms then that's probably not great; they'd be competing directly with something like QuestDB, which is a phenomenal product I'm leaning toward more and more. You don't need millisecond view update times if you can query the whole db in milliseconds.

Real-time aggregates is the core value pitch of Clickhouse.

A big difference is the removal of windowing: Flink lets you aggregate or join events only so long as they arrive in the same temporally-bound window. You're required to have a window, and it's core to the semantics of your workflow.

Flow's model doesn't use windows, and allows for long-distance (in time) joins and aggregations. There's no concept of "late" data in Flow: it just keeps on updating the desired aggregate.

dataflow things like Flink (or even better differential datatflow [0]) are far more flexible and subsume map-reduce. This article feels like hyping up the durability of the Model T.

[0] https://github.com/TimelyDataflow/differential-dataflow

IANAE on Flink, especially when it comes to the internals. But I think that the decomposition of computations into distinct map and reduce functions seems to afford a bit more flexibility, since it can be useful to apply reductions separately from map functions, and vice versa. For example, you could roll up updates to entities over time just with a reduce function, and you could easily do so eagerly (when the data is ingested) or lazily (when the data is materialized into an external system). That type of flexibility is important when you want a realtime data platform that needs to serve a broad range of use cases.

The big hype was always due to the ability to shed large oracle based data warehouses. When Hadoop was full of hype in 2010 Oracle was charging 150k/cpu core/year for a rac cluster license.

Considering that oracle is not in fact magic, this meant that a large number of firms were spending 7-8 figures annually on oracle licenses. Map reduce/Hadoop was the first accepted alternative that didn’t involve spending outrageous license fees and instead involved outrageous hardware expenditure.

My interpretation is that is why it’s so brilliant.

It’s incredibly simple for the end user conceptually but encapsulates optimizing processing across a distributed file system, fault tolerance, shuffling key value pairs, job stage planning, handling intermediates ect.

Hadoop a big data framework that reduces the level of competence required to write data pipelines because it was able to hide a massive amount of complexity behind the map reduce abstraction.

Id even argue that hive, snowflake, and other sql data warehouses have taken this idea further, where most sql primitives can be implemented as map reduce derivatives. With this next level of abstraction, dbas and non-engineers are witting map reduce computations.

I think my point is that abstractions like map reduce have had a democratizing effect on who can implement high scale data processing and their value is that they took something incredibly complex and made it simple.

Isn't the entire field of software just people patting themselves on the back for some extremely basic relational algebra?

I don't know what I should be proud of when I learn something new, it all seems extremely basic compared as soon as I learn it.

It was more about scaling than about relational algebra though.

Apart from Google, which has a patent related to their 2004 paper, I don't know how much people are trying to "take credit" for map-reduce. I'm certainly not. But I do think the approach of running map-reduce continuously in realtime is interesting and worth sharing. And I hope that some folks will be interested enough to try it out, either with Flow or in a system of their own design, and report on how it goes for them.

Versioning is indeed an issue, but that's the case for anything with long-lived state. Our current rely on JSON schemas, TypeScript, and built-in testing support to help ensure compatibility. Those things actually help quite a bit in practice. But I think we may also want to build some more powerful features for managing versions of datasets, since there's a real need there, regardless of the processing model you use to derive the data.

Yes, but when talking about MapReduce we generally talk about distributed frameworks for doing it on big data.

But most people don't have "big data" in the sense of having data that requires more than a single machine to process.

Most people who think they have "big data" still don't have big data (e.g. I've done work on datasets where people insisted on using "big data" solutions when it could all easily fit in a Postgres instance with or without a columnar store with most of the working set cached in memory for a fraction of the cost).

It "went away" in the sense that more people realised they could avoid it with a few simple steps (e.g. pre-processing during ingestion), and/or fit the data they needed on fast-growing individual servers, and so the number of people continuing to use it more closer approximated the set of people who actually work on big data.

For those who actually needed it, it of course never went away.

I think this is referring to a specific paradigm within Apache Hadoop but I'm no expert.

But they say ‘MapReduce’, not ‘map reduce’. They’re talking about the specific model, not the general idea of map reduce.

Really, they should be calling it "Distributed MapShuffleReduce", I let Sanjay and Jeff know this (in person) but they didn't really seem to care. Neither map nor reduce is a method of parallel processing, the shuffle stage is.