MapReduce is a useful tool in some contexts, but the the hype surrounding it always felt like people patting themselves on the back for re-inventing relational algebra.
Considering that oracle is not in fact magic, this meant that a large number of firms were spending 7-8 figures annually on oracle licenses. Map reduce/Hadoop was the first accepted alternative that didn’t involve spending outrageous license fees and instead involved outrageous hardware expenditure.
Con: OSS is less optimized than proprietary solution, requiring bigger hardware
Pro: OSS allows you to buy bigger hardware, use all of it without logical restrictions, and scale infinitely beyond the arbitrary point you were locked into with licensing.
And then the new-found efficiency frees up time to discover/identify $(x,)xxx,xxx+ in manual work that can also now be done with your new-found compute...
Wow. Way to prevent us from progressing beyond the industrial revolution.
($catchup_speed++)
It’s incredibly simple for the end user conceptually but encapsulates optimizing processing across a distributed file system, fault tolerance, shuffling key value pairs, job stage planning, handling intermediates ect.
Hadoop a big data framework that reduces the level of competence required to write data pipelines because it was able to hide a massive amount of complexity behind the map reduce abstraction.
Id even argue that hive, snowflake, and other sql data warehouses have taken this idea further, where most sql primitives can be implemented as map reduce derivatives. With this next level of abstraction, dbas and non-engineers are witting map reduce computations.
I think my point is that abstractions like map reduce have had a democratizing effect on who can implement high scale data processing and their value is that they took something incredibly complex and made it simple.
I don't know what I should be proud of when I learn something new, it all seems extremely basic compared as soon as I learn it.
It’s pretty easy to simplify things down until they sound unimpressive.
However, in most other cases there are now far better alternatives (although tbh I'm not sure how many were around when MapReduce was introduced).
The main limitation around mapreduce is the barries imposed by the shuffle stage and after the end of the reduce if chaining together multiple mapreduce operations. Dataflow frameworks remove these barriers to various degrees, which often lowers latency and can improve resource utilization.
It dramatically reduced the cost of entry for many ad-tech applications.
You say that as if it was unequivocally a point in its favour.