Hacker News Reader

Comment by meritt | Hacker News Reader

Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing. I feel like this is a bit overboard. And this is before we talk about the non-Apache stream-processing frameworks out there.

* Apache Flink is an open source stream processing framework

* Apache Flume is a distributed, reliable, and available software for efficiently collecting, aggregating, and moving large amounts of log data.

* Apache Storm is a distributed stream processing computation framework

* Apache Samza is an open-source near-realtime, asynchronous computational framework for stream processing

* Apache Spark is an open-source distributed general-purpose cluster-computing framework.

* Apache Apex is a YARN-native platform that unifies stream and batch processing.

* Apache Kafka is an open-source stream-processing software platform

View on HN

> Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing.

Well, no, you went too far.

Kafka is primarily used for communication & data transport, by most people (can be used in other ways, and it has the KafkaStreams library that enables you do to some computation on said data - but it is, primarily, a transport & communication mechanism; also maybe storage, if you squint right)

Spark and Flink might be similar on first sight, but if you look a bit closer you realize Spark is primarily geared towards batch workloads, and Flink towards realtime. Sure, you can do micro-batch in Spark and pretend that's realtime stream processing, but the focus of it is fairly clear - as is the focus of Flink. So both have legitimate rights to exist.

I'm not sure about the others, haven't used them. There may be indeed consistent overlap - but I'm sure they are different approaches. What's wrong with that?

On the contrary, I didn't go far enough. I didn't talk about Apache Gearpump, NiFi, Beam, Ignite, or Trident.

I know there are subtle differences in each specific technology, and that's probably the same justifications used to support the thesis when building yet another very similar framework.

> What's wrong with that?

I believe it drastically reduces adoption of these tools because many of us avoid what appears to be bandwagon technologies as we don't want to consciously add layers of future technical debt when a majority of these projects will be abandoned.

In the case of Spark and Flink, I wouldn't say that batch processing versus realtime stream processing are "subtle differences". That's akin to arguing that relational databases vs. document stores vs. timeseries databases just "muddy the waters".

Hacker News and Reddit have a lot of interesting discussion. But the audience skews toward client-side webdev, and students or younger developers. An audience accustomed to libraries and frameworks that you can reason about with fairly low learning curve, and spin up in a Codepen to see visually right away.

Heavy-lifting server side tools, especially those who only earn their keep at scale, are a different beast. And that's OKAY. Quite frankly, if you're "not sure" whether you need a stream processing platform in your architecture, then YOU DON'T. Aside from some consultants and salespeople, no one's really going to push you toward adoption of this stuff.

In the overwhelming majority of use cases, what you need is a tiny microservice (in your language of choice). Which reads from a Kafka or Rabbit topic, and stores state in your cache system of choice. By the time you reach the scale where that's not suitable, your organization probably won't need a web forum thread to educate you on what the vendor landscape looks like.

To be a bit more future-proof you should give Apache Beam a try. The same code should (theoretically) work with any of the supported runners[0] and so you could deploy it on top of the most suitable framework/technology for your specific workload. Moreover, at this point, the community has several examples of how to add an additional runner.

Edit: I didn't see you mentioned Beam in your second salvo :)

[0]: https://beam.apache.org/documentation/runners/capability-mat...

>I believe it drastically reduces adoption of these tools because many of us avoid what appears to be bandwagon technologies as we don't want to consciously add layers of future technical debt when a majority of these projects will be abandoned.

Doesn't seem to be impacting adoption of the major ones from what I can see (Spark, Kafka, Flink). Assuming you're at a scale where you need these technologies. Which most companies are not so it's a good thing they don't implement them just for kicks or as "future proofing." If you have a business problem that the main frameworks can't solve then trying out some of the other ones may be worth the cost. There's also ongoing support for older frameworks (Storm) so it's not like your code becomes useless.

The approach open source is taking to resolve this issue is to embrace the diversity but create unified APIs on top of it. Apache Beam and Arrow for defining data workflows and data exchange formats respectively. There's also always SQL which works with a lot of the more data warehouse solutions out there (with some tweaks per solution unfortunately).

> I believe it drastically reduces adoption of these tools

I really don't see why. From afar they might appear to perform the same task but once you take a closer look you quickly realize they follow fundamentally different architectures and have significantly deployment and performance characteristics.

Just because there are all kinds of hammers, but there are also plenty of different uses, and even in the subset involving driving nails there are significant requirements.

I said "you went to far" to claim Kafka, Spark and Flink do basically the same thing. It's reasonable to use all 3 of them in the same team - so they clearly don't do the same thing.

> I believe it drastically reduces adoption of these tools

As does any competition. Tons of smartphone makers = less adoption for any one of them - and many will close down. Still not a bad thing.

> As does any competition.

Those are open-source frameworks, which is antithetical to competition. The whole point of open-source is that the contributions of one entity can benefit everyone, while in this situation, having so many frameworks "dilutes" the efforts because the same problems need to be solved in each framework. Smartphone makers aren't here to share their technical advances with everyone, they're here to make money.

I'm not saying we should purposefully agree on killing all but one framework, but the ideal situation is if one or two come out at the top as being just the best in their category so that everyone just use them, and all efforts converge towards those.

I don't think that's how it works; "best" is sooo hard to define. Humans are very diverse, and have different needs/preferences. Also what is best today may not be best tomorrow (not because of the system itself, but because of changes in the environment), so it pays off to have "sub-par" systems evolve in parallel with different slightly different goals or implementations.

I don't see "open-source" as being antithetical to competition, at all. In fact, e.g. Databricks only exists because of Spark. There is competition in the open-source world too, and IMO that's a good thing.

Since when is there no competition and diversity in open source? Isn't too much choice usually a major complain with linux, programming languages, frameworks, etc?

Nobody said that. But having too many competing overlapping products does mean more fractured effort spread amongst all of them.

>I'm not saying we should purposefully agree on killing all but one framework, but the ideal situation is if one or two come out at the top as being just the best in their category so that everyone just use them, and all efforts converge towards those.

That assumes you can have a single framework with all the features people want without causing issues (conflicting basic designs, high cost to maintain, deployment costs, configuration costs, etc.).

Not quite. All of these are open-source projects but they are very different in many aspects:

* Apache Flink

Sophisticated stream processing framework with focus on robustness (managed memory) and correctness (exactly-once semantics)

* Apache Flume

Tailored towards log data.

* Apache Storm

First stream processing framework. Legacy.

* Apache Samza

Only used at LinkedIn. Tight to Hadoop's YARN.

* Apache Spark

Only great in batch processing.

* Apache Apex

Dead project. Tight to Hadoop's YARN.

* Apache Kafka

A distributed message queue with simple stream processing built on top via the Confluent Platform.

Nice list. There are also non-Apache projects like NATS, hosted by CNCF.

Here are the 19 streaming and messaging projects and products that CNCF is tracking: https://landscape.cncf.io/category=streaming-messaging&forma...

> Only used at LinkedIn.

And Intuit, Uber, Netflix, VMWare, ....

https://cwiki.apache.org/confluence/display/SAMZA/Powered+By

As a frequent user of almost half the programs you've listed... I couldn't disagree more. They're completely different.

I'm surprised to see this level of misunderstanding posted with such confidence.

It makes you wonder about some of the other insightful sounding top comments on topics that you aren't as well versed in

In the beginning, there was Hadoop, which was just a MapReduce clone, more or less.

And then there was Hadoop 2, which was kind of meant to do everything.

And that didn't really work out, so there was specialisation.

I don't really see the problem. Many of these do different things, others do similar things different ways. Some are essentially dead ends (I doubt there are many new users of Storm, say). This all seems pretty normal.

No Hadoop does more than MapReduce, much more. Primarily: MapReduce does not have a distributed file-like block storage.

OK, so, to be more precise:

Hadoop MapReduce does what Google MapReduce does, and Hadoop Distributed File System does what the Google Filesystem does.

And Apache Spark does the same thing as that other Google product whose name I forget.

And Drill vs Dremel, etc etc etc.

I think that the OP is plenty right for the purposes of the point they were making. And I think (from a non-Googler's perspective) that it's a worthwhile point, since the FOSS ecosystem has since evolved along similar lines to Google's internal tools.

Apache Spark does what Google FlumeJava does, express a chain of map/reduce steps in a DSL (and inline away some intermediate results).

Apache Flume is unrelated except for the confusing name.

... I am using Apache Pulsar to replace current Kafka setup * Apache Pulsar is an open-source distributed pub-sub messaging system

What compelled you to replace Kafka with Pulsar?

Not OP, but I'm keeping a close eye on pulsar.

Im not sure I'd want to replace an existing / working Kafka setup with it, but if pulsar catches on, I would very strongly consider using it for newer projects.

I haven't used pulsar extensively in production for anything, but all the setup and testing I've done on my own was a pleasure. Hoping pulsar continues to grow.

> Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing.

Yes, conceptually they are very similar. If youu want something radically new then check out Bistro Streams: https://github.com/asavinov/bistro

Solving problems of the internet behemoths -> creating opportunity to be acquired.

> Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing.

That's like saying C++, Java, C#, Scala, Python, Clojure... all do basically the same thing

isn't this just a sign of a still young field (data science)? maybe we'll soon start to enter a consolidation phase.

Reply on news.ycombinator.com