Stream Processing with Apache Flink | Hacker News Reader

Stream Processing with Apache Flink

blog.brakmic.com

43 pointsbrakmic9 comments

Here's a good intro on Flink streaming: https://dato.com/videos/data-science-summit-seif-haridi.html

Summary: advantages are real-time (10s of milliseconds) processing latency, exactly-once semantics, failure-recovery model based on Chandy Lamport, backpressure support.

Hey, I know you're saying you're not an expert on Flink or Spark, but do you know if Flink and/or Spark allow you to merge a stream in real-time into a data set that you can run queries against?

Generating analytics directly off a stream has never seemed that useful to me. Being able to take all the data in the stream and merge it with previously collected data, then query that merged data is the sort of thing I need to be able to do.

Hi DannoHung,

Maybe this article could be of some value to you: http://flink.apache.org/news/2015/02/09/streaming-example.ht...

Btw. handling DataStreams and DataSets in Flink is done via almost similar API calls. In DataStreams you only have to additionally insert the Time-Window parameters. Everything else (transformations like filter, map, flatMap etc.) remain the same.

Kind regards, Harris

"Being able to take all the data in the stream and merge it with previously collected data, then query that merged data is the sort of thing I need to be able to do."

Sorry for the shameless plug, but Striim does that with a SQL-like interface. Joining streams and previously collected, cached data is as easy as writing a join.

You can try it out here: http://www.striim.com

If I'm not mistaken, "striim" neither is an open-source project nor an open-sourced product.

I don't know about Flink or Spark, but streaming joins is a use case Apache Samza was explicitly designed to handle. It uses node-local caching to make the join efficient, and persists the cache updates in Kafka to make the cache persist across node failures.

Really impressed with Flink and the Data Artisans team. If I had to bet, I'd say they're the future of streaming.

Curious that there's no comparison with the other stream processing platform of choice, Apache Storm.

Hi,

Some time ago I wrote an article about Apache Spark: http://blog.brakmic.com/data-science-for-losers-part-3-scala...

However, I don't think that writing a comparison of such complex technologies within an introductory article would be of much value to readers. Of course, it would be easy to create some tabular structure containing "pro/contra" about Flink and Spark, but this would be way too simplistic, imho.

To be honest, I don't consider myself an expert on Flink and/or Spark so it would be unfair towards my readers to embed such a comparison within the article.

But a properly written comparison could be of value for those who have to decide what kind of processing (batch/streaming/machine learning etc.) they need to fulfill some specific (that is, "business") tasks.

Kind Regards, Harris

Reply on news.ycombinator.com