Hacker News Reader

Comment by DannoHung | Hacker News Reader

Hey, I know you're saying you're not an expert on Flink or Spark, but do you know if Flink and/or Spark allow you to merge a stream in real-time into a data set that you can run queries against?

Generating analytics directly off a stream has never seemed that useful to me. Being able to take all the data in the stream and merge it with previously collected data, then query that merged data is the sort of thing I need to be able to do.

View on HN

Hi DannoHung,

Maybe this article could be of some value to you: http://flink.apache.org/news/2015/02/09/streaming-example.ht...

Btw. handling DataStreams and DataSets in Flink is done via almost similar API calls. In DataStreams you only have to additionally insert the Time-Window parameters. Everything else (transformations like filter, map, flatMap etc.) remain the same.

Kind regards, Harris

"Being able to take all the data in the stream and merge it with previously collected data, then query that merged data is the sort of thing I need to be able to do."

Sorry for the shameless plug, but Striim does that with a SQL-like interface. Joining streams and previously collected, cached data is as easy as writing a join.

You can try it out here: http://www.striim.com

If I'm not mistaken, "striim" neither is an open-source project nor an open-sourced product.

I don't know about Flink or Spark, but streaming joins is a use case Apache Samza was explicitly designed to handle. It uses node-local caching to make the join efficient, and persists the cache updates in Kafka to make the cache persist across node failures.

Reply on news.ycombinator.com