MillWheel: Fault-Tolerant Stream Processing At Internet Scale [pdf]

db.disi.unitn.eu

It's a good time for streaming processing infrastructure. Twitter has just released Summingbird, and LinkedIn has released Samza:

- https://blog.twitter.com/2013/streaming-mapreduce-with-summi...

- http://samza.incubator.apache.org/

Of the three I'm most interested in Summingbird. Unlike Google's system I can actually download it, and it seems to provide a better query abstraction than Samza. I haven't spent much time investigating any of these systems, so I might be incorrect in this assessment.

Aren't there also at least one other stream processing framework running atop Hadoop / HFS ?

Edit: Guess I was thinking about spark[1] but it doesn't really fit.. others?

[1] http://spark.incubator.apache.org/

I'm not sure what you'd be able to do with Google's system if you could download it; the code is thoroughly dependent on the rest of Google's idiosyncratic internal codebase.

Is Internet scale bigger or smaller than web scale?

The web (http) is a subset of the Internet, so by definition Internet scale would be bigger.

Reply on news.ycombinator.com