Do We Need Distributed Stream Processing?

A couple of great posts by Frank McSherry along the same lines a few years ago, doing page rank for a 128bil edge graph on a single laptop (in the second post below) -

The first post - "scalability! But at what cost" http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...

The second post - "bigger data, same laptop" http://www.frankmcsherry.org/graph/scalability/cost/2015/02/...

I work on IBM Streams, which is a distributed stream processing system [1,2]. When the cost of communicating between hosts is greater than the cost of your computation, distributed processing is not going to help. When your computation is basically a form of word count, then, yes, using multiple hosts may not help. But if your computation is, say, online speech to text for a large call center [3], then the application will eat just about as much compute resources as you can throw at it. That's going to mean using multiple machines, even if each machine is quite large.

As a plug for our system: we scale down as well as up. If you only have a single host, we will run the entire application in a single process, using threads as needed to achieve parallelism using either dedicate threads or a lightweight operator scheduler [4]. Users can configure which without changing application logic. If you have multiple machines, we will automatically partition your application across those machines for you [5]. By default, we do this all for you, without needing any input. But if you want to explicitly control what executes where, and using how many host machines, that's easy to configure, also without changing application logic.

[1] https://www.ibm.com/cloud/streaming-analytics

[2] http://ibmstreams.github.io/

[3] https://www.ibm.com/case-studies/verizon

[4] Low-Synchronization, Mostly Lock-Free, Elastic Scheduling for Streaming Runtimes. Scott Schneider and Kun-Lung Wu. PLDI 2017. http://www.scott-a-s.com/files/pldi2017_lf_elastic_schedulin...

[5] Automatic Fusion and Threading. https://developer.ibm.com/streamsdev/docs/automatic-fusion-a...

I work on Hazelcast Jet [1], which is a Java based distributed stream processing engine. The core engine is fast enough that it can be used with very good throughput on a single node (several times faster compared to Flink or Spark) but usually several nodes are not only needed strictly for parallelization but also for tolerating node failures and being able to restart where you left off. As others have pointed out, not every computation can be parallelised efficiently. Jet also offers in memory storage, so adding more nodes also increases your storage capacity.

Since the core of Jet is small enough (~400kb JAR), we also considered making a non-distributed version that runs strictly in process. Mainly for lightweight usage or embedding but would also offer a path to distributed execution, if it was ever needed.

[1] https://github.com/hazelcast/hazelcast-jet

It's a bit of a chicken and egg problem, since when designing new systems, management asks "How fast can we run this?", to which engineers reply "How much data is there and what speed are we aiming at?".

At this point, a failure to specify the desired volume and latencies leads to a sad cycle where scales are overblown ("just in case") and a complex solution chosen. Distributed solutions necessarily come with a massive overhead, so it is later discovered that even larger systems are needed…

Many such anecdotes circulate the industry, and I can add our own: In 2012 we implemented SVD, a math algorithm at the core of many machine learning techniques like PCA, LSI etc. This was a fast streamed "local" SVD implementation in Gensim (and now even faster in ScaleText). Suddenly, use-cases that needed a cluster of 12 beefy Hadoop (Mahout) machines could be performed on a single laptop, faster, and after a single `pip install`.

Our perf testing of word2vec (yet another streamed ML algo) showed a similar ROI pattern of local-vs-generic: https://rare-technologies.com/machine-learning-hardware-benc... (note the Spark graph at the end).

In any case, there are two ways to distribute stream processing tasks:

* [~Heavyweight] Use a cluster with multiple runners like [Spark Streaming](https://spark.apache.org/streaming/) or [Flink](https://flink.apache.org/)

* [~Lightweight] Do stream processing in applications, at the edge, on gateways or devices like [Bistro Streams](https://github.com/asavinov/bistro/tree/master/server) or [Kafka Streams](https://kafka.apache.org/documentation/streams/)

Normally, distributed stream processing requires also partitioning the data, e.g., by user or session ids, so that these isolated streams can be processed independently at different nodes.

There is a clear value in making Stream processors work closer to hardware. That does not mean single node can handle does better under all use cases. When the complexity of queries increases and when queries can be portioned, there are use cases where still distributed setup can do better.

I work on WSO2 Stream Processor, https://wso2.com/analytics ( Opensource, Apache Licensed). We have chosen to support both worlds where SP has an HA mode that can two servers in a Hot-warm mode that do 100K event per the second and give you a two-node deployment. If you want more, SP runs on top of Kafka, scales, and support multi-data center deployments as well. If the user is in doubt, he can start small and later switch to Kafka without changing any code.

Of course these streams are more rigid than the kind the article discusses, but the tradeoffs are similar, and anyone architecting streams should know about the state of the art in database distribution.

The Yahoo Streaming Benchmark isn't really much of a streaming benchmark, its a "reading things off of kakfa, deserializing JSON, sticking things in Redis benchmark". Really not a good benchmark for understanding the system at test.

[0]: https://www.usenix.org/system/files/conference/osdi12/osdi12...