How does the performance compare to the traditional stream processors?
Word count of 1.13B messages - Storm: ~16K QPS/node, 100ms per event (P999) - Spark Streaming: 100K QPS/node, 1s batch window - Concord: 500K QPS/node, 10ms per event (P999)
Server log processing (29G server log, ~260M msgs) 7 different computations including deduplication, counting, pattern matching, windowing... 4 nodes, 8 vCPU, 32GB RAM each Concord: 1M – 1.8M QPS / cluster Spark Streaming: 72K – 2M QPS / cluster
Concord generally performed in the consistent range of 1-1.8M QPS for whereas Spark's throughput varied differently based on window sliding / amount of internal shared state.