Concord – High Performance Stream Processing with C++ and Mesos

If by 'input/output' sources, you mean computations that will pull or push from an external system such as kafka/cassandra, there isn't any documentation currently. However we have written connectors to Kafka and Kinesis. You can check out the Scala Kafka Source here [1]. Internally we are working on a high performance Kafka Source in C++ (based on librdkafka). At the moment this source can push records downstream at a rate of > 350K QPS.

[1]: https://github.com/concord/concord-jvm/tree/master/concord_k...

Concord's FAQ has a section on getting data into the system [1]. An example I've used is processing xml files at scale. A short input function digests the xml stream from our servers and another short output function creates a new document in Elastic.

[1] http://concord.io/docs/faq.html#how-do-i-get-data-into-the-s...

http://concord.io/docs/faq.html

Since Mesos uses Zookeeper and we are tightly bound to Mesos, at the moment this is not possible.

Yep, agreed. Seen it first hand and I'm impressed.

DC/OS[0] is a fast way to set up Mesos (via Terraform[1][2]) if you just want to kick the tires without having to learn a bunch of stuff.

Note: No relation to the DC/OS people, just a happy user.

[0] https://dcos.io/

[1] https://www.terraform.io/

[2] https://dcos.io/docs/1.7/administration/installing/cloud/pac...

Is this open source? Can you please point me to the schedular code?

Yes, it is. Storm 1.0.0 was released on Apr 12.

Yes Storm is still under active development. Yahoo, Baidu and other big players continue to use Storm.

To be honest Storm's API is richer. Our approach however was to make stream processing available to all developers. It doesn't get simpler than four callbacks:

  void init(CtxPtr context);
  void destroy();
  void processRecord(CtxPtr context, FrameworkRecord &&r);
  void processTimer(CtxPtr context, const string &key, int time);
  Metadata metadata();

Here's some benchmark:

Word count of 1.13B messages - Storm: ~16K QPS/node, 100ms per event (P999) - Spark Streaming: 100K QPS/node, 1s batch window - Concord: 500K QPS/node, 10ms per event (P999)

Server log processing (29G server log, ~260M msgs) 7 different computations including deduplication, counting, pattern matching, windowing... 4 nodes, 8 vCPU, 32GB RAM each Concord: 1M – 1.8M QPS / cluster Spark Streaming: 72K – 2M QPS / cluster

Concord generally performed in the consistent range of 1-1.8M QPS for whereas Spark's throughput varied differently based on window sliding / amount of internal shared state.

One of the cofounders here. We're in C++ - no 15-20% cost on JVM. Plus we've made a lot of optimizations on our communication and networking including libevent.

There are a lot of reasons why, probably enough to write a blog post about. To summarize I would say that this is mainly a byproduct of being close to the metal, and our use of lock free code on the fast path. Also our system is relatively simple when compared to Storm which I also believe lends to less work and improved performance.