Reaching 200K events/sec | Hacker News Reader

Reaching 200K events/sec

aphyr.com

87 pointsmattyb28 comments

If you're wondering about the workload, this is the trivial Riemann config this benchmark uses:

  (streams
    (rate 5 (comp prn float :metric))

Which means for each five-second interval, sum all the metrics of the events flowing through this stream, divide by the time elapsed, and print that rate to the console.

I'm using this setup to put the heaviest load possible on Riemann's client and TCP server while I optimize those layers--it's not meant to stress the internal stream library, the state index, or pubsub. When I start optimizing those components, I'll have more "real-world" numbers to report.

I should also explain that this particular post explores the high-throughput, high-latency range of the client spectrum. End-to-end TCP latencies (not counting wire time) for single-event messages are on the order of ~100 microseconds-1ms, with occasional spikes to ~30ms depending on JVM GC behavior.

You are likely doing something wrong. My in-memory redis server clone that also uses netty reaches 1.5M requests/s on an arguably heavier workload with a more complicated protocol. I'd love to help you optimize it if it matters.

Thanks for the offer! I, uh, certainly didn't mean to claim that Riemann is extraordinarily fast. Up until this point I've been working as hard as I can just to make the darn thing turn on and accomplish something useful. This is quite early in the optimization process, haha. :)

The principle bottleneck in this particular test is actually the client--Riemann itself spends ~94% of its time waiting on epoll in this test. The client is a total hack using Java OIO, calling flush() on every message, Nagle's algorithm disabled for low per-msg latencies... it's a wreck, haha. Part of next week's optimization push is replacing it with a Netty client and tuning TCP options for various workloads.

Things will probably look better once you switch your client implementation to Netty as a similar test I did with Netty 4, Protobufs and batching of messages produced low seven figure numbers too.

Ah ok, my test was run using the C redis-benchmark client.

Is the source available anywhere?

1.5M locally or over a network? What is the size of each message?

http://github.com/spullara/redis-protocol — server module

The test was run with "SET key value" which in the redis protocol is something like 15-20 bytes / message.

>> Throughput here is measured in messages, each containing 100 events, so master is processing 200,000–215,000 events/sec.

So in reality it is ~ 2k messages/sec. This is a rather poor throughput, as even off-the-shelf generic web servers (e.g. nginx) have the throughput an order of magnitude higher, and proprietary systems can reach 500k messages/sec over the network.

Ah, I didn't intend for this post to reach Hacker News absent context. If you haven't been tracking Riemann, this post might not make sense. ;-)

Riemann is not an HTTP server, or anything analogous. It's an event processor, and reacts to incoming events by running them through an arbitrary set of functions. Events are the logical "requests" against the system, if you're thinking in HTTP terms. Messages are just a bundle of events for synchronous transport, and events can be repackaged in varying bundles of messages depending on latency/throughput requirements. The clients can do this for you.

For instance, the code which generated this benchmark looks like:

  (send client
    {:host "test"
     :service "drop tcp"
     :state "ok"
     :description "a benchmark"
     :metric 1
     :ttl 1
     :tags ["bench"]})

which is a synchronous call, returning when the event is acknowledged by the server. It's making that call 200,000 times a second (in various threads). The clients are doing all sorts of internal buffering and pipelining to make that possible--this particular test uses a batch size of 100 events/msg.

Take a look at http://riemann.io for more. :)

Well, I'm sure there are specific use cases where Riemann would be preferable to a generic web server. But for most developers in most situations, it is a no brainer to choose an HTTP-based protocol with off-the-shelf HTTPD server over a 10x slower proprietary system.

Did you even read what aphyr wrote before posting this? It detailed that unlike HTTP, which in general supplies one request per message (GET, POST, etc), a Riemann message contains potentially hundreds of different requests that must be processed individually.

You can put as many "events" in the body of an HTTP POST request as you wish. What really matters in distributed messaging systems, from the performance point of view, is the number of distinct messages per second.

And if a system designer wants to send a stream of "events" to another system to be acted upon, and if this designer cares about throughput (which is assumed here, given the title of the post), then this designer is likely to choose a faster messaging system, especially if it is more flexible, due to its ubiquity and universal support, protocol (e.g. HTTP) over a custom protocol.

Is that why HTTP is always used for high-throughput network services like MySQL, Memcached or Redis? Oh wait it isn't.

What similarities between the two pieces do you see? What task would you replace Riemann with httpd? Genuinely curious, though slightly skeptical.

They're completely different kinds of software, at least as I understand them. Riemann is about pushing lots of little hashmaps (events) through a DAG of streaming functions; HTTP is about synchronous gets/puts/updates/deletes to a tree of resources.

If you are trying to make sense of Riemann in HTTP terms, sending an event to Riemann might look like POST /streams, with a body containing a single JSON object. There's no notion of GET, PUT, or DELETE though--the state inside Riemann streams has no name or external representation.

There are other components in Riemann which can be expressed as HTTP resources--the index, which is used for tracking the most recent event for a given host and service, and the pubsub system for example. Those have HTTP APIs for making a query (GET /index?q=service = "www" and state = "critical"), and a websocket variant which streams down updates for that query to you.

But as far as a general replacement, I'd say no, it doesn't make any sense. This is more akin to... a slow, insanely flexible, less complete version of Esper than an HTTP server.

Don't let the afterhours haters -- sorry, hackers -- get you down. Riemann looks to be a fantastic piece of open-source software for scratching a particular itch.

Coincidentally, I was just working on a tracing system to dump data to Riemann while eating HTTP logs and/or handling live requests from browsers. It seems to be just what we need to aggregate, monitor and graph our trace data. Thanks!

Thank you. :) If you have any questions, feel free to hop on Freenode #riemann and I'll do my best to help out.

BTW, you're not the first to wonder about streaming events directly to Riemann from client browsers. I... don't recommend it, just because I don't have the time to appropriately guarantee Riemann's performance and security characteristics as an internet-facing service (yet), but adding an HTTP POST path to (ws-server) is definitely on my list. Even if the HTTP+JSON interface is much slower than the TCP/UDP interfaces, I think it'll be plenty useful for many deployments, especially those making requests from JS.

Er, I don't mean to be contrarian, but Riemann is anything but proprietary. I'm an unemployed OSS developer. Every bit of code from clients to dashboard to server to integration tools to the website is open source: http://aphyr.github.com/riemann/. I'm not trying to sell anything. I'm just building a tool to solve a problem I faced in developing distributed systems.

Second... I guess I can reiterate. Riemann is not an HTTP server. It's an event-stream driven monitoring system. The protocol uses existing standards (e.g. protobufs), is simple to implement, and the community has written clients for many languages: http://riemann.io/clients.html.

As an aside, I do plan on adding an HTTP interface to Riemann, but HTTP processing (and using JSON for serialization) comes with certain unavoidable costs in bandwidth, memory and latency. It'll fill a complementary space to the existing TCP and UDP interfaces.

Sorry, by "proprietary" I meant "custom". I admire people who have skills and dedication to built OSS, this was just a wrong word to use.

I completely agree that for specific uses Riemann is great. Your post was, though, about the performance/throughput, and so my comment was about the performance/throughput. Streaming messages/events over the network is an old problem, with well-known limitations, and this was what my comment was about.

Please don't criticize what he's built before you even barely understand it.

TLDR: Burned by framework magic. Talk about side effects.

Ten layers (and probably buffers) traveled through until your data hits the wire. Layer x decides to change its IO model and your throughput takes a dive. It's exactly why there was a post recently about building an operating system just to run some network daemon.

What post was that?

I dunno if I'd call it magic, per se. Netty is a big library and it has good reasons for behaving this way. I just don't think I came through the right documentation path is all.

  (defn execution-handler
    "Creates a new netty execution handler."
    []
    (ExecutionHandler.
      (OrderedMemoryAwareThreadPoolExecutor.
        16       ; Core pool size
        1048576  ; 1MB per channel queued
        10485760 ; 10MB total queued
        )))

It is a farce, isn't it?))

Heh, I originally cargo-culted this line from somewhere in Netty's docs: https://github.com/aphyr/riemann/blob/732a6e8986f75c638c30b5...

After finally digging into how execution handlers worked, the puzzle started to unravel. Netty's docs are pretty good, but you have to understand what all the names mean before you can understand, well, any one part of the system. Bit tough to piece together, at least for my little brain. ;-)

All I'm trying to say is that switching to the prefix notation adding parenthesis does not make Java a Lisp. ;-)

I'm also not sure that JVM itself is a good idea, especially for serving content, but I do respect people who are trying nevertheless.)

Some of Riemann's code is very much like Java, especially where I interoperate with Java libraries. Other parts of the code are... definitely not Java:

  (defn where-partition-clauses
    "Given expressions like (a (else b) c (else d)), returns [[a c] [b d]]"
    [exprs]
    (map vec
         ((juxt remove
                (comp (partial mapcat rest) filter))
            (fn [expr]
              (when (list? expr)
                (= 'else (first expr))))
            exprs)))

I wouldn't categorize Clojure as a "pure lisp"--it relies heavily on the JVM type system, for starters--but idiomatic Clojure feels closer to Lisp than Java, to me.

Offtopic, but it's refreshing to read blogs with clean designs and readable texts, without commercials or widgets inviting to click on things and all that crap.

Reply on news.ycombinator.com