Four billion messages an hour: benchmarking Deepstream throughput | Hacker News Reader

Four billion messages an hour: benchmarking Deepstream throughput

deepstream.io

131 pointswolframhempel44 comments

So, roughly 1.1m messages per second of about 23 bytes.

We're handling 4m to 6m 1.5k log lines per second using Apache Kafka on a cluster of around 100 nodes.

Do you mind if I ask what size of machine in CPU/Mem?

Each node is something like this:

    2x Intel Broadwell E5-2630 v4 (10 core, 2.40 GHz)
    128GB - 4X DDR4 32G ECC Registered 2133MHz DIMM
    6 x 2.5" 240GB Intel SSD 540
    x1 S2S 1G LAN/B W/OCP BRKT ASSY(1IN1)
    x1 Solarflare 10G SFP+ SFN7002 PCIe card
    TPM Module
    Dual Power Supply 1100W each (x2 / chassis)

We have this stuff built by Quanta for us.

for the curious, if quanta pricing is anything similar to supermicro this is about $750k in hardware.

1.5k log lines is how many bytes? Would've been helpful if you sticked with the same unit.

Sorry, that wasn't clear.

Each log line (in JSON) is roughly 1.5kB.

I think he means 4 to 6 million log lines with 1.5 kB each.

That's almost 70gbit/s (are those cloudflare http logs by any chance?) on 100 nodes vs ~170mbit/s on 6 nodes.

Or, in other terms 700mbit/s per host with your kafka setup versus ~30mbit/s per host in the benchmark. Allthough your machines seem to be quite a bit beefier (I wonder if all that RAM is actually used?).

A lot of it is log data from requests passing through CloudFlare. We run them through Kafka and consumers do stuff like attack detection and generate statistics for our customers. We have about 4 million sites on CloudFlare and each customer has access to analytics about their site which are stored in a CitusDB database.

Impressive. Any chance you could tell how much data is stored for the analytics service after pre-aggregation? (In terms of TB/day or so - I guess it can't be the full 70gbit/s?).

No, nothing like that much data. It's aggregates like "requests per second" and "attacks per second" etc. The actual log lines aren't stored.

More details: https://blog.cloudflare.com/scaling-out-postgresql-for-cloud...

Thanks. I had read that article some time back :)

Sadly it does not give a figure/order of magnitude of the amount of data that's stored in citus after aggregation, but I guess it's just not public information. [I'm working on a system that is somewhat similar to CitusDB (eventql.io) and am always really interested in these numbers]

EDIT: I can't reply to your other comment for some reason but many thanks for digging that up, it's very interesting info to me!

I'd be happy to make the numbers public I just don't have them in front of me. It looks like data is going into the CitusDB cluster at about 15Mbps.

would love to know what field / business domain this is in, sounds fun!

I work for https://www.cloudflare.com/

Here's the thing, this isn't really just a PUB/SUB queue. Their forex data example is disingenuous since it's a bad application of their product. For comparison, our market data stream via ZeroMQ using UDP multicast gets around 20 million msg/s (~15 gb/s) with <10 microseconds of latency. But we also buy very high end switches and NICs (that guarantee near-zero packet loss) with built-in redundancies. We use quorum and hardware fencing for fast failover. Even then our total network hardware cost is < $20k for 8 nodes. Our stream aggregator/producer and clients use fast serialization formats like flatbuffers that minimize dynamic memory allocations. JSON is suuuuper slow in comparison. We also use ring buffers designed to fit on a cache-line and bypass the kernel network stack (using DPDK). Fact is, if you're just aggregating logs there's no point in going through all this trouble. Nor will a cloud provider set up custom networking hardware just for you.

These high level messaging frameworks attempt to provide some degree of reliability and easy clustering out of the box while keeping things simple. For specific domains where performance is paramount, you can always find light-weight solutions that outperform them. For instance, with market data it's almost always better to drop a quote if it's out-of-sequence or got dropped somewhere than to add latency waiting for a retransmission. This simplifies things since you no longer need durability.

I'm interested to know what happens when there is a failure or a deployment? Having static servers handling load is only part of the problem in our experience. The true complexity and scalability of a system comes when you consider how it copes under load with unexpected failures (network, hardware), but more importantly expected maintenance such as regular deploys, scaling up and scaling down events. Do you have any metrics for that? Those are the problems we've been focussing most of our energy on at Ably (https://www.ably.io), not just message per second rates which is often not really the problem.

yup, failover times under load / HA setup metrics will come out next week

t2 instances have credits that are replenished at a constant rate, and used up when you use the CPU - is this sustainable for more than 1h?

That's ~25MB of data per second for an in-memory workload over 6 machines. I think they missing a zero on the size of the messages?

What does this compare against? Having a little trouble navigating the "real time" apps landscape.

I think the most obvious comparison is Firebase, but as a self-hosted solution deepstream is much cheaper (5-10x, dependant on the usecase). The creators of deepstream.io have actually written a blog post that covers this exact topic: https://deepstream.io/blog/realtime-framework-overview/

  Deepstream relies on garbage collection to free up dereferenced
  memory. If a machine's CPU is overutilized above 100% for a
  consecutive time, garbage collection will be delayed and memory
  can add up. If this continuous for a prolonged period, the
  server will run out of memory and eventually crash - so be
  generous enough when it comes to resource allocation to make
  sure that your processors get some breathing space every once
  in a while.

"Be generous when it comes to resource allocation".

> The costs of running a six-instance cluster for an hour on AWS are 36 cents (6 x t2.medium @ 0.052$/h + 1 x cache.t2.medium @ 0.068$/h)

AFAIK, pricing in AWS world depends upon the region. Bandwidth, hard disks and so on also contribute to the price.

I'm not sure what to make of such conclusions.

How does Deepstream compare to Meteor?

deepstream solves similar problems as meteor, but is conceptually quite different. It's a standalone server that you install the same way you would install e.g. Nginx or a database. Clients connect to it using small SDKs that come in different languages.

It similarly provides data-sync, pub-sub and request response with no opinion about your frontend framework or technology stack and has an open ecosystem of connectors that make it work with all sorts of databases, caches and message buses.

It's also significantly faster than meteor, making it possible to also use it for multiplayer gaming, realtime trading etc...

Isn't Meteor a stand-alone server? It sits on top of node.js, and has a number of extra packages included to make it more of a complete framework, but those packages can easily be removed or replaced.

How does deepstream compare to MQTT?

I see a lot of advantages on using a more standard protocol such as MQTT over deepstream

deepstream is a very different thing from MQTT. It's a server that provides high level realtime data structures to clients and backends alike.

It's closer to a self hosted version of Firebase or Parse than MQTT

How many concurrent connections can you get with the test nodes?

Is it possible to cluster deepstream in multiple AWS regions? It may be beneficial when it comes to latency...

yup, you can cluster deepstream across multiple regions. The only thing to take into account is that its datalayer (e.g. redis cache / some combination of cache and db) needs to be clustered as well

If you only need one hour then indeed it's great price but if you need it 24/7 then it's 260$ a month which is really expensive if you compare alternative (you can get dedicated server capable of same throughput easily for half the price).

true, but Amazon's own messaging / pub sub services, e.g. SNS or SQS go for 50 cents/million or 2000 for four billion - not really a fair comparison of course as they do something quite different.

What you mean exactly with the compared alternative? Capable of the same throughput? In theory or with a real reproducible benchmark? And which type of throughput? What kind of messages?

Tell us, what's this product that you have that's so successful that it needs 4Bn messages/hr full time, that you can't put together $260/mo?

First of all you are replying to something i didn't write. Or maybe show me where i wrote that I HAVE product that can't put together $260/mo ?

It's not about 260$ it's about tens of thousands of dollars saved by companies i work with after migrating from AWS.

AWS is overpriced service for people that do not have time or resources to do things on their own and pay massive prices for that.

Saying that you "have" was in this case more of a rhetorical choice than anything. I didn't really think you had such a successful product.

You didn't understand again what i wrote, i will let you live in your imaginary world of assumptions.

I did understand. Like I said it was a style choice. I could've just as well have said "What kind of project is that successful and can't scrap together $230/mo?" and avoided the "your project" reference altogether. Choice of style. Didn't think you'd get hung up over it.

"AWS is overpriced service for people that do not have time or resources to do things on their own and pay massive prices for that."

Isn't that the point of AWS?

The time / resources cost to build (monitor and maintain..) your own infrastructure isn't zero.

Did these companies have spare teams lying around the place, such that no new hires were needed to make the transition? If not, then it's not so much of a saving at the current cost of dev / ops personnel, just now on a different balance sheet.

Yep. Aws is funny. You want to at do it first, then it doesnt make sense at e.g the 10k per month price point. So you do something else.

Then you come back when you have due diligence and compliance and availability requirements. And you see how much that is really costing you with the arrangements you have.

Finally if you're super successful, you could self host or otherwise. But "rack costs" (naively) trade off against people and auditors.

Summary: AWS is expensive, especially for bandwidth. Practically if your business requirements are not bandwidth intensive, it ends up being paradoxically cheaper than the alternatives if your business involves legal agreements and things like availability commitments to customers.

It's not really rocket science you know, to install elasticsearch/redis/cassandra/0MQ/puppet etc and scale it to hundreds of servers. You do not need to build your infrastructure from ground up, just go with dedicated servers or with colocation if you want your own hardware and you will save tons of money comparing to AWS.

It's not overpriced, you just don't understand the concept of maintenance, monitoring, doing updates ect ...

Although I generally disagree with the parent comment, I have to say that AWS actually IS overpriced. Just look at Cloud plattforms like OVH.com or even digitalocean.com, they charge almost nothing for the traffic, whereas it comes at a whopping 0.09$/GB at AWS. If you aren't dependant on specific AWS products like DynamoDB you will definitely find a much better deal elsewhere.

I understand it very well i am doing it from before the time AWS existed.

Reply on news.ycombinator.com