Here's the thing, this isn't really just a PUB/SUB queue. Their forex data example is disingenuous since it's a bad application of their product. For comparison, our market data stream via ZeroMQ using UDP multicast gets around 20 million msg/s (~15 gb/s) with <10 microseconds of latency. But we also buy very high end switches and NICs (that guarantee near-zero packet loss) with built-in redundancies. We use quorum and hardware fencing for fast failover. Even then our total network hardware cost is < $20k for 8 nodes. Our stream aggregator/producer and clients use fast serialization formats like flatbuffers that minimize dynamic memory allocations. JSON is suuuuper slow in comparison. We also use ring buffers designed to fit on a cache-line and bypass the kernel network stack (using DPDK). Fact is, if you're just aggregating logs there's no point in going through all this trouble. Nor will a cloud provider set up custom networking hardware just for you.

These high level messaging frameworks attempt to provide some degree of reliability and easy clustering out of the box while keeping things simple. For specific domains where performance is paramount, you can always find light-weight solutions that outperform them. For instance, with market data it's almost always better to drop a quote if it's out-of-sequence or got dropped somewhere than to add latency waiting for a retransmission. This simplifies things since you no longer need durability.

That's almost 70gbit/s (are those cloudflare http logs by any chance?) on 100 nodes vs ~170mbit/s on 6 nodes.

Or, in other terms 700mbit/s per host with your kafka setup versus ~30mbit/s per host in the benchmark. Allthough your machines seem to be quite a bit beefier (I wonder if all that RAM is actually used?).

Do you mind if I ask what size of machine in CPU/Mem?

would love to know what field / business domain this is in, sounds fun!

1.5k log lines is how many bytes? Would've been helpful if you sticked with the same unit.