Hacker News Reader

Comment by packetized | Hacker News Reader

We run an average of 14k logs/sec through a two-node RMQ cluster, with max sustained throughput in the ~50k range. You're spot on with the bottleneck being Elasticsearch, but the latest releases in the 2.x train have a lot of fine adjustments that have drastically improved our indexing rate, such that we actually index at a 50k/sec rate. Would be interested to hear about your ES cluster configuration.

View on HN

I'm embarrassed to say that at the present moment we currently don't use ES clustering but rather a monstrous powerful bare metal machine as we had issues with the cluster failing with some network issues we had with Rackspace.

BTW I didn't mean to denigrate Elastic Search (I assume that is why I'm getting downvoted.... a comment would help). We just haven't had the chance to upgrade it and properly configure it.

In fact Elastic has been pretty darn speedy as of lately particularly since we purge some of the data after 6 months (we still have permanent filesystem storage of logs of course).

> cluster failing with some network issues we had with Rackspace

Were you using Zen Discovery at the time?

I haven't kept up with ES development in the last year so maybe they fixed this, but a flaky network can cause a cluster using Zen to freak out a lot.

You can turn off multicast discovery and write in unicast peering addresses. If you are in the cloud and you are clustering this is step 1 =)

Are you using HA functionality and also on disk backing? These two things bring down performance roughly 5-10x and are mostly required for situations that can't afford message loss. I still like the rabbitmq solution, it is my own, but i've found it takes more hardware than you are suggesting.

Yes that's my experience as well.

> We run an average of 14k logs/sec through a two-node RMQ cluster

How many MB/s are you indexing?

What are the new options that help improve indexing rate?

Reply on news.ycombinator.com