Hacker News Reader

Comment by chuhnk | Hacker News Reader

We are using NSQ at Hailo for sending and receiving billions of messages every day. It scales incredibly well and is extremely reliable. We'd be happy to speak more about this with anyone who's interested.

View on HN

How do you handle reliability? I'm reading http://nsq.io/overview/features_and_guarantees.html and I'm curious how you've set up your architecture to handle node failures without losing messages, or if losing messages is acceptable in your use case.

I'm also curious if you've had issues with duplicated delivery, where a message is delivered more than once, and how you've handled that case.

Hi. Good question. We actually have both use cases. In the case where we require reliability and guaranteed delivery we publish to multiple nodes and standard channels that persist messages in the case of subscribers disconnecting or overflow in what can't be buffered in memory. In the other case we just publish once and can use ephemeral channels. We scale clusters depending on usage but start with a minimum of 3 and use nsqlookupd to find topics in the cluster. In the case of things that can't handle duplicate deliveries we offload deduping to simple things memcached.

Can one use NSQ as a replacement for Kafka?

Posed this question to the creator of NSQ, Matt Reiferson just a few minutes ago. His response https://twitter.com/imsnakes/status/559827559806808069

At Uken Studios we use NSQ as a chain of queues for log processing.

  rails -> fluentd -> NSQ (firehose) -> log splitter (python script) -> NSQ (elasticsearch, s3, hadoop)

We have this setup live since November and processed more than 25b messages without hiccups.

Kafka offers persistence, which seems extremely rare in distributable queue. (Is it even available outside of Kafka?)

On that note, what is everyone using these queues for that they can ignore durability?

The standard JMS spec supports durable queues and topics. Generally a nightmare to manage though - trying to track down a unconsumed message on a clustered durable topic with lots of subscribers is like trying to find a needle in a stack of needles.

Also, Kafka guarantees message order, where NSQ does not. This can be dealt with in the client, but something to note.

Rabbit offers persistence.

Depends on how you use Kafka, but yes overall you can.

Reply on news.ycombinator.com