Hacker News Reader

Comment by nivertech | Hacker News Reader

Can one use NSQ as a replacement for Kafka?

Posed this question to the creator of NSQ, Matt Reiferson just a few minutes ago. His response https://twitter.com/imsnakes/status/559827559806808069

At Uken Studios we use NSQ as a chain of queues for log processing.

  rails -> fluentd -> NSQ (firehose) -> log splitter (python script) -> NSQ (elasticsearch, s3, hadoop)

We have this setup live since November and processed more than 25b messages without hiccups.

Kafka offers persistence, which seems extremely rare in distributable queue. (Is it even available outside of Kafka?)

On that note, what is everyone using these queues for that they can ignore durability?

Rabbit offers persistence.

The standard JMS spec supports durable queues and topics. Generally a nightmare to manage though - trying to track down a unconsumed message on a clustered durable topic with lots of subscribers is like trying to find a needle in a stack of needles.

Also, Kafka guarantees message order, where NSQ does not. This can be dealt with in the client, but something to note.

Depends on how you use Kafka, but yes overall you can.

Reply on news.ycombinator.com