Hacker News Reader

Comment by be_erik | Hacker News Reader

Absolutely, but you can't rely on it to be persistent like Kafka. We use it extensively and are incredibly happy with it, but we follow best practices around not putting state in messages, making changes idempotent, and ensuring that we can always replay a message if needed.

We've yet to lose any messages in production, but it could happen and we're okay with the tradeoffs between that and the operational complexity of kafka.

View on HN

Yeah, at-least-once is definitely the way to go, sounds like you've got a good architecture around that.

Another nice thing I've heard with Kafka is that it can store all messages since the beginning of time (if you want) and you can replay them to retrieve all your MQ-related state. Does NSQ do that, do you know?

With NSQ there is a built in utility nsq_to_file which just becomes one additional consumer you'd use to archive each message topic to disk. It provides dead simple archiving of messages, but doesn't provide any native replay ability.

NSQ does nothing like this out of the box. You could create a channel for each topic that accumulates messages, but those messages would not be distributed across the cluster and after consuming them once they would be effectively gone. If that feature is a hard requirement NSQ wouldn't fit your use case.

I see, thank you. I'm a bit confused by "after consuming them once they would be effectively gone". Surely NSQ supports more than one subscribers, and won't remove messages after they've been delivered to a single client?

It supports more than one subscriber as long as you have a separate channel for each. But if the channel gets created after some messages have been delivered, they will not be delivered to the new channel, only the messages after that.

To do replay with nsq (one way is) you'd attach their file consumer to its own channel as the first operation on a topic. Then in application code you'd need to read from that file whenever you need to do a replay (and you'd need to know when/where to read from/to).

That makes sense, thank you.

Reply on news.ycombinator.com