Hacker News Reader

I love seeing how people tackle these sorts of problems. Here, the trade-off is duplicate messages reaching a client. I know others have tried a no-"ack" approach. I'm still working in the land of RabbitMQ solving all my business needs and I like it.

I'd love to see some numbers, reasons for decisions made, and suggested best practices for this solution other than "manual de-dupe."

View on HN

Some of the background behind our decisions are on our blog post http://word.bitly.com/post/33232969144/nsq

In terms of manual de-duping, we strive internally for idempotent message processing so it's fairly irrelevant, but to handle cases where it matters, all of our messages have unique id's added to them outside of NSQ.

The actual cases where messages are handled multiple times is limited to when a client disappeared during message processing (a hard restart) or it passed the allowable time window to respond and was given to another client.

If there are any specific numbers you are curious about, please ask.

When people say "billions of messages per day" do they realize that 1 billion messages =~ 11500 msgs/sec ? RabbitMQ as well as any self respecting message broker is able to handle that on a single machine

Reply on news.ycombinator.com