I think this has a lot of roots from NSQ.
But, NSQ has no replication support.
I think built in replication is very nice to have. Would like to try once this arrives.
I think built in replication is very nice to have. Would like to try once this arrives.
We don't have that kind of guarantee in NSQ. To get rid of that, we need to maintain the replication our own by publishing the message to two different NSQ servers.
Each of our app servers has its own local nsqd. If that nsqd stops responding, we take the app server out of the load balancer. The local nsqd publishes to multiple channels, which get consumed by hosts in different datacenters.
There are still potential failure cases: nsqd loses connection to consumers, messages start building up, then the machine somehow goes away. The only way to prevent it is to take the machine out of the load balancer any time nsqd messages start backing up, but we prioritize serving requests over sending messages.
We maintain NSQ close to the consumer. We don't our publisher to take responsibility to the message processing. Once it's push to the queue, we need to make sure it's getting process anyway.
That's why we publish same message to multiple queues. All our DB operations are idempotent. So, we are okay with processing the same message multiple times.