Hacker News Reader

Comment by tacticus | Hacker News Reader

> inherently have all these features, without needing any other tooling.

Providing your queuing service magically scales. and you can just scale that infinitely

If your queue is the bottle neck (yes it happens) how do you know there are more nodes getting added to it? How do you rebalance the topic over multiple new nodes in the queue?

> I shudder when thinking of how we would have handled such failures if we used sync processing.

easily. oh look i got an error and retried a few seconds later and it went to a new isolated backend running separately somewhere else or after it started again.

View on HN

> If your queue is the bottle neck (yes it happens) how do you know there are more nodes getting added to it? How do you rebalance the topic over multiple new nodes in the queue?

In a message queuing system, there is no rebalancing - you just add nodes, which then also get messages, typically in a round-robin fashion. This is not Kafka.

But of course, you can max out your message broker. As you can with a network switch for that matter.

> oh look i got an error and retried a few seconds later

So now you need a retrying mechanism. Which is provided as a basic feature of a messaging system. And with the transactionality, you do not get the problem of wondering whether the first attempt went halfway through, or not.

> In a message queuing system, there is no rebalancing - you just add nodes, which then also get messages, typically in a round-robin fashion.

And hope that the DNS\config propagates to the calling client and that the library can reliably add and remove nodes.

> Retrying

Wanna bet it's a built in feature in your messaging system?

Cause the whole publish op can fail at which point you have to retry sending it. or it published but you lost the ack (seen with AQ more than once) so now you send that message again and hope you remembered that whole duplicate message handling

> And hope that the DNS\config propagates to the calling client and that the library can reliably add and remove nodes.

This is not how it works. A messaging broker client connects to the message broker. The client then creates receivers for one or multiple queues. If the broker has multiple clients receiving from the same queue, it uses a round-robin dispatch to the clients.

I fail to see how DNS or config, or "reliably add and remove" factors in here?

> Wanna bet it's a built in feature in your messaging system?

Yes?

This is not using pub/sub, but queuing. (There is an option of using publish/subscribe, but that is only meant for special cases like updating a GUI, or "broadcasting" an event like "invalidate cache for user X")

Message Queuing can be transactional. Mats leverages this. I have written about it multiple places both on this HN discussion, and on the webpage.

Reply on news.ycombinator.com