Hacker News Reader

Comment by dimitrios1 | Hacker News Reader

I have used SQS in the parent's suggested fashion for many years. I feel like your points are overstated. Visibility timeout's "main point" is not to only handle failure nor do the AWS docs themselves state that. AWS's built-in redrive policies have been more than sufficient to correctly handle error scenarios.

> there's no way to ensure that message isn't just processed again by Consumer1 instead of Consumer2!

Correct, but this isn't the job of the pipe. Smart endpoints, dumb pipes.

View on HN

You're welcome to design systems however you want. But this is, put simply, bad advice; and when sharing advice like this to people who may be learning these things for the first time it's critical to communicate not just what these complex components are capable of, but how to best work with them to build reliable and effective systems.

If you have multiple heterogenous consumers, do not use a single SQS queue.

I can't even comprehend how you would engineer around the issue of consumer re-processing. You can quote metaphors all day; if you love the idea of dumb pipes, why doesn't the city transport clean and gray water in the same pipe? Do you want to wash your hands using flushed toilet water?

Similarly, you can't engineer around heterogenous consumers grabbing a message, putting it back in the queue, then consuming it again. You can make them smart! You can have them say "woah hold on, I already saw that message I don't need to see it again put it back". Or, you can make them idempotent so reprocessing isn't undesirable. But its still reprocessing; its still a huge waste, and will probably require external state to manage. Moreover, there's literally no system guarantee that Consumer2 will ever see that message; it'll probably see it, fifty-fifty, well then again if one consumer is faster at accessing the AWS API than the second, who knows, anything could happen, but at least its convenient?

The city doesn't require every household to have gray water filtration. Because that would be insane. The pipes don't have to be "smart". We just build two pipes!

This just blows my mind too. The pipe analogy is apt. Using logic to dispatch to whatever pipe -> consumer you want is the way to use queues. Turning it upside down and using properties of the queue to have consumers decide what they want to take and sending it back to others is just unquestionably bad design when you could just make more queues!

Conceptually it is still a terrible idea to have multiple consumers (By multiple consumers I mean things doing different actions on a message, not concurrent consumers doing the same action) on a single queue. Why overload a queue like that for 2 different actions when one can fan out on an action to 2 queues with SNS? Then your consumer does not have to determine if the message is for them or not. Visibility timeouts are for concurrency/errors by a single action. Yes you could hijack it and have 2 consumers act on one message and do different things but that is confusing and no benefit over just having 2 queues

Weird example, do people actually use multiple consumers doing different things to a single message? You just queue multiple messages with different properties and consumers process things the same way.

The primary reasons for multiple consumers in a queue is availability and SLA reasons on a queue as well as for easier horizontal scaling. Otherwise you’ll need to have a queue scheduler type system that can signal or serve out queue locators to idle consumers and you start getting into technical scenarios similar to freakin’ ESBs. At enough scale you already have that setup though for multi region failover purposes sure but the granularity of queue consumer routing is based not around concurrency to the queues as much as concurrency and routing across several regions with n queues in between that serve as priority queues.

Also, two different queues being two different buffers that have durability issues can in an improperly conceived architecture amount to a distributed RAID0 of messages.

It really depends upon the tolerance to message duplication, SLA needs, and how prioritization should be handled. At a previous place we had multiple consumers for multiple SQS queues representing different priorities within the same region and it worked fine for many years with the primary headache being message de duplication handling being tricky.

This discussion is, at least it seems, mostly about multiple heterogenous consumers; not homogenous consumers/replicas/horizontal scaling. So, if Slack sends a queue message for every DM that's sent, the difference between having 1 consumer that updates the database and 1 consumer that sends a push notification, versus having 2 consumers that both only update the database.

The idea of having multiple homogenous consumers shouldn't be controversial; that's just horizontal scaling. And, well, at least until a few hours ago I also would have said that the idea of having multiple heterogenous consumers is also uncontroversially bad. But I guess everyone has "their way" of doing things.

Its also important to note that there's a third situation I see somewhat often: maybe call it homogenous delegated consumers, whereby you've got messages like '{"type":"SendDM", "content": {}}'. Or maybe: '{"type":"SendDM", "action": "UpdateDB", "content":{}}'. The consumers are still homogenous, they all run the same code, but they may internally delegate the message to do different things depending on enums within the message. This is pretty ok; its different because at least you'd never have a consumer hit message and be like "I don't want this take it back".

Though I'd caution against it; just understand that its something of a 'hack' to make one queue act like N queues, and that's ok if you're small and have a good grasp on the problem domain. The big issue it will inevitably run into is: some queue message "kinds" will take a lot longer to process than others; and so if you're e.g. overloading a queue to handle both a simple email send and a much more complex asynchronous database update, you'll inevitably get delayed emails. Absolutely inevitable. But, it can work for a time.

I’m curious why you would do it this way vs publishing to SNS and having that fan out to multiple queues where each consumer can listen for the things it needs to work on (as mentioned in the original article.)

I found it just was not necessary for most cases. And the way I got there was working backwards from the "web scale" technologies like kafka, kinesis, and dynamodb.

I built a data ingestion system that handled an average of 300 messages/sec, peaking at 1,000, and writing to a single R3 RDS instance. You can do a lot by pushing simple scaling strategies to their limit. Everyone thinks they need to handle web scale, but really you just need to handle your scale.

Reply on news.ycombinator.com