Comment by jetti | Hacker News Reader

Comment by jetti | Hacker News Reader

Where I currently work we are all in on event-driven architecture. For our DLQs, we have alerts on when the queue is growing in size or if messages are in the queue too long. When those alerts come in, we manually move the messages back to the normal queue for reprocessing and if they get DLQed again after that we will look into the reason it is failing.

One of the benefits of this architecture for us is the ability to easily share information between services. We utilize SNS and SQS for a pub/sub architecture so if we need to expose more information we can just publish another type of message to the topic or if we need to consume some information then we can just listen to the relevant topic.

There are two big issues that I've run into while at this company. One is tracking down where events are coming from can be a big pain, especially as we are replacing services but keeping message formats the same. The other big issue is setting up lower environments (dev,qa,etc) can be difficult because you pretty much need the entire ecosystem in order for the environment to be usable, which requires buy-in from all teams in the organization

Do you have any control over the individual services?

One way to ease some of that pain is a standard library to obtain your keys and topic and publish metrics when things are published and consumed, or at least logs on startup.

It's a pain to get buy in, and 10x harder to keep it updated. But if you can solve the problems around getting names and secrets and stuff, folks are usually open to the conversation at least.

I guess it's still harder to track down event emitters, but have you tried using bitbucket or GitHub code search to search all of your repos at once?

Yea, I have use GitHub search in a pinch and sometimes it is helpful enough to show me exactly where to look. Unfortunately, though, there are several events we emit that are many layers of string concatenation, so GitHub search may narrow it down to 4 or so places and I have to manually go from there.

Solution for that is using equivalent of user-agent - write it to message headers. Not sure if SQS supports it like Kafka.

I'm really surprised that not a lot of companies use this.

Thanks for your answer, it really helps.

Does moving the DLQ messages back to the normal queue mean that all consumers can deal with out-of-order scenarios?

if you're using a queue like SQS and expect it to be ordered and exactly-once, you're in for a lot of surprise. If you need ordering, use a stream/log like Kinesis or Kafka.

Exactly-once is not a requirement but ordering is. I had Kafka and Kinesis in mind when writing this question but just in case you haven't seen it, there is a way to get ordering guarantees using SNS and SQS:

https://aws.amazon.com/about-aws/whats-new/2020/10/amazon-sn...

Does that really guarantee in-order processing, or just that messages can be picked up in order. If you have multiple consumers on your queue and consumer A picks up message x from the head of the queue then consumer B picks up message y next, it is possible for y to get processed before x. Maybe consumer A is slow (gc pause?) for some reason, and now we are processing out of order, even though the queuing infrastructure does not see it. If you truly need to guarantee strict processing order (x must complete before you start processing y) I think you may need to build that into your app. Or, I misread, which is very possible.

Reply on news.ycombinator.com