Hacker News Reader

Comment by giantg2 | Hacker News Reader

"As a trivial example, everybody talks about dead-letter queues but nobody really explains how to handle messages that end up in one."

For us, it's either a function that will retry the messages after some time, or manual intervention.

Our department recently said we need to move to even driven architecture for one of our processes that currently runs in a batch. They want us to load data into EMR from an S3 bucket populated by Kinesis. Their suggested implementation is simply to run the batch job more frequently instead of once a day... sorry guys, but that's not even driven...

I suggested maybe just setting a trigger on the S3 bucket and hook it up to Glue, since that would actually be event driven. They said 'no' because they don't want the load to EMR or Glue to run too frequently. I guess that makes sense (not that familiar with the ETL tech), but it sure doesn't make sense to call it event driven.

View on HN

> For us, it's either a function that will retry the messages after some time, or manual intervention.

What does manual intervention look like? Are all downstream consumers blocked until the DLQ is emptied or do all consumers know how to deal with late-arriving events?

> Are all downstream consumers blocked until the DLQ is emptied No other service should know about the state of another services DLQ

> do all consumers know how to deal with late-arriving events In my approach in architecting event driven systems, this question is meaningless. For me some of the core tenants (for me) of architecting these systems are

1. Message ordering should not matter 2. Message delays should not matter (no temporal coupling) 3. Message replays should have no side effects

This means that Service B should have no knowledge of what happens in Service A. If something in Service A fails and gets sent to Service A's DLQ, it should have no impact on Service B. This often involves rethinking business requirements and processes to account for workflows that get interrupted - I've yet to find an insurmountable issue.

If however, we are talking about a message broker trying to deliver a message from Service A to Service B, failing and forwarding to Service B's DLQ, then there is a very simple solution. Have your message broker deliver messages to queues. Service B then processes and handles messages from the queue rather than having to immediately handle all incoming messages from the broker. This protects against a lot of failure modes. The only thing(s) that arrive at Service B's DLQ are exceptions thrown internally in Service B

>What does manual intervention look like?

A ticket or an ops alert depending on severity. The cause gets fixed and messages are retried or dropped depending on your business case.

>Are all downstream consumers blocked until the DLQ is emptied

No, its a diffent queue. The point is that you eventually pull them into the DLQ so you aren't wasting all your resources on failing messages.

>do all consumers know how to deal with late-arriving events?

Hopefully, but an outage is an outage. You're way past happy path at that point.

Manual intervention could be moving them back to the main queue to try again, or looking up the data that failed like if the customer that is being transacted on exists. Most systems don't have an issue processing late messages. The ones that do would have to have some sort of retry/ingress process.

Reply on news.ycombinator.com