Hacker News Reader

Comment by sk5t | Hacker News Reader

What if your process to generate the initial messages crashes halfway through?

You return the error to the user.

Crash handling is a matter of managing user expectation. If the user hits a button and the UI shows the command as success, the system better ensures the command will complete eventually. If the user hits a button and the UI shows an error, well the user can try again when the system is up.

Ah, let me restate the question: suppose the user has clicked a button to approve sending 5,000 emails; the server process puts 2,000 messages onto a queue for later processing--but then, something occurs to interrupt adding the remaining 3,000 messages to the queue.

Presuming the queue doesn't support any transactionality beyond per-message, and optionally supposing a queue consumer has already started sending some of these emails, how do you recover from the fault and help the user send the rest of the email (without duplicating outbound email)?

You update the status of each message when it's acked. You show a live count of messages sent / total messages on the screen where the user sent it. 2000/5000 sent... if those 3,000 never get sent, it will be obvious to the user.

If you want the user to be able to try re-sending, you can provide that functionality... you'd need to "cancel" the outgoing messages using a separate queue, re-sending when each cancellation is acked.

It seems as though you might be describing a process with state stored in an RDBMS or the like. Which, while a perfectly reasonable approach, is not much like the initially-described case of firing a bunch of "send email to foo@example.com"-type messages into a queue, subsequently to be drained and acted upon by possibly-remote workers.

What I am trying to uncover here is how one might expect to use a queue, on its own, to support a very-much-non-idempotent interruptible non-transactional process.

Also, what does it mean to "cancel" an already-sent email?

I have yet to use it, but isn't Java EE's Batch API is used for things like this? It tracks the progress of long running tasks (e.g. the index of the current mail) and can continue from the last sent index in case of some error.

Queue is a component of the solution for this kind of non-transactional process, not the entire thing.

I've solved this problem by adding a status field to my email objects/rows which can assume the values "Unsent, sending, sent, error". I guess it is a bit self-explanatory, but status values are set like this:

- Unsent: Default status - Sending: Message dispatched to broker - Sent: Send task/message ack'ed - Error: Send task/message any other exception

This of course assumes that your 5 000 emails are not ephemeral and are in a database.

If by "doesn't support any transactionality" you mean ack's aren't possible for one reason or the other then of course you have to go a slightly different route (pun somewhat intended) by publishing a "result" task/message and updating your database rows based on what comes back on this new "task_result" queue.

I guess the OP's point is that if you're storing everything on a db anyway i.e. 1 row per user,per email and a sent/not sent status flag against each row - then why do you need a messaging system? The email sending system could just poll the db for emails to be sent (by status=not sent), and then update the status after sending, as well as a timestamp for later cleanup. A more robust system is then achieved completely without messaging.

Of course polling is never a great design, but I guess that's the crux of his question.

Push vs pull, auto redelivery, scalable, distributed storage vs single DB.

Imo I'd resort to an ACID datastore if permitted but I'm curious if there are common patterns that enable this with messaging systems.

In the new stated problem, it's a transaction problem rather than a message bus problem. In any case, you don't want to start sending emails while still accepting (queuing) the user's command. I meant you could do it; just have to make it clear to the user that the process is best effort and can fail half way through (setting expectation).

Anyway, there're ways to address the problem. Message bus can certainly help.

If the message bus supports queuing transaction, great. Just mb.beginTrans(), mb.queue(), mb.queue(), ..., mb.commit(). The consumer won't be called until the whole batch is committed. When crash, the whole batch is aborted. Presumably the user is notified the command failed and retry is needed.

If queuing transaction is not supported, there're several strategies to deal with it.

1. First method.

1a. Pack the 5000 email id as one message and queue it. This is like squeezing everything into one transaction.

1b. The consumer unpacks the message onto the list of id, walks the list with a cursor, and processes each id.

1c. After processing each id (sent email), saves the cursor pointer, in a file or in a db table.

1d. After a crash, the message bus will re-deliver the whole message to the consumer. Unpack the list. Read the saved cursor pointer to see where the last processed id was. Set the cursor to continue beyond that pointer.

1e. In the worst scenario, the crash occurs after the email is sent but before the cursor pointer is saved. At recovery, a duplicate email is sent. Email cannot be rolled back so it really cannot participate in a transaction. That's nothing you can do about it. It's an acceptable business wrinkle.

2. Second method.

2a. Queue the email id one by one, attach the user's session id or some pseudo transaction id along with each one (e.g. today's date as the pseudo id).

2b. On a crash, ask the user to re-submit all the email id again, and queue the email id one by one again, with the same pseudo transaction id. You might have duplicate entries queued up.

2c. Have a table recording the processed id's.

2d. The consumer picks up the email id and the pseudo transaction id. Consult the processed table to see if the (email id, pseudo-id) has been processed. If so, skip. Otherwise, process it and save the id in the processed table.

2e. Again in the worst case, a crash occurs after an email sent but before saving the id to the processed table. In that case, a duplicate email is sent out.

3. Third method.

3a. On some task command that's transactional, unlike email, you can set up a transaction to do the task command and updating the cursor pointer at the same time (with method one). A crash would roll back the command and roll back the cursor pointer update.

3b. Same with method two. Set up a transaction on the task command and the insert of the id into the processed table.

3c. If the transactional task command is performed on a separate system, two-phase commit can be used to bind the command and the cursor pointer update (or processed id insertion) together into a transaction.

Reply on news.ycombinator.com