Crash handling is a matter of managing user expectation. If the user hits a button and the UI shows the command as success, the system better ensures the command will complete eventually. If the user hits a button and the UI shows an error, well the user can try again when the system is up.
Presuming the queue doesn't support any transactionality beyond per-message, and optionally supposing a queue consumer has already started sending some of these emails, how do you recover from the fault and help the user send the rest of the email (without duplicating outbound email)?
If you want the user to be able to try re-sending, you can provide that functionality... you'd need to "cancel" the outgoing messages using a separate queue, re-sending when each cancellation is acked.
What I am trying to uncover here is how one might expect to use a queue, on its own, to support a very-much-non-idempotent interruptible non-transactional process.
Also, what does it mean to "cancel" an already-sent email?
- Unsent: Default status - Sending: Message dispatched to broker - Sent: Send task/message ack'ed - Error: Send task/message any other exception
This of course assumes that your 5 000 emails are not ephemeral and are in a database.
If by "doesn't support any transactionality" you mean ack's aren't possible for one reason or the other then of course you have to go a slightly different route (pun somewhat intended) by publishing a "result" task/message and updating your database rows based on what comes back on this new "task_result" queue.
Of course polling is never a great design, but I guess that's the crux of his question.
Anyway, there're ways to address the problem. Message bus can certainly help.
If the message bus supports queuing transaction, great. Just mb.beginTrans(), mb.queue(), mb.queue(), ..., mb.commit(). The consumer won't be called until the whole batch is committed. When crash, the whole batch is aborted. Presumably the user is notified the command failed and retry is needed.
If queuing transaction is not supported, there're several strategies to deal with it.
1. First method.
1a. Pack the 5000 email id as one message and queue it. This is like squeezing everything into one transaction.
1b. The consumer unpacks the message onto the list of id, walks the list with a cursor, and processes each id.
1c. After processing each id (sent email), saves the cursor pointer, in a file or in a db table.
1d. After a crash, the message bus will re-deliver the whole message to the consumer. Unpack the list. Read the saved cursor pointer to see where the last processed id was. Set the cursor to continue beyond that pointer.
1e. In the worst scenario, the crash occurs after the email is sent but before the cursor pointer is saved. At recovery, a duplicate email is sent. Email cannot be rolled back so it really cannot participate in a transaction. That's nothing you can do about it. It's an acceptable business wrinkle.
2. Second method.
2a. Queue the email id one by one, attach the user's session id or some pseudo transaction id along with each one (e.g. today's date as the pseudo id).
2b. On a crash, ask the user to re-submit all the email id again, and queue the email id one by one again, with the same pseudo transaction id. You might have duplicate entries queued up.
2c. Have a table recording the processed id's.
2d. The consumer picks up the email id and the pseudo transaction id. Consult the processed table to see if the (email id, pseudo-id) has been processed. If so, skip. Otherwise, process it and save the id in the processed table.
2e. Again in the worst case, a crash occurs after an email sent but before saving the id to the processed table. In that case, a duplicate email is sent out.
3. Third method.
3a. On some task command that's transactional, unlike email, you can set up a transaction to do the task command and updating the cursor pointer at the same time (with method one). A crash would roll back the command and roll back the cursor pointer update.
3b. Same with method two. Set up a transaction on the task command and the insert of the id into the processed table.
3c. If the transactional task command is performed on a separate system, two-phase commit can be used to bind the command and the cursor pointer update (or processed id insertion) together into a transaction.