Ask HN: Email handler | Hacker News Reader

Ask HN: Email handler

18 pointssnprbob8611 comments

I am currently building a Django web application which involves listening for email. I looked high and low, but couldn't find an existing solution for implementing automated email handlers in Python. So I decided to build one. I'm still hoping there is a great library out there that I missed. Does anyone know of one?

Getting a simple handler up and running was easy, but I have run into some trouble with graceful handling of error conditions. For example, I don't always get SMTP error codes when sending mail to bad addresses or full mailboxes. Instead, I often get non-delivery receipts as emails a while later. Unfortunately, these NDRs appear to differ wildly by server and do not include structured or standardized meta-data such as the message ID that was not delivered.

I noticed some striking similarities in requirements between the system we are building and Anyvite. Anyvite automatically expanded a gmail address to a full name and parsed a non-delivery receipt (try sending an invite to a known bad email address). I am curious how they accomplished those two tasks. Since Anyvite was a Y Combinator company, I was hoping one of those guys could comment too :-)

Thanks!

Do what ezmlm does with bounce messages: use a different envelope sender for each outgoing mail and encode the message ID in that address. If you get a mail to the envelope sender you can typically assume it's a non-delivery report of some kind.

For finer-grained information, you can search the resulting message for HCMSSCs: http://cr.yp.to/proto/hcmssc.txt

Further reading: http://cr.yp.to/proto/verp.txt http://support.kavi.com/khelp/kmlm/user_help/html/bounces.ht...

This is certainly interesting. In the grand problem of making sense of bounce messages, there seems to be three sub-problems:

1) Determining if the message is in fact a bounce. VERP doesn't seem to solve this.

2) Correctly linking a bounce message to the original message. VERP seems to solve that via encoding the message ID into the return path. I could implement VERP with "plus addressing". Thanks!

3) Identifying the error that the bounce message represents. Unfortunately, the presence of HCMSSCs in the hash format does not seem common. Even Gmail's bounce messages do not provide them.

Hey, sorry we didn't respond earlier, just saw this post and the Anyvite reference.

Our system is very basic right now and needs to be expanded on in the future. We start by setting a return path on all emails we send. This directs any bounce messages to a single mailbox that we can then process.

Parsing the bounce messages for us is fairly easy. We took some ideas from David Rusenko's post at: http://david.weebly.com/1/post/2008/06/startup-email-monitor...

You seem to be trying to tie a bounce back to a specific outgoing message, which may or may not be the right thing to do depending on your application. For us, we only care about the email address it was sent to so we know not to send them anything else (if it's a hard bounce).

Best of luck.

Thanks! I sent a longer note via email :-)

You're looking for this:

http://www.smtp2web.com/

They listen for mail and hit your site with HTTP POSTs whenever they show up.

smtp2web solves the problem of getting the messages.

As far as I can tell, there are 3 ways to get access to the messages:

1) Control your mail server, use something like procmon to forward mail to a script

2) Rely on smtp2web or some other mail server to forward mail to a script

3) Poll for messages via SMTP or IMAP

I chose #3 with IMAP over Gmail (configured for my domain).

The real problem I am having is identifying and categorizing non-delivery responses / bounce mails.

I assume you mean procmail, not procmon. Decent MTAs (like qmail) make it pretty easy to have mail delivered to a script.

You assume correctly. My day job involves a lot of Windows programming and a healthy dose of procmon use :-)

I would use procmail (http://www.procmail.org/). You might also check out the comments at this older post: http://news.ycombinator.com/item?id=186875

I looked briefly at procmail, but it seems to only have a primitive mechanism for identify bounce messages. There does not appear to be a way to extract meta data from them.

""" If the regular expression contains ‘^FROM_DAEMON’ it will be substituted by ‘(^(Mailing-List : |Precedence :.(junk |bulk |list) |To : Multiple recipients of |(((Resent-)?(From |Sender) |X-Envelope-From) : |>?From )([^>][^(.%@a-z0-9])?(Post(ma?(st(e?r)? |n) |office) |(send)?Mail(er)? |daemon |m(mdf |ajordomo) |n?uucp |LIST(SERV |proc) |NETSERV |o(wner |ps) |r(e(quest |sponse) |oot) |b(ounce |bs\.smtp) |echo |mirror |s(erv(ices? |er) |mtp(error)? |ystem) |A(dmin(istrator)? |MMGR |utoanswer))(([^).! :a-z0-9][-_a-z0-9])?[%@>\t ][^<)](\(.\).)?)?$([^>] |$)))’, which should catch mails coming from most daemons """

I still need a way to get meaningful information from the bounce messages.

For example, my system sometimes emails a file on behalf of a user. If I receive a "mailbox does not exist" bounce, all I can do is notify the user that they may have mistyped an email address. However, if I receive a "mailbox full" bounce, I want to try again without the attachment and only with a web download link.

Not python, but simple and works great: http://subethasmtp.tigris.org/

Reply on news.ycombinator.com