Postmark introduces easy inbound email processing for all accounts | Hacker News Reader

Postmark introduces easy inbound email processing for all accounts

blog.postmarkapp.com

162 pointsalexknowshtml62 comments

Some feedback: This is a nice addition to a mail-provider API service, but the way it works is way too complicated. All email I want to parse needs to be forwarded to my special postmark inbound email address? This might be nice for people that have their email setup through google apps (which kindly forwards on all headers, etc when an email is auto-forwarded with a filter), but for other mail systems they might not play so nice with forwarding headers, etc (I know this from running my own email servers for Notifo and dealing with incoming and outgoing email on my own).

With this solution I still have to setup an intermediate mail server solution to first receive the emails from users, and then forward it along. Can't you take that step away from me?

Other providers (mailgun, sendgrid, etc) let me do that and let users send emails directly to my inbound mailboxes (which are then POSTed to my endpoint).

Also, this excerpt from the documentation lowers my confidence that you know what you're doing:

"Rather than leaving this information in plaintext, we recommend using SHA1 or MD5 to encrypt the information after the + sign and include the encrypted value after the +. This will avoid unintentionally exposing any sensitive information."

Thanks for the feedback, Chad.

Regarding forwarding: that step is already being taken away - it's in beta for a number of our customers, and we'll be adjusting workflow then releasing to everyone.

There's lots of people who don't have access to their DNS records but still want a tool like this. We want this utility to be simple for more them, too!

As far as SHA1/MD5 encryption - we know better than to use weak encryption on any TRULY sensitive information. In the context of the walkthrough, we're talking about internal IDs and other things that you might want to be obvious to your customers, not things that pose a security risk to you or them. If you even thought about using something like that as part of a unique identifier in your email replies - I think there would be bigger problems at hand than choosing SHA1 or MD5 to obfuscate that information. :)

Thank you for the reply!

Regarding the md5/sha1 stuff: The documentation as it stands is both wrong and misleading. MD5 and SHA1 are _hash_ functions (not _encryption_ functions), so while trying to suggest that people could encrypt information to put after the + sign is a good idea, you will utterly confuse newbies by giving examples of hash functions. Of course you could use a unique hash value after the +, but then you need a reverse hash lookup table in your app to correlate the hash and the internal app state/data you need (but that is a whole other beast and not nearly as simple as "just encrypt it in the to address, decrypt on receipt, et voilà!").

In the end it is confusing for beginners and outright wrong for people that know the difference.

Now for the constructive part of the criticism: Please change or improve that portion of the docs, since docs are arguably the second most valuable asset to an API-provider (right behind the service itself).

Wouldn't it be great if there was a standard JSON representation of an email, and that multiple providers (SendGrid, AWS SES, MailJet, ElasticEmail, etc) all then provided such an interface.

So not only could you choose (swap out very very quickly) your outbound provider, but also your inbound provider.

After the thread the other day on outbound email, I ended up choosing MailJet who have been wonderful with the questions I had when I hit their new member ceiling within the first few hours.

The things that convinced me on MailJet were real-time outbound stats and meaningful data (recipient + subject), API of status notifications (who bounced), and the fact that they do campaign emails at a cost so low that MailChimp ruled themselves out (one of my email lists has over 23,000 members).

I need inbound handling (currently using /etc/aliases and piping), but would love all the reporting and info which centralisation from an external provider could deliver.

My only concern would be resilience. Email is really good at re-delivery when your end is down or something prevents delivery, would a HTTP REST interface offer the same level of re-try and timeouts to ensure it got through eventually. Would the provider queue for me?

We have plans for handling retries and making sure that the HTTP interface with your app is reliable.

As far as stats go, that's on our list as well.

I need this. Given credentials, go get IMAP and dump everything JSON. This would be brilliant. Anyone up for a hackathon?

Hey David, what kind of stats did you have in mind on inbound?

The topic is a constantly discussed one, however at some point you cross the line from handling email to handling CRM related tasks and there are better tools/patterns to dealing with that.

I'd love to hear what you have in mind: oren@wildbit.com

As a SendGrid Support Technician, I can tell you our Parse API retries for 72 hours, same as our outbound mail management. We use POST params, not JSON: http://docs.sendgrid.com/documentation/api/parse-api-2/ Our Parse API is complimentary, and doesn't use any credits.

We also have POST pushing of live data such requests received, deliveries, bounces, clicks, and opens down to address/link/time-specific detail. This is offered as either JSON strings or POST params: http://docs.sendgrid.com/documentation/api/event-api/

- Jacob

My only problem with your model is that it requires me to be already paying you a non-trivial amount of money (for a brand new app) to get started with your Inbound API.

Everything you're asking for (and much more) is handled by Mailgun (full disclaimer - I work there):

http://blog.mailgun.net/day/2011/11/07

Moreover, our incoming mail parsing doesn't stop at simple MIME parsing, we're dig deeper into the meaning of those characters and extract user signatures, quoted parts, etc.

Moreover, we retry inbound POST, allow you to filter incoming traffic on the server and (very important) we're the only incoming mail API processor that generates proper bounces for 3rd party servers (spammers) trying to hammer you with non-existent addresses.

We use Mailgun at UserVoice.com for incoming mail.

Of particular help to us:

- All emails are converted to UTF-8

- There's always a 'plain' (text only) version of the email sent to us, even if the original email only included HTML.

- They offer a 'stripped-text' field, which lets us easily see the new part of an email thread.

- Pretty great support

$1.50/1000 seems steep and could hurt should a spam attack occur...Someone needs to package charged outgoing, with free incoming. Kinda like the cell messaging systems they have in the UK...

Remember, you can add value by combining valuable services. Sell one as a loss leader to bring in the users and the other is your revenue stream.

You could always use Postmarks spam checker service to monitor emails and if you get say 2 or 3 from the same email address that flag up as spam block the email?

http://spamcheck.postmarkapp.com/

We actually include spamscore information in the parsed emails.

http://developer.postmarkapp.com/developer-inbound-parse.htm...

Also, if you use Gmail forwarding to foward emails into Postmark, you get the benefits of their spam filtering before it even hits our system.

That said - we watch for spam-like activity across our entire system.

Assuming I could craft messages which did not look like spam (to your system), and I could do it really really fast, do you have any throttling options to allow your customer to prevent an excess of mail from adding up too quickly or an alert based on a threshold? (Maybe i'm overthinking it, but I like services that put me in control of how much money i'm spending instead of being at the mercy of botnets)

"Unusual" send activity is pretty easy to spot, especially if it's going to put our customer at risk in any way. We're extremely proactive about making sure that you're aware of anything that might be considered a "surprise" to your service or your wallet.

My first thought: How would this service handle 100k forged emails sent from china? The recurring payments is okay if you set maybe a daily max, example: charge me indefinitely but no more than a dollar a day. If my daily quota is hit then cut that email address off until the next day.

Do they (or a similar provider) already offer a better solution? I didn't find anything in their docs but I might have overlooked it.

As I mention here: http://news.ycombinator.com/item?id=3476311

We use a combination of algorithms and humans to determine "unusual" activity (not just spam or forged emails) and act accordingly.

We're even proactive about finding customers who break into new send volume thresholds and offer them discounts. We'd much rather have high quality senders in our network for a LONG time than just keep the highest billing rate on autopilot.

Lamson makes email processing easy - it is a very modern framework for writing email applications. I have used it for a while now (almost a year) and have not had huge problems. I have written about it here (including code samples): http://amix.dk/blog/post/19608

Lamson is a great project but it has a number of issues with handling international emails, try handling mails that have subjects in two different languages, or have To/Cc fields with non-ASCII names in them. Basically if you're using it you'll have to guard it with some additional code.

I read though the docs and couldn't tell: is delivery reliable? If my app is down and I miss some POSTs, will I be able to tell? Will they be re-delivered?

I'm currently using Exchange to get real-time mail notifications and nice parsing, but am looking for other options.

Also, encoding attachements as base64 is an interesting choice. I'd prefer a URL where they can be downloaded instead.

Your activity feed will show that there was an HTTP error.

We're working on the workflow for retries, it's coming soon.

Thanks for the feedback about the download URLs. Any particular reason for this preference that I can share with the team?

Photos and other large content. Most web stacks don't nicely handle large requests like that, and to the best of my knowledge streaming JSON parsers are both young and rare.

Totally fair - thanks!

This is awesome, will play with it when I get a chance.

As an aside, I've begun to think that email is/should become a platform for app development. Some sites are already using incoming email for active user engagement (posterous, idonethis, etc)

I think it's a great avenue to fight app/user fatigue. Most people constantly check their email no matter what, but few are making it past the app download to regular usage. The assumptions of emails limitations should be rethought.

Parsing email is a real pain point, kudos to Postmark for trying to solve it. However, I need this functionality in a more abstracted way, not locked to a email address. I'm thinking about taking a swing at it myself and GPL'ing it. (Basically full fledged IMAP->JSON.)

For a real world example, at my startup https://zapier.com/ we have "new email received" or "new email tagged in Gmail" as one of the many, many inputs which you can map to any other write (IE: Tweet something, add a note to Basecamp, add a contact to Highrise, etc...). However, this means I need to log into a IMAP account, not receive an email at a predetermined address. Parsing that raw email is an absolute bear (even with Python's built in IMAP and parse libraries).

Anyone heard of a service or library like that?

http://lamsonproject.org/

Looks interested, without having the chance to give the docs a one over, does it do IMAP as well? Looks more focused on SMTP/sendmail.

At it's core Lamson is just a really sweet wrapper around smtpd (http://docs.python.org/library/smtpd.html) using asyncore for asynchronous socket IO.

When a mail is received you can optionally process it and when done processing you can drop it on the floor (e.g. you're done with it) or you can send it to a 'relay' which means a IMAP or POP server (or really anything) if you so choose.

I would generally say that yes, lamson is much more focused on receiving email via SMTP and doing [smart] processing on it.

I would imagine that its parsing functionality could be a boon to someone wanting to do something higher level with IMAP. Thanks for the response.

that is awesome! I'll be watching this project. lots of potential when it matures.

I always laugh when I see somebody say: "…uses friendly regular expressions".

Lamson seems to be a solid framework, but after trying to use it for a project I dropped it. Two assumptions it makes didn't work for me.

First, it takes over port 25 as your default SMTP server. This is great in that it saves you from dealing with the messy world of aliases etc, but not great on a shared host that does other things with mail as well.

Second, the FSM routing, while convenient, was ultimately limiting. Lamson routes mail based on the state of the sender (for example, 'subscribed', 'new', etc). But this state storage is abstracted away -- so if you wanted to change an address's state outside of the email flow (for example via a web app) or append additional data to the state, you have to re-implement the model logic for the FSM. I wonder if this is the reason why librelist, Zed's mailing list server implemented in lamson, has no web interface for subscription or list creation.

Lamson doesn't help you with parsing email any more than Python's standard library does (which is pretty decent, if a little verbose), so if you can handle setting up routing/aliases of mail to your application and storing state yourself, it might not add much.

http://context.io/ provides an API for looking at the contents of a email account through IMAP, if that's what you're looking for.

At SendGrid, we offer watching a whole dedicated domain/subdomain. For instance, you set up the MX record for incoming.example.com to sendgrid.net, and we Parse everything and POST it to the URL of your choice.

We've got more in store for this feature set, but we're really excited about this as a start - we use it ourselves at beanstalkapp.com and I know that a number of people HN were using the beta.

I'm happy to answer any questions, or field any ideas you might have.

Hey, finally it launched, will definitely come in handy.

I found a bug in the docs: On http://developer.postmarkapp.com/developer-inbound-parse.htm... the link named 'API developer email list' links to 'file://localhost/Users/alexhillman/Sites/wildbit/postmark-design/docs/groups.google.com/group/postmark-api-developers/'

Thanks for letting us know! Already fixed.

Good thing I don't keep docs in my pr0n folder. That would've been embarrassing.

Fantastic! I'm very excited to try this out. I've been using postmark to quickly build out my MVP for a new site I'm working on. This is just the other side of the equation and I am very eager to start playing with it. Integrating with their nodejs library was literally as simple as putting the library in the right place, adding the api key and calling the function. Could not be simpler.

As much as I like the simplicity of Postmark, their pricing makes it very hard for me to justify using their service. Sendgrid and Amazon SES are charging $0.10 per thousand. Postgrid is 15 times more expensive.

If your volume is ginormo this might not be the right solution for you. But as someone who simply needs the functionality and as minimal an install footprint/learning curve as possible this is really hard to beat. Hopefully my startup will take off and I will have to look long and hard at a replacement.

Also, MailGun is $1 per thousand. Which seems to offer a similar proposition.

I'll point out that we have no "account minimums", only the $1.50/1000 email credits for both inbound and outbound that never expire.

Also, we're home to many happy customers that are sending very high volume - we offer them bulk-credit purchase rates as low as $0.50 per 1000 when buying 2MM credits at a time or more.

This is great! Perfect timing too. I've been using postmark for over a year, and have recently been looking to other providers for an inbound service.... soooo glad they did this!

Try EmailYak. It is simple and elegant http://www.emailyak.com/ It is my friend's but I'd still recommend it otherwise. See: http://docs.emailyak.com/quick-start-guide.html

It's also quite pricey - same price as Postmark at its cheapest, and $5/thousand at the first pricing tier.

There are a subset of applications that imo could rely exclusively on email as the platform.

A few months ago I built such an app, a small side-project called betabrokers(email trade@betabrokers.com with subject:about to signup and learn more). The simplicity of the signup process for this type of app (simply sending an email) is a huge benefit.

Email-based apps especially make sense in a mobile context, which is somewhat ironic because the transactional nature of applications built on them would suggest that they are slower, therefore less likely to be used. In our case however, users keep coming back to it, probably because the email client is the single most used app on their phone, and receiving replies to your actions via email becomes addictive.

We've been using Postmark for quite a while now (maybe 2 years now?) and it's fantastic. We recently began using the inbound processing while it was in beta, and it too is fantastic. The API is great and it's absolutely painless to setup and begin using quickly...

Thanks for the kudos, Jeremy. If you can share where you're using the inbound API, I'd love to see it in action!

This is neat, but it's not too difficult to handle incoming mail yourself with postfix and a simple ruby/python script to process it. The learning curve is slightly steep due to postfix and some really bad suggestions, but the actual work is straight forward.

I setup a simple ruby program to pipe the postfix email into a resque queue (redis backed ruby jobs system) to be handled by my work pool...a couple easy regexes determine what the email is: bounces get auto-delisted after 2x, unsubscribes delist immediately, replies get shoved into the correct in-site comment thread or conversation, etc. This is a day of work....

You are correct, of course. Nothing stops you from doing this yourself. The basic functionality in SaaS is always easy to accomplish (hosting, email sending, dropbox, etc). Its the little things that get you and eventually drive you to the bottle.

Interestingly we just had to build this type of processing into one of our applications, and I would have much rather sent this through a simple API.

Alex are there high volume discounts?

Yup, the same discounts apply for volume credit purchases.

Our credits work for inbound and outbound and never expire.

* 500,000+ - $1/thousand ($500) * 1MM+ - $0.75/thousand ($750) * 2MM+ - $0.50/thousand ($1,000)

Sending even more? Email me - alex@wildbit.com and we can talk.

I've been using this amazing service http://mailnuggets.com for a while now (6 months or so) and the guy is absolutely fantastic (I emailed some ideas and he implemented them :D) buuuut I use Postmark for outbound email, it would be nice to consolidate everything into one account. Will give this a try. I love postmark :-D

Google apps + configured filters + inbound email processing = awesome possibilities.

This got me thinking... Are there any startups that are doing something similar with physical mail? I can picture a large centralized mail room with a bunch of industrial sized scanners, mail gets opened, scanned, and uploaded. Customers could access pdfs of their daily mail through a Web interface and could search through archives. Through in some super OCR and I think you'd really have something...

You know, when I started working on Postmark I thought that would make a fantastic April Fools joke.

then I discovered there really is a company that tries to do that. I can't remember the name though…

Earth Class Mail?

http://www.earthclassmail.com/

I'd love an email API that you could forward an email thread to, which would be parsed into a set of linked emails with contacts attached. Email to JSON is great, but not when the body is the text of 20 forwarded emails. Seems like a gimme for a CRM app -- does anybody provide this?

This is a cool idea, we'll keep it in mind!

How well does this deal with the cruft of a "Replied to" email

Are there particular requirements/restrictions for emails to be robustly parsed? Even Facebook had problems early on with their email-response parsing.

I can see how this would be helpful but it seems like the best way to do this would be with a POP/IMAP client in your code. But I'm guessing those libraries are probably not as user-friendly as one would hope?

We use google for our domain email. In order to handle uploads via email on this same domain we simply create a new account and send emails to this account. You then can have a hook (imap push) in google imap to detect when an email is received per account/box and process is right away. Or you can just run a cron every minute the check if new emails have been received. We then fire a script that pulls the email via imap, and then processes the emails/images into json format for use.

This allows us to keep everything on google and we do not need move our MX somewhere else. Also we keep our "upload" email on the same domain.

Pretty simple solution with no additional costs.

Reply on news.ycombinator.com