Giving away the secrets of 99.3% email delivery
37signals.com
37signals.com
Almost every spamfilter in the world will accept your e-mail and then silently discard it. The 0.7% instant rejects that you see are merely the tip of the ice-berg.
Thus, the only secret you gave away is that you don't seem to understand how spamfilters work.
37signals has an easier time of it because the vast, vast majority of their emails are transactional, so users will pipe up and complain if they are not getting them. Marketing emails are a lot harder to troubleshoot.
And before the masses get up in arms about spam, when I say "marketing" I am talking about even the most compliant, user-friendly, double-opt-in email campaigns. Even if people happily sign up for your emails, that doesn't mean they will tell you if they stop receiving them.
Plus, when they decide they don't want it six months later, they'll use "Report Spam" to stop them from showing up in the inbox.
"We do track open rates for emails that are already HTML formatted and making remote requests for images, but you’ll never get 100% accuracy with that metric because many people use plain text emails or don’t load images. Our experience is that the best you’ll ever see is between 60-70% “open” rate because of this. Some of our applications in some contexts only send plain text emails as well, so we don’t track open rate there at all.
Why is remote server acceptance rate important anyway? Some thoughts:
1) First, because hard bounces really do happen a lot, and at our scale, a 1% difference in hard bounce rate means 160k messages per week that aren’t making it to users, which means a poor experience for many and many support requests coming in to us. Based on all the information we’ve been able to find, we’re pretty sure a 0.7% hard bounce rate at our scale is pretty good.
2) Second, because it is a relative metric of overall deliverability. Our experience has been that when we do get on a blacklist, servers start hard bouncing our mail until we get off of it. As we’ve improved our SpamAssassin type scores over the last few years, we’ve seen an improvement in hard bounce rate. We also see a strong correlation between hard bounce rate and the number of email delivery related support requests we get. While it’s not the perfect measure, it’s the best measure we have available that we can reliably monitor.
Again, there’s no perfect way that I know of to reliably tell whether an email is getting to a user, since read status isn’t particularly accurate. We use whatever we can (hard bounce rate, open rate, number of support requests relating to email) to get as close to that as we can."
I don't think that's true. I've been actively running email servers for going on ten years now and the vast majority of mail servers do the opposite. They reject at SMTP time. Accepting and then silently discarding email is considered bad practice, and is not as common as SMTP time rejection.
Server-side content-based filter (e.g., spamassassin): usually marks up the email based on rules, and may either reject/discard it, or leave it for the email client
Email-client filter (e.g., in thunderbird, outlook extensions, etc.): silently discards
I've been using Postmark for a while now and it's been fantastic. They provide a great admin interface with insight into what is being sent and why mail might not be getting delivered. The setup process is very simple (none of the downsides 37signals mentions... ) you essentially add some SPF and DKIM DNS entries and you're off to the races.
37S is doing a great job with all of their techniques. Of course, they also have an incredible sending volume on their side that helps improve the impact of any of these techniques.
One of the biggest benefits of our approach at Postmark is that you get those same benefits regardless of your send volume: http://blog.postmarkapp.com/post/14127210172/the-false-promi...
See http://developer.postmarkapp.com/developer-inbound.html for details.
The list immediately after that of some of the headaches that getting e-mail delivered entails (monitoring and responding to blacklists, various configuration, feedback loops, etc.) is a very good argument for paying somebody to learn and handle the details. I don't see a convincing argument that getting 1% better delivery is worth spending time on instead of doing something else; indeed, they make the argument that improving validation and reporting on the app side is a much better use of time than fighting for that extra 1% on delivery.
Because it costs us less and we have better deliverability. Also, 1% of 50 million emails a month is a lot of undelivered mail.
Campaign Monitor isn't what you'd use for your apps, you want something like SendGrid.
SG is probably the most pleasant email-related experience I've ever had.
the discussion around managing email in-house vs. paying a 3rd party to do it for you, is of significant complexity, as is evidenced by the length of the original 37sig post and this comment thread.
i think it's important for each and every company to evaluate their own unique situation - the needs of their product, the resources at their disposal, the role email plays in their overall business model, their relative experience/expertise in email vs. other elements of the customer experience they are building, the maturity of the company/product, etc. it would be very unwise to make a decision on this type of matter, based solely (or predominately) on factors like "successful company X does it this way", or "successful company Y does it that way."
every company is different, and they often face this specific decision at different junctures in their life cycle.
the most useful lesson that can be gleaned from this conversation (which i've really enjoyed following), is this: as a business leader, entrepreneur, developer, etc, you have options! if you want to do it yourself, it's possible -- if 37signals can do it, so can you. but if you don't want to do it yourself, or aren't confident in your ability (for whatever reason), then there are several awesome companies out there to choose from, each of which has its own strengths and weaknesses, which puts you in the position to select the most ideal solution for your unique circumstances.
that's all :)
Obviously the blog post was only talking about how effective it is for 37signals, but I fear many young startups will misinterpret it and waste a ton of time rolling their own mass e-mail solution.
But it gets much harder when you are running with a recently purchased domain on cold IPs or with the spammy subnet neighbors, your subnets are blocked by "know-it-all" small ESP admins.
It is very similar to having a great credit history: you're getting approved for much nicer interest rates, there's no secret. So I don't believe an average person will get 37signals results simply by following Noah's advice.
I work at Mailgun and we help startups and established companies get "37signals level" deliverability :) We also offer quite powerful parsing of incoming email into your app, so check out http://mailgun.net
I wonder if you could hack up an 80% solution by comparing the submitted domain to common email domains, and giving a warning if it looks like a misspelling. That way, 'joe@gmal.com' would see a second step in the signup process asking him to double-check his email address. Any idea what percentage of misspellings that would catch?
$ for t in mx a; do dig +short gmial.com $t; done
0 nullmx.domainmanager.com.
208.87.34.15
74.86.197.160
$One of my consumer-facing websites gets lots and lots of typoed email addresses. Based on the kind of support emails I get, my impression is that the general audience of this site is borderline illiterate.
I mined the user database for common email domains where users had signed up, but never confirmed the email address by clicking on the link in the welcome email. Based on that, I created a bunch of regexps that detect the most common misspellings of gmail, yahoo, hostmail, msn, etc. I also check for things like <domain>.con, <domain>.cm, <domain>.om, and the other various typo permutations.
If the user enters a suspect email address, the system asks them whether they're sure they entered it correctly. In most cases, it will also suggest what it thinks they were trying to type: "You entered example@verzon.cm as your email address. Did you mean example@verizon.net?"
This reduced the bounce rate significantly.
For those cases where I still get a bounce to the welcome email (mistyped username, or a domain I couldn't autocorrect), I have a process that parses the bounce messages and flags the user's account as bouncing. If that flag is set, every page on the site includes a warning box that basically says "hey, your email bounced... please update your email address". When the user updates their email address, the system sends them a new confirmation email.
This email update dialog also requires the user to type their correct email address twice, because at this point they're known to be a bad typist. :) The original signup form only asks for it once, which improves conversion rates over requiring double-entry.
The combination of both of these techniques has reduced my support load for bad email address cases down to basically nothing.
I'm sure that there are a lot of people who would find this valuable and you would gain a bigger dataset to refine your responses.
For everybody else, you can save a lot of time and money by going for Amazon SES, Google App Engine, SendGrid or Postmark (etc.) A lot of these services also include analytics and monitoring and will be cheaper than rolling out your own customized solution (in terms of time and money).
Even for them they would only pay around $6000-$7000 pr. month by using Amazon SES.
SES requires each sending address to be verified upfront. For us that's an issue.
There goes my plan for moving my personal accounts away from GMail before March 1st...
I should not have to jump through non-standardised and somewhat broken hoops like SPF and DKIM just to get a goddam e-mail delivered to my friend. When you reach that point, it's not your e-mail client or domain registration that's broken, it's the receiving e-mail system that's so paranoid about spam that it's regularly diagnosing and (silently) rejecting false positives.
Graylisting + Delayed SMTP prompt + Blacklists + Whitelists works pretty well for filtering incoming spam.
For outgoing mail you just need to make sure that you're not sending mail from a dialup IP range, or some other IP range included in common blacklists. It also helps to register on one of the more common whitelists that are available for free. And of course you need a valid reverse lookup - many mailservers don't accept mails from hosts without them.
Here's a secret - Monitor the ip to domain ratios , usually gmail will allow 1k of mail from the same ip per hour.
Example list of supported backends:
Django's ORM
web.py's simple database library
Tokyo Tyrant
Raw SQLite3
SQLObject
CouchDB
Mongo DB
SQLAlchemy
Let me know if this helps you, if not, tell me why so I can try to figure something out.
So a couple of things I noticed that might be missing from Lamson:
(1) Spam blocking auto-updates -- does this tap into Spamhaus, etc.? (2) UTF-8 transcoding? (3) Signature, quote stripping (like Mailgun). (4) DKIM signing? (5) Simple client libraries for sending mail (i.e., I don't want to build a "Lamson application", I just want to talk to a Lamson server on a privileged port using JSON).
Of course, all these concerns might have answers that I'm missing.
A bit more about my (fairly common, I'm guessing) use case: you have a VPS and a MongoDB instance. I want my MTA to take a minimal set of config parameters, say:
-- MongoDB database name -- domain -> MongoDB table mapping -- privileged API sending port
Now I want the MTA to receive mail for me, perform spam blocking, transcode to UTF-8, parse out signature and quotes in a separate field, and dump the whole object to a MongoDB document with the index field = the recipient's email address. I want it to auto-update it's spam blocking rules using whatever external services and internal analysis necessary. I want it to periodically dump some usage statistics into a separate MongoDB table (maybe a circular connection).
Now for sending mail, all I should have to do is connect to the privileged port from a process on the same machine (it's firewalled to the outside world), and submit an HTTP POST request that specifies recipients, message body, attachments, etc. The MTA should accept the request, perform whatever queuing, rate-limiting, and retrying is necessary to send the message.
All failures and diagnostic information are dumped to separate MongoDB tables. The applications deal with the database/storage system directly, to keep the MTA simple.
I'm glossing over a lot, naturally, but I hope that helps.
With that said, specific functionality is more a question of learning lamson, and tying in libraries that do do what you want, or finding someone that already has on Github.
For your use case, you would be tying together and implementing bits and pieces of this yourself. Part of the reason for Lamson existing is to enable programmers to solve their own problems with a common sensical base to work off of.
It's more of an equivalent to a web framework than a CMS.
If that isn't workable, you'll have to either hire somebody familiar with Lamson to hack it up for you, or you'll be at the mercy of existing plugins to tie an MTA into MongoDB. (Doesn't exist).
Do you have a more specific question for me to address? The answer to pretty much everything you brought up is "Cool, so go make it". The point of Lamson being the way it is, is so that you're not limited by the creator's intentions, just by your programming ability.
LinkedIn noticed that I never read one of the group messages I was getting and so switched to a far more infrequent digest of highlights. I think they may even have unsubcribed me completely from one group.
sadly any properties that use yahoo servers or sympatico.ca, bell.ca, always return "OK" and you physically need to bounce an email to verify it. many of our customers have @yahoo addresses so we still have about 25% that simply cannot be validated without a "probe" message :(
This is standard technique used to prevent spammer from mining the whole userbase.
[1] https://support.msn.com/eform.aspx?productKey=edfsjmrpp&...