At the same time, if you are willing to install and manage Postal on your own servers, it's not that hard to maintain your own IP with a great reputation. You just need a good hosting provider (probably not AWS), you need to set up your infrastructure like DKIM, SPF, DMARC, rDNS, and Return Paths, and most importantly you need to maintain good engagement (low bounces, high opens). At a glance, Postal looks like a nice option if you want to do it on your own for cheap. You just might lack the stability, support, maintenance, and performance that goes behind an ESP.
This is probably a key element of good performance. To keep my mail admin duties part time I simply whois the IP of evil senders and drop the resulting entire CIDR block into the our local blacklist.
A couple years ago I did the work to use "Mandrill" in an app and they got merged with "MailChimp" who changed how my app could use their service and I had to redo those routines again.
Right now I'm using Mailgun and they're awesome.
* You get IPs from a shared pool, and the reputation is nowhere near guaranteed. In fact, many of my mails were blackholed or rejected.
* The "bad" IPs that were used by someone for spamming are not immediately removed from the pool, so you will encounter them.
The net result for me was that Sendgrid is not a solution for my transactional E-mail, because of the high risk of my E-mails getting blackholed/rejected. I use it for newsletters/marketing only.
Given this, and my volumes of mail were increasing (once you go over 100k emails/mo SendGrid's pricing goes up a lot), I setup my own mail server using Cuttlefish [0]. I set it up on an OVH instance that I pay £2.50/mo for. I contacted them to enquire about spam policies before opening an account, and they said they take it very seriously and monitor outgoing SMTP for spam.
The IP address I ended up with was still on one blacklist, but the process to remove it was pretty easy (fill out a form, and explain the situation) and took about 1 day. I set this up around 6 months ago and have had no problems with deliverability since then, I'm now up to sending around 300k emails per month.
I've heard this many times, and maybe it was true before but it doesn't seem to be any more. I suspect now companies like Sendgrid still spread this FUD so people are more willing to buy their services and assume it's the only option.
And I don't show any bounces either. Stuff getting sent to spam doesn't give you a bounce message usually. It will appear to have been delivered successfully.
As for needing to warm up the IP, I think there is still something to that. I am surprised you were able to just start right off sending large volume. My guess would be that that you lucked out and got a gently used IP address by chance. The fact that it was on a blacklist would indicate that it probably was previously used rather heavily... just enough to get blacklisted at one place. But not abused in a way that got it blacklisted other places.
https://cuttlefish.io https://github.com/mlandauer/cuttlefish
https://github.com/jimktrains/email_ng is an overly complicated idea I had once, but the basic gist is that the receiver could give the sender a signed receiver email address that is for the sender's email address. This way, transactional email (and sure, I guess marketing from your company) would be able to get through and the email given out wouldn't be able to be used if sold or stolen, as the receiving mail server would reject it because of a bad signature (bad sender, and coupled with DKIM and SPF, a malicious user wouldn't be able/allowed to spoof the sender's email).
Your proposal requires at least the status quo (aliased email address at signup), but the problem is it also requires cooperation of the receiver's email provider (e.g. gmail) and the sender's email provider. You are unlikely to ever see any new standard adopted by all major email providers, unless it came through a standards committee.
But your idea is interesting. Perhaps you could lower the cost of adoption by replacing the dependency on TLS/SSL with some sort of pgp signing scheme. This way all the "protocol" happens within the message body, so email providers do not need to adopt a new standard. As long as at least one website and one user implement the protocol, it can work without any cooperation from third parties.
As an aside, it would also be nice if password managers included functionality around generating temporary/isolated email addresses.
It’s not a comment. Some mail services allow you to have aliases in the form of normal-username+something@service.com but that doesn’t mean all do. If it were a comment the address +@example.com would be invalid but it’s not. Comments in email addresses are written in parentheses like username(i'm_a_comment)@example.com.
I rarely receive email spam these days. The spam I get is on my voice line. I'm getting several robocalls per day. If you have the skills to work on this stuff, I'd love to see a solution to phone call spam because email spam feels solved.
we definitely need a better solution to fighting spam.
For phone calls, somewhere between 30 and 50 percent of my calls are robocalls or other telemarketing (I receive far more emails than phone calls).
If there were an easy way to tell my phone to just drop callers if they aren't in my phone book, I would. Right now, my phone lets me know if a caller is a suspected spammer, but when I hit dismiss, it goes to voicemail. I'd rather have the call just dropped or for the caller to get a busy signal. That reminds me - I can't remember the last time I heard a busy signal.
My greatest fear is that Google and Facebook will co-opt email and charge money for companies to get access. Google has already made some progress with this by selling ads in Gmail that look like email while pushing commercial messages in to other inboxes.
This is bad because it penalizes companies with low margins while shifting the advantage to those who are able to squeeze more money out of the users (you can see the effect of this on Google Search, Facebook, and now Amazon. It isn't pretty.)
A DIY home mailserver will still let, like, 5-15% of spam emails slip through.
The last few times I went to spam and saw emails that shouldn't be in there, a careful inspection revealed they were actually very well crafted phishing messages.
Ever since, I don't bother checking the spam folder. Too dangerous. Gmail is smarter than me.
I'd start with the "email postage" proposal—the one where either a proof-of-work or physical payment (which can also be digital, e.g. a Bitcoin private key) is required to get an MTA to take your message. This proposal was declared unworkable due to rendering high-volume transactional email impossible.
But then:
1. as you say here, divide everyone's email account into a set of "channels" (i.e. subaddresses in a+b@c form, that each have a corresponding message signing key), with one or more published or well-known channels, and then individual private channels for each contact/list, or for each conversation(!);
2. make MTAs aware of "channels", and extend both SMTP and email web services to allow users to configure their provider's end-of-line MTA, over-the-wire, to set the amount of "postage" required to message each channel they own. (This way, the MTA doesn't have to be aware of the distinction between private and public channels; they're just destination addresses with a stored config parameter.)
3. make the MTAs that use channels, reject any Internet message directed to the base address without a channel. (Messages generated by the MTA itself can arrive without a channel.)
4. Set the clients' default new-channel configurations such that public channels have a cost, and private channels are free.
5. In email clients, add a "Subscribe" or "Allow" action-bar item to messages that identify themselves as email-confirmation/newsletter-opt-in emails (presumably with a header), that, when clicked, creates a new channel for the sender, and replies to the message with the signing key you want them to use attached. (All this would be hidden from view; the reply message wouldn't end up in your Sent Items.)
6. In transactional-email-sending services like Sendgrid/Postmark/etc., create a distinction between "opt-in messages" and "ongoing transactional messages"; allow each account to send one "opt-in" email to each previously-unknown destination-address (and charge for this); but then, for that account, put that destination-address into holding state, where the account can't send them any "ongoing transactional messages", until the email-sending service receives the user's conversation key. (You probably want to allow accounts to re-send the opt-in email after a 24-hour-cooldown, though. Though they'd have to pay again!)
7. In any product/service backend that uses a transactional-email service, consider new signups unconfirmed until the transactional-service reports (by polling or webhook or whatever) that the user has replied with a key and is now in the "authenticated and free to send to" state.
A bit complicated in the initial changes, but in the ongoing state it's nearly ideal: it costs money to initiate contact with an address, or to continue pestering an address that doesn't want to speak with you, but not to send messages to someone that wants to hear from you. (Though, a user can "unsubscribe" from your list simply by telling their MTA to begin charging you to deliver to their channel again—maybe, UI-wise, by just deleting the channel. Transactional-email sending services could detect the bounced-with-payment-required delivery error, and put the user automatically into an "unsubscribed" state, which could webhook you to let you know!)
https://senderscore.org/ per https://news.ycombinator.com/item?id=9156312
http://whatismyipaddress.com/blacklist-check per https://news.ycombinator.com/item?id=11329371
http://multirbl.valli.org/ per https://news.ycombinator.com/item?id=11330382
This is because the big consumer mailbox providers often don't rely on public datasources for assessing reputation, and because reputation is tracked at the domain level, in addition to - or sometimes instead of - the IP level. For example, there have been some indications from Gmail that they no longer use IP reputation at all, starting a year or two ago.
And re: blacklists: +1 to Spamhaus, but in practice it's one of very few blacklists that have meaningful impact to net delivery.
I can believe it. I posted already about my experience with the free tier of Mailgun's service. I went through all their instructions about setting up the service and verifying my domain etc. but still had a lot of email rejected, especially by yahoo.com but also hotmail.com due to poor reputation IPs
Gmail.com typically delivered everything.
(I do not have anything to do with the product or team behind it)
(I am likewise unaffiliated.)
In years past, many RBLs would categorize IP Addresses by type. e.g. dynamic IPs assigned to DSL/Cable subscribers. This would enable a receving SMTP server to check if the email came from an ISP subscriber, rather than an email server. If so, it was usually a good guage that the email was "spammy", because it was sent by a subscriber's infected computer.
In this sense, it was (and may still be) possible to simply block, or at least score differently, email from any/all AWS Elasic IPs.
Neither, though, helps with checking if one of the big providers, like Google, would not like your IP. For those, you have to send an email and watch/parse the SMTP error responses.
I have found their nighttime support team to be terrible though, resulting in 14hrs downtime because they didn't have authority to re-enable my account.
I'm unsure what problems Sendgrid is battling but their customer support staff is certainly understaffed or they simply gave up on fighting spam. So yes if you want to send spam and can start with highest paid account they offer, probably Sendgrid is your best bet.
Also their spam@ and abuse@ is a waste of time. It came to point that they simply started ignoring my inquiries at all! Here is example of one of my emails that is not getting any response, if Sendgrid is actually reading this: support+id1021573@sendgrid.zendesk.com
As sending spam is a Federal Crime, I have already reached out to FBI, FTC and next plan to write my State Attorney General.
If you are seeing continuous problems, I'd love to take a look! My email address is on my profile.
In the case of Mailgun, you should be assigned an IP with a neutral reputation. For example, before dedicated IPs are reassigned we leave them dormant for at least a month, usually much longer, before assigning to a new customer.
Mailgun will do some chores for you, for a fee. If those chores are things you can do in your sleep, fine! Mailgun does nothing magic.
I gave up and decided screw them, I don't need to talk to anyone with an ATT address, and bounce messages from them with an explanation.
- You have to ramp up sending from new IP addresses VERY slowly if you want to avoid getting grey / black listed by Yahoo and AOL, this makes it quite a tedious process
- Make sure you also have rDNS set up, Gmail uses it as an important quality signal
- Accurately handling bounces is a bit of a nightmare because each mail server has a slightly different format and you want to make sure you don't process out-of-office messages
- Bounces and "message delayed" messages can arrive days later
So we ended up switching to mailgun and although they do sometimes give you a bad IP the time saved is worth it, especially their option to have a web hook back to your backend to process the bounces.
Until then, all is fine I agree.
Why does Gmail hate my domain? | https://news.ycombinator.com/item?id=9855030
How to Avoid Spam Filters | https://news.ycombinator.com/item?id=10465639
If there are others I'd appreciate a link!
Of course, if it's that important, then you'd be foolish to not have invested in a better solution, or to not have a proper SLA with a third party.
The answer to why better spam filtering AI hasn't been developed in a non-proprietary setting is fairly straightforward: the success of any filtering method is heavily dependent on consuming data at scale - including recipient behavior (e.g., mark as spam, vs. spend time reading a message), and this underlying data is just as important, if not more so, than the machine learning approach used to compute filtering assessments.
In short - you more or less have to be a large mailbox provider in order to have a chance at doing this well.