Google App Engine Silently Stopped Sending Email 5 Weeks Ago

I've recently compiled a list of free transactional email services with a feature comparison: https://www.metachris.com/2016/03/free-transactional-email-s...

I completely agree with your recommendation on Mailgun. Using it for past 4 months, never had a problem with their service.

Love Mailgun. You can configure it to just be a very easy mail provider or use the APIs to send mail and turn on all kinds of things like campaign monitoring, tags, click opens, etc.

The best part for me when using it for customers has been the bounce/click through rate tracking. When someone asks me, "how do I know they got it?" It's incredibly nice to point them to the dashboard and show a less than 1% bounce rate (people putting in bad email is almost always the reason) with a log of every single email sent.

Most my clients get this service for free because their volume us low enough. They have quite a generous free tier.

+1 as well, after the changes to Mandrill recently, was looking for a new provider and can also recommend Mailgun (at least, for ease and configurability in set up and testing so far, not yet in production).

Thanks for the suggestion! I was having issues with SendGrid and had never heard of Mailgun. Took literally 2 minutes to get set up on Mailgun, add the credentials to my app, and send my first email! It's a super small app so don't need anything fancy, Mailgun is perfect.

edit: Not to say Mailgun isn't fancy, I have no idea if it is or not, I can say that it works in minutes though.

Hi, if you haven't already sent Chris an email, can you send me the details of your account and I'll get answers for you today? lilkim@google.com

You might look at Sendgrid. It's free for those sort of volumes

The recent post about Candy Japan's attempt to combat credit card fraud gives a good sense of what fighting fraud and abuse is like: https://news.ycombinator.com/item?id=11431881

As an email service provider it's like that and more, since (1) once an abuser uses your service, they've gotten the benefit immediately and keep it even if their account is discovered as fraudulent later, e.g. stolen CC number & chargeback. (2) Abusive users can directly harm good users such as by harming the deliverability of the overall platform. It's not just bad debt, it's bad experience too. (3) Unlike Candy Japan where fraudsters mostly just wanted to check CC numbers and not actually buy product, email abusers really want to send emails (4) It can be hard to tell good and bad senders apart because some companies with an internet presence aren't email savvy and might make mistakes or might get hacked.

Spam filters are always tough because if you give someone transparency into which actions of theirs that you consider abuse, then they will quickly detect and route around your attempt to block them. (See Candy Japan article) It's pretty easy for a human to guess what might be the sign of their fraud and run a few experiments to see what gets flagged e.g. By comparison a machine learning system might be hard to outsmart, but then it's also challenging to explain and troubleshoot false positives. Hence what's effective is often a combination of machine-learned filters and heuristics along with manual overrides by human judgment.

All other things equal, new users are a lot more likely to engage in fraud than existing ones, and so tend to be under more suspicion. Aside from B2B fraud where companies take out lines of credit and then go bankrupt intentionally, it's uncommon for existing established customers to turn fraudulent - they're already vetted. (Consider: who is more likely to be fraudulent. The first time subscriber to Candy Japan, or a subscriber who has been using it for 12 months and is about to buy their 13th month?) It's not a great experience as a new user to be under suspicion, but if it's temporary and easily overridden by a human it can be a decent trade-off - the need to reach out acts a deterrent to spammers but does not deter legitimate users as much (speaking generally).

Yep, this has been an unfortunate thing for us as well.

I really have to highlight how dangerous an approach this is. You have no idea if the previous user of that IP was spamming. You should seriously consider mailgun or sendgrid. They also bring a lot of features that enable you to track when an email was read etc.

Infrastructure always changes. You need to have something in place to detect failure of important features of your apps, that get checked every day, every hour or even every minute, depending on how costly an outage is for you. There are always doublechecks possible.

Why do you think services like sendgrid, mailgun etc exist? It's pretty common knowledge that if you need to send mission critical email, you should use a service specifically designed for that. Sending email programmatically directly from a web server has ALWAYS been a gray area.

At the VERY least you could have easily set up a gmail account and sent them from smtp. Choose the right tool for the job.

I think this is a terrible attitude. I will never implement it in my own support model. People shouldn't have to pay me in order to report a bug in my system. Arguably I should pay them, because one report is the tip of an iceberg of silent affected customers.

More often than not, what most people want is simply an acknowledgement. They're not seeking a bug bounty. And definitely not the brush-off you just gave the GP: when someone's house is on fire, you don't demand prepayment for the water.

I can't comment on the paid support. I've just noticed that the non-paid support engineers seems to be under pressure.

I feel little bit bad for them sometimes. They don't seem to have the time to look into problems. It must be quite a boring job for those assigned on the public issues.

Maybe their higher-ups aren't aware of that.

Paying for the service should be enough to get support.

We wouldn't use it.

To me, the whole point of Appengine is to abstract out system administration. GCE (like AWS) abstracts out the systems, but not the administration - you suddenly become responsible for all the scaling, fault-tolerance, and all the subsystems that Appengine manages for you.

I wrote about this last year, but I feel it's still relevant. Spam filters, especially Google and Microsoft's (whose e-mail servers don't play by any of the standard rules nor do they send DMARC reports), are horrible over aggressive.

http://penguindreams.org/blog/how-google-and-microsoft-made-...

I have seen more phishing/ransomware e-mails come through mine (lots of malicious Javascript made to look like Excel files), so I can understand how great the risk is. The fact still remains though. E-mail is broken and is less reliable than an envelop dropped in a letterbox.

I think there is not a spam filter but this is a rule to prevent people from setting up phishing scams or other deceptive emails and they are using an appengine instance to host their scam.

Oh, is that all? Someone should let Google know.

Running a mail server is simple, even for programmers.

Running a mail server with reliable outbound delivery, and inbound spam filtering, is anything but simple.

Except as soon as you tell the spammer what is blocked and what isn't, they just add that to the playbook.

I agree with you that some measure of notification that there's spam filtering would be good, but it's Google we're talking about. There's spam filtering.

Also: Mandrill.

No, gmail constantly drops messages without putting them in spam. That's actually a huge problem. Microsoft does the same. That's for e-mails that don't even contain any links!

Interesting. I have been somewhat weary of the "machine learning for everything!" approach that google has taken recently -- i just don't think it's there yet. I wonder if a learning algorithm on the google side just started deciding outgoing spam and not giving correct feedback. It is very curious that no url works and with url doesn't. That should give straightforward repeatability (if they can simulate as your account). Have you tried sending to different target e-mail addresses during the test? I saw once it just worked and a bunch of stuff flew through. Also saw there was some question as to quotas. Did you get it clarified? Are you maybe above usage quotas with the regular e-mails so test e-mails are being stopped/delayed? Are test and production separate? It seems like if it were a google side issue there would be an uproar from the users.

When will App Engine finally upgrade Go to 1.6? It's currently stuck at 1.4.

Also, why the long delay? Is it because of the switch from C to Go in 1.5?

Appengine's whack pricing says: we'd like to kick you off appengine onto GCE but that would scare away all the enterprise users from GCE. If we announce regular Moore's Law following price cuts for GCE and nothing for appengine, well hey, we can't stop you leaving. We're totally not twisting you're arm! (aside: we are)

"A commitment to Moore's Law: https://cloud.google.com/pricing/philosophy/ "

Jeez, if the Snapchat backend looks anything like the ux works this isn't much of an endorsement. Just because some company moves a lot of bits doesn't mean they know what they are doing. The easiest way to move a lot of bits is to be bad at infrastructure and have investors with deep pockets. Fuck just run a rack of open relays and bad NTP/DNS servers.