Mailgun Security Incident and Important Customer Information
blog.mailgun.com
blog.mailgun.com
Months ago I received spam from a Mailgun server and tried to use their web form[1] to report it, but it was broken. I reported both that bug and the spam email to their support, which acknowledged it. Weeks later I got another spam email from that same domain, popped open that report form and it was still broken (FWIW as of today it seems to be working again). So I followed up on my initial support request with that info but got no response. Just a few days ago I received another spam message from that domain.
I personally consider all that a very bad sign in an email service provider and wouldn't use Mailgun myself. In contrast, I've been very happy with Postmark.
> We offer a Free Trial plan for testing purposes only. The Trial is limited to 100 emails a month with no overages allowed.
https://postmarkapp.com/support/article/1107-how-does-monthl...
Still, I can recommend their free tool to monitor DMARC: https://dmarc.postmarkapp.com/
Postmark's service is great but their new min $10/month pricing scheme is a retrograde step and penalises small companies sending less than 1000 emails a month.
Deeply unhappy with the change, and wish more companies would follow the Amazon AWS pricing model.
I find it amusing they still have a "trusted by Reddit" blurb on their homepage after this!
We have been working to investigate the issue and coordinating with Mailgun, a third-party vendor we’ve been using to send some of our account emails including password reset emails. A malicious actor targeted Mailgun and gained access to Reddit’s password reset emails. The nature of the exploit meant that an unauthorized person was able to access the contents of the reset email. This individual did not have access to either Reddit’s systems or to a redditor’s email account.
As an immediate precautionary measure, we moved reset emails to an in-house mail server soon after we determined reset links were indeed being clicked without access to the user's email, and before Mailgun had confirmed to us that they were vulnerable. We know this is frustrating as a user, and we have put additional controls in place to help make sure it doesn’t happen again.
We are continuing to work with Mailgun to make sure we have identified all impacted accounts. At this time, the overall number of confirmed impacted users is less than twenty. For those affected, we have resolved the issue and assisted in account recovery.
Additional information about Mailgun’s security incident can be found on its blog here. We’re committed to keeping your Reddit account safe and will continue to monitor this situation carefully. u/sodypop, u/KeyserSosa, and I will be sitting around in the comments for any general questions.
Many services state in the password reset emails that "if this was not initiated by you, ignore it", but it really should be the exact opposite - click the link below to report it!
It looks like the target was "bitcoin-ish tipped into /u/someredditor" and the hack/vuln was "intercept mail password resets in order to auth account in order to snatch crypto-currency"
ie: most people's reddit accounts (IMHO) are on the "not that important" on the scale of password protection. (Personal Email/Financial => Work => Medium Security [facebook, amazon, etc] => Low Security [discussion forums])
It's another way of saying that I would expect phpBB or reddit or pinterest to have lower password/server security than my gmail or bank websites.
However, because reddit is relatively high profile, and there was mixing of "cash and reddit", all of a sudden not just reddit was target of a hacking attack, but also reddit's 3rd party service providers.
I can choose to use reddit or not, but I can't choose that reddit uses or doesn't use some other random service provider that may or may not be vulnerable.
We ended up going with Mandrill which does offer the option to not log sensitive data ^1. Whether they log it somewhere else for the compliance reasons that Mailgun mentioned isn't mentioned anywhere in their docs or privacy policy, but doesn't seem to be accessible from everything I could find. You should never log or allow others to log password reset urls or other sensitive details.
1: See documentation here: https://mandrillapp.com/api/docs/messages.JSON.html#method-s... and search view_content_link
I just thought the attitude/assertion was in discord with my own experience/understanding.
That includes resets, username reminders, signin notifications, etc.
Also secure access to your transactional mailer account with 2FA and restrict access to those who need to be there (i.e. not your entire support team).
edit: worth noting that there are obviously other ways a hacked Mandrill/Mailchimp account could be abused. This just shuts down one of the major abuses you could perform.
Those are two entirely separate companies (unlike Mandrill and Mailchimp which is the same company.)
Furthermore, this seems to indicate that the API keys are not hashed. I would expect some bits of the API key to work as an identifier and the rest of the bits treated as secret material (properly hashed).
As a Mailgun customer, this is concerning..
1. How was the employee's account accessed? No 2FA?
2. Do employees ordinarily have access to customer secrets (e.g. API keys) or was there some further exploit?
3. The advice in OP for affected customers is to roll keys and SMTP logins. Couldn't/shouldn't you do that for them? Surely security should trump up-time/deliverability?
In an ideal world, every customer would have a good setup where they can rotate third-party supplier API keys painlessly and have plenty of bandwidth to handle security emergencies. Alas, there's a lot of bad setups out there, and some of them are critical to their customers' operations.
Nothing I've personally worked with had a setup bad enough to make that painful, but I'd be very worried about how reckless a service is to rotate API keys that aren't being actively exploited to do something dangerous without getting a positive confirmation from the customer.
I seriously suspect this was the job of an insider, not a compromised employee laptop.
For example, I'm confused by this kind of statement:
> Mailgun has now completed its diagnostic of accounts that were affected and has notified each of the affected users. At this time, we believe less than 1% of our customer base was potentially affected. If you were not directly notified by Mailgun regarding this incident, then your account was not affected.
If you believe that less than 1% of users were affected, it means you don't know for sure how many accounts were affected.
From there, how can you state that "If you were not directly notified by Mailgun regarding this incident, then your account was not affected"?
Doesn't this last statement mean you know for sure my account was not affected? Isn't it in direct contradiction with the previous statement?
Yes, definitely true. Although some contexts, like a security disclosure, might warrant a very carefully non-contradictory worded statement that leaves no doubts of interpretation.
> the language can also indicate potential false positives, again because of the nuances of language.
Yes, but in this context, false-positive aren't important to the audience of the disclosure. Nobody really cares if their account was "identified as affected, but in the end wasn't".
If you announce that 1% of your user base was affected, and it turns out that 50% of this 1% were false-positive, great! You were still right in announcing that 1% of your user base was affected. You can always correct this later and announce that things panned out better and only 0.5% of your users were impacted.
So very seriously that they don't even use https for their blog...
This is Chris from the Mailgun team. I'm sorry that this happened, this shouldn't have been the case. I'd be happy to help rectify this issue, would you be able to send an email to help@mailgun.com with details so I can review?
Let's Encrypt is free and takes less than 5 minutes to set up (using certbot).
Automating that crap in ansible is almost too easy.