Security Hole in Sendgrid | Hacker News Reader

Security Hole in Sendgrid

chunkhost.com

181 pointsndaiger92 comments

Social engineering will almost always work. I don't really fault Sendgrid for this (though I could see this not working as well if you were using Amazon SES...no support to even talk to!). It sucks that they got caught with their pants down but I bet a good social engineering attempt on ChunkHost might have yielded similar results.

The lesson here is to have multiple defenses. 2 factor auth is a great start and it worked in this case.

In a nutshell, social engineering can be countered by proper software & processes engineering.

So if social engineering is possible, blame the software architects.

Maybe in extreme cases changing the email of an account might be needed, but there's no excuse a first level rep was able to do it. Least thing, he/she should've been forced by the system to escalate to someone above her, who has a much lesser chance to screw up.

They didn't say it was first-level rep. Maybe the rep passed it up the chain.

This was a pretty weak request, though. This one should have been pretty easy to at least make some attempt to verify.

But as a human being I can verify how hard it is to not help the person crying on the other end of the phone because they need help RIGHT NOW. This didn't get to that level, but if you are designing a recovery process, you really need to think about how you handle that situation, and make sure the people making the call have the guts to say "I'm sorry but we need to do this one by the book."

You should fault Sendgrid as they specifically have a policy NOT to perform this change of email request (from the article).

Sendgrid can also change their systems so that phone support personnel can NOT perform this change or perform this change with approval from a supervisor.

Sendgrid being in the business they are in should also know that they are susceptible to these types of attacks and what they can lead to (many, many systems which can have password requests sent to email addresses).

I don't know... Mistakes happen. There seems to be little to gain from faulting SendGrid, but faulting SendGrid would force them to take some kind of action such as terminating the rep. I think I'd prefer the rep remain employed, because I trust they'd never make this mistake again. Also, now all the other reps know to avoid it.

EDIT: May I ask what can be gained from faulting SendGrid in this case?

I agree with you that terminating the rep is not interesting, and I think you're mistaken if you feel like that's what anyone thinks will solve this problem.

Actions SendGrid could take:

* Make it impossible for their front-line support staff to change the email address on file. If you want that -- which should be extremely rare! -- you talk to a high-level manager who is competent at authenticating you.

* Send the email that says "hey, we're going to change your email address now" with a lead time to allow for the possibility that, even after your authentication, you've been conned.

* Make a phone call to the phone number on record, too.

You ask what's gained by faulting SendGrid, because you take it as a given that they will make these changes. But that's not how blame works. The blame serves a function of ensuring those changes by holding them accountable for their current problems.

Given the contact email address is so important, perhaps SendGrid should have a way of confirming it before they allow it to be changed?

Or perhaps they should allow customers to require that they contact a secondary authority to confirm the change should be made?

>I don't really fault Sendgrid for this

They literally handed over a customer's credentials over the phone. What's more, there didn't appear to be any ID verification.

I don't see how that could be anything but a huge, glaring, faultable security hole. I'm a Sendgrid user, and this is pretty scary.

Another title for this submission could have been:

"Massive Security Hole in ChunkHost. Non-2FA accounts can be owned."

Because it turns out anyone with a Sendgrid Support account also effectively had potential access to any account at ChunkHost not using two-factor authentication. Which is also true of thousands of other companies that are relaying their password reset emails through third party SMTP services.

SendGrid seems lame, for allowing this and for their response promising to yell more loudly at their support people, but they're an SMTP relay service not an authentication service.

Are there any web hosting companies that don't rely on the "send a reset link to your email address on file" model of password resets?

You're right, that model is deeply broken if anyone can intercept those emails (as happened in this case), but it seems unfair to single out ChunkHost for criticism.

With that same logic in mind - we could say that anyone working at amazon AWS or Rackspace (or any other hosting companies) could gain access to your application.

The thing is we trust these companies to have processes in place so that their representatives won't have the ability to potentially do something destructive and if they can because they are the highest ranked rep and they need that kind of access - then at least auditing and training should be in place to avoid that kind of behavior.

SoftLayer still asks for admin/root passwords to your boxes in pretty much any support scenario. Their ticketing system actually has a field for it in the submission form.

If you omit it, the assigned tech will frequently ask for it. Always sends a little shiver down my spine.

I don't see how this was specific to ChunkHost, or how anybody else using SendGrid and supporting password resets through email wouldn't be vulnerable to the same attack... Since presumably password reset emails aren't outside SendGrid's target market, it seems reasonable to suggest that this is a problem SendGrid should try to fix.

Generally, an outsider doesn't care that you got owned because someone you relied upon got owned. They consider it part of your job that you make sure all the people you rely upon are reliable.

I'm not saying this is fair.

Depends on what outsider. Sure, for your customers, everything is your fault. But as an outsider that is also building services with email components, I do care exactly how they got owned so that I can think about which part of their systems were actually problematic.

That would be because sendgrid is lame.

I've recently tried to use their service to send emails, club member newsletters. I've never thought much about bulk e-mails before and I thought it was a "solved problem" by now. Sendgrid are well known so they were my first choice.

During initial testing I found both bugs in the API and missing functionality. I've worked over 20 years with IT and yet sendgrid support is easily on the top-3 list of my worst support experiences, a total waste of time.

We ended up using mailgun instead and so far it looks much better.

We looked into sendgrid and also went with mailgun. Been with them for over a year now. Originally started with their mailing list API but had enough service delay problems that we were considering switching entirely. Don't remember exactly but may have had some outages as well. Their support suggested we change to using their batch sending API instead and I'm glad we stuck with them because it's been solid for over 5 months now. (Well they did just have two issues recently, ssl cert and duplicated email sending but it didn't affect us too much.)

Overall they've been a good service provider and their tech support has always been helpful and pleasant to deal with. (I know! Crazy, right?).

I work with SendGrid frequently because of my day job, and they have hands down the worst software and design of any company in the industry.

I use Mandrill for my private projects, and it is so much better it's shocking SendGrid has a single customer.

Would you apply the same logic to your DNS provider?

The problem is that it was technically possible, for a representative, to make this change without the proper verification. You just can't rely on humans for that.

> "... confirms your suspicion that these people convinced one of our representatives to change the email address on file."

This is the part that scares me. Do they not have auditing in the system where the representatives are able to change the email address on file?

SendGrid once went over my head and changed my account settings on behalf of a customer of mine, without even consulting me beforehand. I had a customer with a history of reporting mails as spam (activity reports he requested in a webapp). When you report a mail as spam, the address goes into a blacklist to avoid causing sender reputation issues. After the third or so time of having him ask to get the mails again, then mark one of them as spam, I told him I wouldn't be offering that feature to him any longer. He contacted SendGrid about it directly, and the SendGrid rep actually went into my account and added his address to a whitelist to bypass the blacklist, where I intended it to remain.

I thought that was just the strangest thing, and it didn't sit well with me. Accessing a customer's account when they request service is one thing, but making changes on behalf of a stranger you know has no authority over that account? That was when I moved the last of my apps over to Mandrill.

And people complain that Google has bad support! At least you can be certain that there's no way an attacker can get the Google support rep on the phone. There aren't any.

Looks like a deeply unsatisfactory response from SendGrid. They don't even know for sure ("it appears .. pretty much confirms") that their own support staff changed the email address?

This shows why so-called PR-speak is necessary. Email like this should not be in a conversational style because it can easily be misinterpreted.

are you serious? How was it misinterpreted? It point blank says that that's what happened? Please illustrate how exactly it was manipulated to say something else?

So what's the answer? Here's two very legitimate scenarios:

1) You sign up, enable two-factor auth, then lock yourself out (lost password and your second-factor). How do you prove to the service provider that you are you?

2) You sign up, enable two-factor auth, then Mallory claims that they locked themselves out. How does the service provider prove that Mallory is not you?

In this case, looping in the original email address on the SendGrid account before changing to a new one would have kept this from happening. SendGrid's support personnel should almost certainly not be able to change an email address without the change being signed off on through the old address first.

But what happens in those rare cases where that first account gets lost / locked permanently?

Yes, this is part of what I'm trying to get answers to.

Do you tell the user on signup to print an in-case-of-emergency-break-glass password which is only ever to be used to get into a locked account and other special circumstances?

It may seem over the top but seeing as it's unique across service providers, I think it's a hell of a lot better than the overly abused "what is your mother's maiden name" type questions. I consider these questions to be in the same boat as sharing passwords between websites (since they are)!

The solution is to do what everyone who actually needs authentication from a company does; require a posted signed letter from a director, possibly along with an outbound (from SendGrid to the director) phone call to confirm. There's plenty of low-tech ways to confirm that a company really wants to do something.

Please, no.

Consider a determined attacker. A posted signed letter has zero cost and is easily forged and a phone call is free via Skype. There's plenty of low-tech ways to circumvent security.

How exactly does Skype let me take over a business's phone number? I am saying that SendGrid should call the company to verify, not the other way round.

Ahh sorry, my mistake. I missed the word "outbound".

Presuming you're paying for this service (and thus have a credit card registered to it), how about the "we've made two $0.00 - $0.99 charges on your card; tell us what the cents digits are and we'll refund them and give you a reset link" model? I've only ever seen it used to initially verify a card--but, provided a card has been verified, continued access to it can be used to re-verify a compromised account.

(And if someone has managed to break into both your personal email account and your business's online-banking account, getting your web-host to recognize you will be the least of your problems.)

Require that the company submit a legally binding/notarized document before changing the e-mail address.

Lol. So what you're saying is all I need is photoshop to get the keys to the kingdom?

Electronic notarization uses digital signatures, and SendGrid could just require them. Good luck breaking those with Photoshop.

You need some kind of "okay, you can get your password back, but it's going to take some time. You cannot get back up instantly."

Maybe they FedEx the password to your physical address on file. Maybe they contact all phone numbers and emails they have for you and say "someone has requested an emergency override, if you object call us back in the next 4 hours." Maybe they do a Skype session and compare your photo to the one they have on file.

All this costs money, of course. That's the price of doing business.

I think these services could use timed release. If you have locked yourself out, the timer starts running. If after 30 days no one has denied the request, access is granted. I'm pretty sure this approach would foil most if not all social engineering.

Text message or phone verification of details only you should know about the account history, payment methods, etc. As others mentioned, a waiting period during which they try to contact using any means s previously authorised for a response. Compare IP addresses and deny logins from strange countries or origins without further verification, etc. Of course, every measure and countermeasure needs to be justified, since there's an implementation and upkeep cost, but... It's possible to be "more certain," that something is legit or not.

Yep, so text messages and phone verification could really be considered a "third factor". I guess anything information you have already provided to your service provider is considered an X-factor.

You should review the work NearlyFreeSpeech.NET recently did on customizable account recovery options. It's easy the best I've every seen. It works like this:

1) You decide how valuable the account is, the probability that you will lose access to the account, and the probability that the account will be attacked. 2) You selected the required number of recovery actions, from one recovery action to completely unrecoverable. Possible recovery actions include (copied from NFSN):

* You provide a scanned copy of a government-issued photo ID. * You provide a scanned copy of a statement showing both the most recent deposit and a name and address matching one of your accounts. * You complete SMS verification. (SMS must be previously configured.) * You complete 2-factor verification. (2-factor auth must be previously configured.) * You correctly answer your security question. (Security question and answer must be previously configured, below.) * You use an ssh key to create a file with a specific name on one of your sites hosted here. (Must be previously configured, won’t work if account is empty.) * We try and fail to contact you via your currently configured email address. (This one may take a long time.)

As far as I'm concerned, this is the way it should be done. The public details are on their blog: https://blog.nearlyfreespeech.net/2014/02/28/price-cuts-more...

Use Amazon SES to generate your outbound emails; ensure proper IAM policies, and that you're using 2 factor auth to login to your AWS account.

Is AWS any less susceptible to Social Engineering attack of this type?

Specifically--can AWS support staff grant access to AWS accounts, and if so:

what are their criteria for doing so, and what are the policies in place to ensure those criteria are met, and how are those policies audited?

As a TechStars alum, my company was granted $50k in AWS credits, which were tied to my AWS account[1]. When I left the Company, the CEO was able to get the credits moved to a different AWS account that was company owned, without my intervention at all, even though I was the only account owner.

The fact that he could have credits moved out of the account without any kind of verification from me[2], should be cause for concern.

[1] I should have created a new Amazon account for a group email [2] Obviously the credits belong to the company; they weren't mine to use, so I would have authorized the migration.

It doesn't directly answer your question, but I don't think AWS has support staff -- just engineers that answer issue tickets.

Amazon (non-AWS) is infamous for being vulnerable to social engineering attacks. I don't know if they've changed their polices more recently but they are (or were) often the first attack vector for social engineering. If you can get access to an Amazon account you can get the last 4 digits of the user's credit card number(s). You can then use that info to reset accounts over the phone with other companies.

What I found most interesting was they were targeted due to Bitcoin. Today most services would never store credit cards themselves but rather have them stored at a secure payment gateway. So they're unlikely to be attacked to get to those cards. T

Trust me I know that CC's are getting sniffed in transit too often so I'm not saying they're 'safe'. I'm just wondering if there is something unique to Bitcoin that suddenly makes you a target as though you were known to be storing CC's data at rest onsite.

Bitcoin is like cash. It's far more valuable than credit card data.

Email accounts are the weak link for many things... DNS registrations, web hosting, etc. Get the email account and you have it all.

I'm coming around to thinking you should get a secret email address for your sensitive accounts. Treat the string of that email address as almost as important as the string of your password. The problem is that you do have to tell it to people who probably don't share that philosophy to the same extreme.

We use Sendgrid and have hundreds of thousands of customers that might be phished by this social engineering trick. It's absolutely unacceptable that such a crucial piece of infrastructure is vulnerable to such a simple trick.

I'm going to bring this up with our team and see if there's another vendor that can more reliably protect our customers.

Your logic is a bit weird.

Sendgrid just experienced this major embarrassment and are currently re-training their staff to avoid it again at all costs.

And you're going to move away from them now?

I'll talk it over with my group. Retraining doesn't guarantee anything. It could be that this problem is endemic to the company itself.

For example, why should 1st level support have the ability to make major changes like this? It sounds like only 2nd level support, a smaller group of more highly trained support staff, should have the ability to do this. Does SendGrid have enough money/resources to split their team into 1st and 2nd level support? Would a larger company have those type of resources that would better protect my customers?

These are questions I will talk over with my team.

The fact the Sendgrid even has to do re-training in the first place is the problem. They only need to do and handful of things well and keeping their users accounts secure is arguably the most important. If they are having issues like this this far along in their lifespan, it's a sign of more systemic issues in their company and does not instil much confidence to potential/current customers. Wanting to switch vendors doesn't seem weird at all to me.

I think you're being lenient with SendGrid because it was a close call and not a full-blown catastrophe. First of all, SendGrid lied about its support staff's permissions. After the incident, SendGrid then sends an "oops! our bad!" e-mail where the employee in question will apparently be gently tapped on the wrist and maybe send a passive-aggressive reprimand.

I mean, keep in mind that this is the same company that publicly crucified a female employee in order to stop a DDoS attack. Clearly there are some priorities out of whack there, and given the insecure nature of e-mail in the first place, I would never want to deal with a company that is so clearly unprofessional.

It's an important lesson for all of us. I've seen a lot of privacy ignorance when it comes to support (e.g. folks handing over sensitive data after an anonymous request on Olark). We should all go an extra mile and verify the identity of the requestor.

1) If your support chat doesn't enforce authorization, always ask the requestor to send you an email. 2) Make sure the domain is correct (that's where Sendgrid screwed up). 3) Never agree on replying to a different email address than the one of the sender.

BCC every message is evil, as it can be misused as in this case. SendGrid should never allow that, or at least should flag such behavior. At the minimum, they should notified account owners of this change.

The attacker got SG to change the email on file, so the notification would just be sent to him.

It's not a security hole, misleading title.

Should read "social engineering resulted in security breach" and guess what, this happens all the time whether sendGrid or not.

My first thought was to whois chunkhost.info, which returns a clear name, email address and phone number.

Or is it that easy to register a domain with a fictitious persona?

It's that easy. You can use anything you want.

> Or is it that easy to register a domain with a fictitious persona?

Virtually every registrar allows instant update of whois records via a web interface. I can have Bill Gates be the technical contact on my personal domain in about a minute.

So what they are saying is that SendGrid should have had two-factor auth and this would have never happened.

SendGrid's policy (as stated in their first email) is that the support people shouldn't be changing account emails in this fashion. Even if SendGrid had 2 factor auth, who's to say the support guy wouldn't have just disabled that?

If their policy is that support staff should never be able to change an accounts email address... why does the system let them do it?

At some point someone has direct DB access and can do this. Sure, normal support people don't but this just makes the social engineering aspect a bit more complex, not impossible.

With TFA if the email got changed it would not make a difference. The attacker would need the second factor to rest the password and to log in. So the worst they could do is to lock out the account owner. The support staff being socially engineered is a different story, but yes this is a security hole in SendGrid's system and an easily patched one at that.

http://twofactorauth.org/

SMS verifications should be sent out to allow over the phone changes.

Alas, the weakest part of the system is...

Sendgrid seems to be its own worst enemy (Adria Richards debacle, now this).

I suggest we dub the act or state of being one's own worst enemy, being "sendgriddled."

Send your emails yourself.

It's not like it is hard. At least not harder than integrating to a third party email sender.

If you use EC2 or the like your IP address or the entire address block could have easily ended up on a spam list so your email will be blocked. I wish SendGrid was only necessary for people who sends lots of mail but the reality is that no cloy provider can guarantee that email from their IP addresses will be delivered.

If you invoke sending email through Amazon SES, it's free for the first 2K per day, due EC2 blocks being on spam lists.

http://aws.amazon.com/ses/pricing/

"You can send 2,000 messages for free each day when you call Amazon SES from an Amazon EC2 instance directly or through AWS Elastic Beanstalk."

We just switched off of SES because of the lack of bounce/rejection diagnostics, and their internal blacklisting policies are really aggressive. If someone's email server goes down for a few hours, they're blacklisted for quite some time, even after it comes back up.

After using SES for close to two years, I'd suggest looking elsewhere if you really care about deliverability or stats/metrics. We switched to Mandrill a few months ago and have been very impressed. It's still a tiny, microscopic percentage of our budget, but we get so much more (open/click reporting, sub-accounts, rendered email body history, much better blacklist/whitelist management).

Thanks for the recommendation! Will have to check them out.

Does open/click reporting still work for GMail with the changes to the way they load images?

Check out the bottom of this post for where they stand on that: http://blog.mandrill.com/rejection-imports-subaccount-enhanc...

They get accurate data for clicks, but geolocation/browser stats for opens aren't 100%. They still track the rates, but it looks like they're coming from one of Google's proxies, so this is the best they can do for now.

Another +1 for Mandrill. We've been quite happy with them as well.

> [...] the lack of bounce/rejection diagnostics [...]

My employer has been making more and more use of SES, and we've just started looking at automated processing of feedback notifications [1]. Did you find them lacking?

[1] http://docs.aws.amazon.com/ses/latest/DeveloperGuide/notific...

Is sending emails yourself as easy as changing two lines of code in your app? That's what it took to integrate Mandrill for me. If you think that's as hard as sending email yourself, you are wrong.

Yeah, it's not hard: until you want to reach mailboxes at some obscure providers like Yahoo or Outlook.

Deliverability plays an immense part why people rely upon an ESP.

Yeah, I have to imagine anyone who claims sending email yourself isn't hard probably hasn't sent enough email to know how hard it can be. There's a new problem with every order-of-magnitude increase in volume.

Yes, SendGrid sends emails, but there is value add on top of that. (e.g. negotiating with hotmail to ensure your emails get through, webhooks, APIs, statistics, templates, etc.).

Yes, sending an email is easy enough. It's all the other stuff we have services like this for.

Sending email is hard and for most people not a thing that provide them with a proper ROI, because among other things...

- You need to make sure that your email servers IPs are not on black lists.

- That you use DKIM properly

- your multipart mime encoding is correct

- bouncing e-mails are handled...

- take care of scaling & operating the servers

- and so on....

You can spent ( and waste ) a lot of time on this... especially if you are new to email sending..

> - You need to make sure that your email servers IPs are not on black lists.

This alone is a huge, huge chore. Especially if you run a hosted service that allows some user-specified content in outbound email bodies. It's a lot cheaper for us to pay Mandrill to handle all of that for us, provide us excellent metrics and diagnostics, and let us know if one of our users is sending junk mail before it gets out of hand.

Even if you do everything right you can still find yourself on a blacklist. Then you get to pound sand.

Blacklists suck. Not as much as spam sucks, though.

Frankly, with the amount of bullshit blacklists out there that real mail providers actually trust, I've come to detest blacklists more than spam.

Has anyone set up a blacklist full of random IPs as a joke to see how many people follow it?

Well, we have been put on blacklists that the operator wanted money to "expedite" removal from. I'm assuming that blacklist was pretty much random IPs... And it did impact deliverability.

Reply on news.ycombinator.com