Hacker News Reader

Comment by flyt | Hacker News Reader

Especially bad for people that were using MongoHQ to manage their DB backups to S3:

As a precaution, we took additional steps on behalf of our customers to invalidate the Amazon Web Services credentials we were storing for you (for the purposes of backups to S3). While this prevents the abuse of your AWS credentials by any malicious party, it may have resulted in additional unintended consequences for your AWS environment if you were utilizing the same AWS credentials for other purposes.

If you had S3 backups configured with MongoHQ, when possible, we have disabled the IAM access keys via AWS. In any case, please go to the AWS Management Console and regenerate any keys given to MongoHQ .

View on HN

I was one of those people bitten by this last night. My client called and told me he was getting access denied when trying to upload files through his CMS. After some digging I found the S3 key had been revoked. This was concerning, as I hadn't touched the CMS code I wrote in like 3 years and I've had issues deploying old stuff to Heroku in the past. I really wish MongoHQ had contacted me first about revoking the keys.

Best practice would be to create an IAM user for each purpose rather than sharing the same AWS key across all of your apps, for this exact reason

Yes, for this reason, and because having separate keys allows you set appropriate access controls limited to the function they are being used for.

At the time this project was put together, IAM didn't exist. But I agree that this would be the best approach going forward.

We're very sorry about this, we went into a little bit of a panic when we realized that we held IAM credentials that gave full access to peoples' EC2 accounts and did what we thought was best. In hindsight, we should have gotten ahold of Amazon immediately and let them manage that process.

So this is why our app broke yesterday. We've spent the last 24 hours wondering how our AWS key was removed. Thankfully I was able to learn about this via HN. Still waiting on a note from MongoHQ. And when we went to file an issue with AWS there was no Premium Support option.

I am commenting without knowing the specifics of your application, so apologies if it doesn't apply.

You should look into using separate AWS keys for your DB backups and whatever it is your app uses those credentials for. This not only prevents any future availability issues because of key revocation, it also allows you to set fine grained permissions on your access keys limited just to what they're being used for.

We're enacting these changes at the moment. Thanks!

You should have a support issue open in your AWS portal now, if you need any help getting new keys for other apps. If you can't find it hit us up at support@mongohq.com and we'll escalate.

Thanks. I appreciate being overly cautious when it comes to security rather than under cautious. And this has made us realize that we should have a unique profile for each service.

Our application also had problems due to this. Would have been nice to receive a phone call or email.

Also I believe you should alert us as soon as you find out how long your servers have been compromised for.

As a company we are now trying to figure out how long our access tokens have been out in the wild.

You are 100% right on both points. We'll be updating our security page as we get more details, I expect we'll have some rough timeline information tomorrow.

While annoying for users who we're using the credentials they provided for other things (which they really shouldn't have been doing anyway, and hopefully MongoHQ makes it clear that they shouldn't) this really seems like about the only thing they could safely do under the circumstances.

Reply on news.ycombinator.com