Are we really going to run this story every month?
"I posted my private AWS keys by mistake on github, a spider got them, I got a huge bill"
Yeah, you don't say
"I posted my private AWS keys by mistake on github, a spider got them, I got a huge bill"
Yeah, you don't say
Is it, though? Committing them to the same repository as one's code lives in, sure, but committing them to a separate production-deploy repo seems okay to me (although I'd much prefer that private repo never to hit a centralised service like GitHub).
Yes, it is. Period. Full Stop. Don't ever check access keys into any repo, public, private or even self-hosted. AWS needs to do a better job of making you realize that access keys are like 100-year-old sticks of dynamite and should be handled with an equal amount of care considering they can cause a similar amount of damage.
To their credit, they basically noted this when they changed the way access keys are handled in the command line tool[1]. Quoting:
An important point is that the default location for the credentials file is a user
directory. It's no longer part of a project file structure, such as an app.config file
(.NET) or .properties file (Java). This can enhance security by allowing you to keep the
credentials in a location that's accessible only to you, and it makes it less likely that
you'll inadvertently upload credentials if you upload a project to a developer sharing
site like GitHub.
So, how do you propose to handle version control of your exact production configuration? Do you think that recording one's production configuration is a valuable (or even useful) technique? Do you propose to have a single file with production configuration relying only on point-in-time backups for history? That gets us back to the bad old days…
A local or self-hosted repo is 'a local location that's accessible only to youation that's accessible only to you.'
That looks pretty cool, at least at first. I'll have to take a deeper dive into it.
I like that it looks to be all shell scripts. Wish it were sh rather than bash, but c'est la vie.
Version control is perfectly acceptable for that, but the key point is that access keys aren’t part of your production configuration. I personally don’t allow my team to have long-lived AWS access keys that have access to production. Production is only changed via an assumed role requiring MFA or an IAM instance role.
If this kind of setup is too complex to create on your own, solutions like Vault (https://vaultproject.io) can make it much easier to get right.
Why not? If your production configuration is stored as static-file in version control, then either those credentials have to live in that version control, or they have to live outside it, as a separately-backed up file of configuration variables, which are then fed to a static-file-generating tool, which means a) that one has to select a templating system and maintain those separate templates and b) that there's another step involved in developing the prototypes for those templates. This isn't a bad course of action, of course, but it does have its costs (just like storing production configuration—including credentials—in version control has its costs as well as its benefits).
> I personally don’t allow my team to have long-lived AWS access keys that have access to production.
Of course! Developers shouldn't have access to production systems, period. But operators can, and must, and I've no problem with them having access to their own version-controlled repository of information that no other team can access.
> Production is only changed via an assumed role requiring MFA or an IAM instance role.
Of course! I'm certainly not arguing for personal credentials to be stored in an operations repo.
Generalizing here (and for the record I agree with your point), is that to a developer, source code is as valuable as anything else, and if a private repo is secure enough for source code, it should be secure enough for access credentials.
I speculate that this is a by product to moving towards a developer-centric engineering culture.
> I am certainly not innocent here and some mistakes were made on my part. [...] To this end, having encrypted access keys or excluding configuration settings from GitHub would have prevented the AWS charges - and this is certainly the approach I would take from now on.
Also if you use a proper IAM profile locked down to specific resources then you wouldn't expose your entire account. The author stated he didn't use EC2 so why wasn't the key/secret pair an IAM account with a policy set for minimal access?
What I'd worry more about is that the VS bug exposed private source code, data and proprietary intellectual property.
We use github and I worry every day someone will public fork one of our repos by accident. That would be a grave fuck up but it's waiting to happen. We should have stuck with centrally controlled active-directory integrated SVN from a security perspective (even if it is a pain in the ass).