Hacker News Reader

Comment by impostervt | Hacker News Reader

About three weeks ago I started working on a new open source project meant to interact with AWS. Coding fast and dumb, I cut and pasted my personal AWS credentials into my source code, committed it, and pushed it to Github.

The next day I got an email from Amazon, alerting me to the problem. Apparently, they scrape github looking for just that kind of stupidity. I instantly deleted the project, but it was too late.

Amazon ended up waving the nearly $3k in EC2 charges I incurred, thankfully. I'm now a zealous advocate for making sure a person never even HAS AWS credentials. Instead, make a new user without a password for each use case, and manually select the privileges that account has.

If you have a password to AWS, you shouldn't have credentials.

View on HN

I posted to the AWS forum and accidentally copy/pasted my secret key. Within 24 hours $11k of charges. I called them, they wiped them. It's amazing how quickly people find and use these things.

What kills me is there is no easy way to stop all the instances for an account, in a region. It took me hours to kill all the instances. They had maxed out the number of instances in every single region. Very, very annoying.

You can automate the stopping via the CLI interface. While I don't think there is a single command to stop all instances, you should be able to whip up a script to get all id's then call the stop-instances command on each of them.

Any idea what they were doing with them? Mining bitcoins? Hosting CnC for a botnet?

It's always Bitcoin mining. The attacker spins up a bunch of GPU EC2 instances and mines as long as they can. I don't think the profit ends up being very large (those GPU instances put out a puny amount of hashing power compared to modern mining ASICs)...

It's not that bad. The smart attackers don't do Bitcoin mining, they do altcoin mining, and they pick altcoins that don't have ASICs out. Then they are competing with other GPUs.

I'm not sure what the profit is, but I'd guess it's between 30% and 70% of the bill incurred.

ROI for BTC mining on standard AWS pricing is about -90%. So attackers get 10% free BTC on spend.

So, out of $11k, they make $1k. That's the kind of money they need to make between once and three times a month minimum depending on where they live. Doesn't Amazon notice patterns in terms of source, scripts which are uploaded and scaling profile: Who uses 2 medium instances for a year then spins off 2,000 in 20 minutes?

Certainly there's a pattern there, but it's not THAT far away from people who intermittantly scale stuff for short bursts of huge processing. False positives in those cases for people that really intended to spend $10-50k in a short time might mean a HUGE loss of revenue and/or incurred customer support and service costs.

You must lose a fair amount of water when you're greeted with a surprise bill of $11k.

Yeah, it's a rare moment in life. Especially fun since it happened on a Saturday morning (the notification, that is).

When people find the accounts, what do they use them for? Mining bitcoin?

> Amazon ended up waving the nearly $3k in EC2 charges I incurred, thankfully

For what it's worth, the word is "waived". :-)

I'm glad Amazon dealt with you well!

Github provides a public near-realtime stream of events, I'm sure blackhats are constantly monitoring it looking for private keys all the time. Even if you undo a commit within seconds, it may be too late.

https://api.github.com/events

If Amazon (and my coworker in a 3 person dev shop) can automate the scanning of common API key patterns as a pre-commit filter, I wonder why Github itself doesn't flag for this sort of thing before publishing.

What should they do? Reject a "git push"?

That's fairly reasonable. At least they can prompt you "Did you really mean to do this?" -- they have enough infrastructure to update your project page with a "Make pull request" button after you push, they can repurpose that for a "Approve next push without asking questions."

Alternatively, if they supported custom pre-receive hooks via e.g. webhooks, someone could publish a script or stand up a web server to check for such things.

Supporting git hooks on github would be really great. Always wanted to have them for rejecting debug statements, or as a rudimentary poor-man's-ci / linter

IIRC, a regex that matches a generic AWS access key will also match a git hash.

Not sure what I said that warrants multiple downvotes. Git commits are identified by SHA1 hashes, so they would be caught by the same regex that would catch AWS keys.

According to the AWS security blog, this is the regex you should use for secret keys: `(?<![A-Za-z0-9/+=])[A-Za-z0-9/+=]{40}(?![A-Za-z0-9/+=])`. To match git hashes, you can use this one: `[0-9a-f]{40}`.

See the likeness?

I did not know that existed and that is pretty awesome!

The real mistake here is using your account owner API key in your code, when you shouldn't use that key for anything ... In fact you should delete the account owner access key!!! Otherwise you may have used the key for a Power User or Administrator level IAM user in your code instead of a custom user with least privileges.

Always use least privileged IAM keys in code if you must, preferably use EC2 roles instead of access keys, and never give your IAM users permission to run instances ... That way if the key is leaked then evil hacker can't do much with it.

I agree. I haven't used AWS for a while but IIRC for a small side project I was looking into with a couple of friends I set up a low privilege IAM access for them but I logged directly into the VPS using a password if I needed to update something.

> I cut and pasted my personal AWS credentials into my source code, committed it, and pushed it to Github.

This paradigm is very puzzling to me. Why do people feel the need to publish every small project to the public? Is it because github is so easy to create a new repo? Why don't more people use private repos like self hosted or bitbucket?

The whole reason why I spawned my own source code hosting service is so that I can work on projects in private without worrying about random people looking at what I'm working on (some of my C++ projects would give even Stroustrup a heart attack...). Note - this isn't saying bitbucket or gitlab or any of the others aren't good (oh I have my own opinions and comments about them...) - I've become rather bitter/paranoid/resentment of offers of free hosting.

> Why do people feel the need to publish every small project to the public?

Github acts as a portfolio site for programmers.

The usual refrain when doing hiring is "check a candidate's Github." Thus every potential employee makes sure there is something other than cobwebs on their pages. This is especially explicit in startups around San Francisco. Hell I'm guilty of it as well.

Likewise, many language communities actively encourage library development. Ruby and Javascript are great examples of this.

Then there are entire tools built around git paradigms. Heroku's push to deploy a perfect example. Many services have easy application hooks into git actions. Github has many built in. Pushing code and then kicking off a build system with automated tests is worth every penny I spend. Other services have this as well. However, it's harder (read: more than 5 minutes work) to build these hooks on a server you are running yourself. Paying Github for the work is a no brainer.

I've used Bitbucket for private repos, but usually the user wants their code public. They want to show off. But if I wanted to keep some code private (like my latest app) I'm all for private hosting.

> Github acts as a portfolio site for programmers.

No [1] [2]. It is not your resume or your CV. You should be able to highlight projects or accomplishments on your CV - github gives you no control over the layout of your profile.

> The usual refrain when doing hiring is "check a candidate's Github."

(I'm assuming you mean something that they do rather than something they don't do)

I'm not saying you can't look at be like "oh those are some cool projects he is working on" - but actually using it to say "man this guy is a loser coder - we can't hire him!" I think you should just step outside for some fresh air and just relax and listen to the birds for awhile.

If you need reasons [3] why you shouldn't [4] - there are plenty [5].

> However, it's harder (read: more than 5 minutes work) to build these hooks on a server you are running yourself.

You should check out Jenkins. Within a couple of mouse clicks I can ask it to automatically build, run tests, archive the binaries, and send them somewhere. And even email me if it fails.

[1] https://blog.jcoglan.com/2013/11/15/why-github-is-not-your-c...

[2] https://tommcfarlin.com/github-is-not-your-cv/

[3] https://github.com/gelstudios/gitfiti

[4] https://github.com/will/githubprofilecheat

[5] http://mikeboers.com/blog/2014/10/26/the-evils-of-gamifying-...

Portfolio != CV

I guess it really depends on your definition.

For example Mahara claims to be an online portfolio creator - but you can also build your CV with it [1].

I just wish someone would come out and say "this is the format your resume/CV should be in" - not just for my own sanity but so that when applying for jobs. A lot of companies now they have their crappy resume reader that attempts to read your resume into normalized text boxes - and usually fail miserably forcing you to retype your resume.

[1] http://manual.mahara.org/en/1.8/content/resume.html

I'm surprised that of the quoted comment you focus on the "paradigm" of open sourcing code, and not the "paradigm" of mixing credentials with code.

Trying to protect against leaking creds by not open sourcing is a bit of misdirected effort.

I pointed this out because I've seen this over and over again with the single theme of "I was working on some silly project using AWS and I pushed my credentials to github".

I don't think I've ever seen a professional team push AWS credentials (or any other credentials for that matter) - and if they did it's very rare to the point where I don't remember.

As far as mixing credentials with code - that happens all the time. I'm not saying it's right but I have much bigger concerns - such as why people keep silly projects to github with their credentials. The first time I saw an article about it - I found out that there are bad people who are monitoring github for exactly that - and they will use it even before you realize it what you did. And not just AWS credentials mind you - MySQL, SSH, anything they can get their hands on...

I think my comments are getting lost in translation -

I'm not saying I'm against people pushing their silly projects. Push your 1 line GPL code projects all day long - I don't care. That's why github is there. What I find puzzling is not that people want to create new projects - but they feel like they must create a git repo for EVERYTHING they do and push it up to a public github like it's going to change the face of computer science as we know it or something. I've seen people keep documents and even bash profiles as public repos. I'm not saying it isn't a good idea to use git for those purposes - but I wouldn't want my bash profile common knowledge (especially if it contains commands or functions that I use at my day job).

One reason why I like to publish even frivolous things to Github is because it forces a bit of rigor upon whatever I'm publishing. I'm more likely to clean it up and keep it tidy, and I can use the same workflow for it that I use for everything else, like using Github Issues as a TODO list.

But I suppose that might be more of a backsplanation.

More likely is that I do it as a backup strategy, because I have limited private repos, and because I'd rather just keep everything on Github than use multiple services.

It turns out that unless you commit AWS creds, nobody cares about your repos.

> It turns out that unless you commit AWS creds, nobody cares about your repos.

Oh there are people who care - just not the people you want to care. Imagine a wolf circling it's prey.

Why does it matter? I think your mistake is assuming that people believe the "change the face of computer science" bit. And perhaps this person had a particular reason for putting this up (wanted to show someone), and doesn't actually feel the need to push every line of code they write. I feel like very few people are as obnoxious about it as you describe, but even then - again, so what?

You're more guarded about your code, that's perfectly fine, but not everyone feels that way. Code that's shared has the potential to benefit and inspire others (even stuff some might see as trivial or silly), code that's hidden doesn't. I see nothing wrong with erring on the side of openness (except with account credentials of course!).

You can use jgit with S3 for cheap and private git hosting.

Or BitBucket or GitLab.

Or AWS CodeCommit.

> Coding fast and dumb,

And that's why you should use Bibucket first and foremost. And then when you vet your stuff and you know it's clean, you can publish it on publicly Github.

If you rush pushing code on github public repos that's exactly what will happen. If you start with a private one then move to a public one you have more time to think things through.

Now one could imagine a third party service warning users of potential issues with files pushed,based on their name/extension/folder name. But privacy first , then open the code to the public.

You could just pay for GitHub to get private repos. If it's something this important to you, it's probably worth a few bucks a month.

This solution is also strange because if you ever committed anything private it will be in the history. So to make this work you either have to rebase over some history, or lose all the history.

> You could just pay for GitHub to get private repos. If it's something this important to you, it's probably worth a few bucks a month.

Github doesn't have a personal tier big enough for my private repos. (And I'd host my own before paying $50/month.)

I do pay Bitbucket, though, because I have a consistent and small-enough group of collaborators that it makes sense.

I have a lot of private repos. One for just about every little project I've done. Some larger projects have more than one. It would be pretty expensive to keep all of this in Github that way. The alternative is to combine a bunch of unrelated stuff into fewer repos (or not keep all of my projects in source control), neither of which is especially appealing.

I'd be happy to pay Github something for my usage and to support development. I believe in paying people for work that I find useful. But the pricing model makes the cost disproportionate with the value for me.

Or you could, you know, just take 20 seconds to set up a gitolite on some random server you have somewhere.

You are advocating for security through obscurity.

No, i'm advocating not to throw every weekend projects on a public github repo. This has nothing to do with security through obscurity. Nobody is supposed to have access to a bitbucket private repo but the owner and vetted collaborators.

I created a small tool[1] to help continuously audit public github commits for secrets, like aws keys. It uses the AWS provided regexes[2] to do to.

[1] - https://github.com/jfalken/github_commit_crawler [2] - http://blogs.aws.amazon.com/security/blog/tag/key+rotation

I'm as close to a "casual" user of AWS as you can get. I find IAM incredibly difficult to use. Any pointers?

Reply on news.ycombinator.com