Weak passwords brute forced
github.com
github.com
Writing it down and leaving it exposed in a public place, like taped to your monitor - bad. Writing it down and treating it like any other important asset - not that bad.
Passwords are HARD to remember but it's better to have it under hand than to ask for a new one every two days because we didn't have time to remember it (my school stored students password in clear text in their DB and gave them out printed on paper sheets. Four years later, I still have this sheet in my wallet, my three passwords never got compromised and I memorized each of them after less than two weeks of heavy usage).
> IT professionals usually run screaming for the hills if you talk about writing a password down
I can understand that. You and I keep our passwords in our wallets. Some employees keep them on post-its taped to their computer screen at work.
If a password I keep in my wallet gets compromised because it's been stolen with the wallet, I have more pressing matters to deal with than "oh crap, a random guy who stole me my wallet in the subway may use it to access our repo and add bad code in my name".
The point about "not that bad" is that security is an illusion: you can use a thousand passwords everywhere with different set of keys for github, server A and server B, change password every week and never write it down, plus a little bit of tin foil hat paranoia,... But in the end, a $5 wrench will make you spit out all the access keys needed by someone who REALLY wants an access.
Solution: https://confluence.atlassian.com/display/SOURCETREEKB/Two-Fa...
I'm waiting for the day that the security people work out a solid federated single-sign-on system that gains widespread adoption.
"While we aggressively rate-limit login attempts and passwords are stored properly, this incident has involved the use of nearly 40K unique IP addresses. These addresses were used to slowly brute force weak passwords or passwords used on multiple sites. We are working on additional rate-limiting measures to address this. "
I'm sure a similar situation would work here, especially with a decent geoip database. If you get 3 - 5 failed attempts on an account separated by more than a few miles (and not along an interstate :-)) you can probably assume someone is using a botnet to try to break in.
Before I switched my sshd ports a few weeks ago, I often saw break in attempts on my servers that would imply checking only a single username and a single password.
My guess on that is that bruteforcers, rather than testing thousands of usernames with thousand of passwords on a single host will try a single username and single password on thousands of hosts.
http://bsdly.blogspot.no/2013/10/the-hail-mary-cloud-and-les...
I double checked my log archives, it seems not to be exactly the same, though. In what is reported here, a same username will be tried with several password, allowing something like a minute to pass to make a new attempt.
In my logs, this is really just a single attempt. Other break in attempt may happen something like two days after, but with an other IP and username.
It may just be yet an other evolution of the same botnet knowing its previous pattern has been spotted, though.
(and in case you wonder, yes, I always have the nose in my logs, having a dedicated monitor for that :) )
Now that is dumb. I expected more from GitHub.
Looking at the security history page I see a lot of failed login attempts. Makes me glad I enabled 2-factor-authentication!
It's a great way to distribute Open Source software. (Previously Mattt hosted it on a personal Amazon S3 account which he paid out of his own pocket; now bandwidth is generously paid for by Github)
If someone would slip in rogue code - it's quite likely some to many would actually run it and deploy it. Especially if it's a fast moving piece of software - like being so rapidly developed that distribution packages can't keep up for either time or stability reasons, leading people to compiling/deploying from source themselves.
- Of all users, only a small subset would have private repos
- Of those users, only a small subset would have private repos that would be of interest to third parties
- Of those users, only a small subset would have a weak enough password to allow brute force
To reverse it, of these accounts that were hacked, I can't see many of them having private repos that would be of interest.
And if so, then this would seem a bit excessive.
I know that's a lot of ifs, but it seems reasonable. I would be interested to see the number of total accounts vs. the number of accounts w/ private repos.
- That private repos are the only thing worth targeting. What if you could inject a trojan into a popular open source project? You could do a lot of damage that way, probably way more than on private repo, because so many people incorporate them in their products. Imagine they hacked the Rails repo, for example. Worse, some repos host binaries, for which a meddling would be harder to detect (a bad idea, but doesn't mean it doesn't happen).
- That the users being attacked are random and not specifically targeted based on who the user is and what the work on. Not sure if that's the case or not, but I see no reason to assume it.
A good bet to find people with private repositories, would be to aim for all or a subset of those who belong to an Github "organisation".
Likelihood of valuable user greatly increased.
The way around this problem is to add source IP into the rate-limiting algorithm so that I cannot mess someone else up.
Unfortunately that can be worked-around somewhat by using 40k different source IP addresses.
No perfect answer. Personally I think GH did ok here.
http://stackoverflow.com/questions/549/the-definitive-guide-...
TL;DR - compute the average number of system-wide failed password attempts, and if it's over the norm, impose small delay on all users (except those that login via a persistent login cookie).
Of course, this works only if the number of attacks is small enough.
An even more secure approach is what Linode does. If you log in from a non-whitelisted IP, they require you to whitelist the IP via an emailed link, then you can login.
But that's overkill and the first one is a lot better for casual websites. You store the user's last login IP and you wait for maybe 1024 failed attempts not from that IP, which ought to be enough to suggest a bruteforce attack. You're protecting "bluedog4", not "password" or "1234" which are going to get cracked anyway.
Two-Factor Authentication is great but even without, it would seem that they could more aggressively lock out any new IP address which is failing password retries several times. (but continue allowing log-in from trusted IP's.)
There is a new SSID with WPA2-EAP replacing the captive portal and the usual DHCP/NAT business replacing our massive waste of IPv4 space, but that was implemented last year. This introduces the opposite problem: one person out of thousands can't remember their GitHub password so GitHub mysteriously drops all traffic from everyone on campus.
IP addresses don't identify people, computers, or usefully discrete units of any kind.
While it might be possible to do all this correctly, it's just so easy to do wrong that it's hardly worth it. Given that there are existing better ways to do it, it's the kind of thing you probably don't even want to mess with.
That would prevent a practical joker at work from locking you out unless he went to a lot of extra trouble.
You attempt a login, they send an out-of-band request for confirmation. You do the confirm, and they let you in. Like a superfast streamlined version of the "reset password" email.
At any rate, don't store really sensitive stuff on Github. It's a bad idea for many reasons, security flaws being one. In particular, keep in mind things like AWS server credentials which might go in your repository.
IMO something similar to SSH security policies like allowing logging in from a set of IPs and/or without password i.e. using public key, may be a good idea.
foobarjones:123456
foobarjones:passwd
foobarjones:test
foobarjones:123456
drsmith:123456
ilikeorangejuice:123456
...
foobarjones:passwd
drsmith:passwd
...The attacker probably tried a password like 'github' on the first pass and 'password' on the second pass.
Luckily the Adobe leak was the final straw and I finally started to use a good password manager and super strong, distinct passwords in all the services I'm using. It wasn't that bad at all, just selecting a good password manager[1], strong encryption and an easy way to sync your passwords between devices (git), I've had no trouble at all with strong passwords.
Although it would be nice to have an Android app for the password manager. I think it's a nice excersize for writing my first Android app at some point...
Its worth mentioning, though, that the rate limiting is at the account level and not the IP level. When I was trying to get access to my account before the email was sent out, my account was locked and remained locked even after developing a new Tor circuit. How were the attackers able to circumvent the aggressive rate limiting? Three password attempts a minute is a generous estimate and it would take longer than the expected life of the universe to crack even a moderately secure password at that rate (my old password had 704 decillion permutations.) Perhaps some part of GitHub isn't being rate limited?
Nope, none of the failed attempts on my account originate from a tor exit node.
Thankfully that even with that many unique IPs and presumably a few tries per IP, you still only have enough tries to crack weak passwords.
More interesting is the attempt. I frequently scan customer systems on their behalf, by spinning up clouds of resources to hammer their login APIs. By using Digital Ocean $0.05/hour boxes, and massive threading, tests tend to cost me ~$50/hour (yes, that's 1000 boxes orchestrated).
Given the real world cost of cloud computing, and the ease of libraries such as libCloud (or ruby's: FOG), the cost of doing this in terms of time and real money has dropped dramatically.
oauth_access.create: GitHub XRP Giveaway - 16 days ago
user.login: Originated from 23.29.121.166 - 16 days ago
With a feature like that, if an IP address from country C attempts to log in to accounts of users in separate countries A and B, they are quickly spotted.
Github is, for better or worse, a high-value target these days. Compromise of a Github account will often get you credentials which you can use to compromise high-value targets, for example production machines.
(Example from the Rails world: if someone gets read access to your git repository, expect them to figure out your secret token which you use for HMACing sessions to prevent forgery. If an attacker can forge sessions, they have remote code execution on your production servers. You can assume they will go directly from code execution to root on those servers, and from root on those servers to a systemic compromise of your entire network (if they want to do that).)
Use your imagination on what a mass compromise of accounts gets you. The simplest possible example is grepping for /bitcoin/, finding the full source code and credentials for a Bitcoin exchange, and rooting them then emptying the hot wallet. That results in fairly obvious economic advantage, right? You could also use scriptable attacks to root thousands of big-n-beefy production machines on fairly trusted IP addresses, then add them to a botnet, which you'd use for spamming or various other nefarious activities.
Then, even farther down the list, you have a dedicated adversary actually go through every repository they got access to and look for something uniquely fun to do with that particular target.
If you were looking to do industrial espionage on a particular target, you'd probably pick a way that was less likely to be detected and cause a company-level security response. (Use passive recon to identify one or a few likely target email addresses, find one or a few likely passwords for them, and try them first. Heads you win, tails you just left evidence in a log of perfectly normal user behavior. You might follow up this perfectly normal user behavior with a social engineering attack.)
Yeah, spear phishing / social engineering makes more sense for industrial espionage.
I am weakly confident, on the basis of some spelunking in the Rails internals in February, that if you get attacker chosen data into the session you're uniformly hosed on (at least) Rails 2 and Rails 3.
I wonder if you could expand on this? I'm not too knowledgeable about Rails or security, so I'd wonder how that works?
See: http://www.exploit-db.com/exploits/27527/
This can be done in an entirely automated fashion and requires no knowledge of the application other than the session's secret key.
In theory, an additional precaution one could take: change the default session serializer to `JSON`. By using `JSON` instead of `Marshal`, you're limited to storing strings, hashes and arrays in the session, but that's a good thing. The remote code execution vulnerability takes advantage of `Marshal` deserializing the session and loading objects deep within the Rails process in order to run arbitrary code. Take away that ability by using `JSON` instead of `Marshal` and you should no longer have to worry about this particular attack vector.
After a brief spelunk of the Rails codebase (and not getting very far, my laptop bricked itself today) there doesn't seem to be an easy way to do this, apart from finding the instance of `ActiveSupport::MessageVerifier` in your rails process and then running something like `verifier.instance_variable_set(:@serializer, JSON)`.
"We have reviewed our logs and it doesn't appear that any actions were taken by the attacker other than to authorize the 'GitHub XRP Giveaway' application against your account.
You should be able to find the OAuth events for that application in your account's security history:
https://github.com/settings/security
We do not believe that the application's authors were responsible for the break-in, rather that the attackers were attempting to game the giveaway.
Ripple's explanation of the giveaway can be found here: https://ripple.com/blog/git-in-the-game-2020-xrp-giveaway-fo...
As of this comment, 2020 RXP is worth ~ 0.03 BTC. Multiply by ~$500USD/BTC and you get ~$16 USD (over $20 when BTC was peaking $800+USD/BTC in the last couple days). Multiply that by the number of compromised accounts that meet the cutoff date criteria, and you get the take.
Potentially some good money depending on your success rate, but maybe not worth the cost of renting a botnet?
From the 3975 RXP I could send, it turned to ~85 USD the other day.
Multiply that by 1698 and you get $167k. Pretty sure it was a win for them.
Where did you get the 1,698 figure -- was that the number of accounts compromised?
My password has not been reset by github but I have updated it anyway.
On a sidenote I think that is a very clear and well written overview with clear details and a nice reminder about a section of the site that I was not familiar with.
If the account has an email address connected to it, you can also try the password against the e-mail account. That can be worth quite a lot if it works.
> 18 hours ago user.failed_login: Originated from 178.245.129.47 (Istanbul)
> 3 days ago user.failed_login: Originated from 180.250.45.186 (Indonesia)
> 3 days ago user.failed_login: Originated from 123.119.141.184 (China)
Is this a distributed brute force attack against my account?
This obviously won't work for everyone, but might prevent a decent amount of brute force successes.
> These addresses were used to slowly brute force weak passwords or passwords used on multiple sites.
..."multiple sites" could possibly refer to a compromised list, like Adobe? Of course, it doesn't have to be Adobe - Macrumors was compromised recently, etc etc.