A deeper dive into our May 2019 security incident
stackoverflow.blog
stackoverflow.blog
we had secrets sprinkled in source control, in plain text in build systems and available through settings screens in the application.
So many sites do this: allowing major changes to be effective immediately (like resetting credentials/password) by simply opening a "magic link" sent by email.
I think that this "immediately" is a major security antipattern.
I prefer it when such changes have a "cooldown" period of, say, 72 hours, during which the change is "ongoing" but not effective yet and during which the user can veto the change (say by either login on the site, where they'd then get a warning that a major configuration change is ongoing, and denying the change on the site or by opening another "magic link", sent by email, which allows to deny the change).
It's not a perfect solution but it stops so many of these oh-so-common attacks dead in their tracks.
Because there's a big difference between being able to read an email meant to someone (as happened here, on the server side) and being able to prevent a legit user from receiving emails while also being able to prevent that legit user from login onto a website with its correct credentials.
The hacker had access to messages from the sender side without access to the account being reset.
As others have mentioned though, if you as a user know someone has access to said password and you're resetting it as an emergency, that's 3 more days of some hacker being able to log in with your password!
I see your point, but a timeout is a suboptimal solution.
That's a different threat: an attacker knowing your password. Well... How do websites that allow instant credentials resets by email typically deal with a password reset asked by someone who knows the current password? Instant change too ("enter your current password / enter your new password twice"). And the good guy is locked out of his own account. I don't see how it's worse than that.
10 minutes like others suggested is way too short that said: this wouldn't catch attacks happening at night.
But to answer your question: it really depends what it is that you are protecting. For most sites I use by very far I don't see how 72 hours without access would be that problematic. Not logged in to StackOverflow for 3 days? Not a problem. Not logged in to HN for 3 days? Not a problem. Not logged in to Twitter for 3 days? I can live with that. Etc.
The question is: how much convenience are you willing to trade for security?
They are trying to find something that they don't necessarily know exists. They also don't know what tripwires exist and after trawling around for so long, they might even have assumed that SE didn't have any monitoring systems.
From the blog post, it sounds like the bit where they were working in the dark was getting things to run on a production database, but the SQL command to give them moderator privileges was prepared ahead of time (which makes sense, they had a local site set up and could prepare and test that part at their leisure). It seems very unlikely to me that they would have spent so much effort on getting that particular SQL to run for no reason, so the moderator privilege in particular probably had some appeal to them from a technical perspective. The main permission I can think of that you get as a mod is the ability to edit posts. It seems unlikely that an attack this sophisticated would be just for vandalism, so I consider the profit motive instead. “For sale: Access to Stack Exchange internal infrastructure. As proof, ask me to edit any post on Stack Overflow.”
What tripped them up was not a technical tripwire, but rather how intimate the SE community is with their moderators. This wouldn’t be obvious from the codebase; only if you’d spent some time on the meta sites would you be aware of the culture surrounding SE mods. On, say, Facebook, getting a global moderator bit isn’t something that a big chunk of the user base would have name recognition for. My stereotype is that someone who’s trying to break in to infrastructure (and not doing responsible disclosure) probably isn’t also volunteering in the review queues, and so wouldn’t realize what a risk this would be.
Of course this is all conjecture and we’ll never know for certain without tracking down the attacker and asking them, but I’d like to think I’ve constructed a reasonable scenario that takes all of the known facts and a probable motive and comes up with a plausible explanation for what was (in hindsight, at least) a fairly significant blunder.
Awesome writeup - this gave me a good laugh :-)
Zawinski's Law: "Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can."
The story behind that route might be interesting... See, originally Stack Overflow didn't have passwords - all logins were done via OpenID, so any credential management you'd need to do was done through your provider (Google, LiveJournal, myOpenID, etc) This made account recovery assistance pretty simple: given a verified email address, the system would just send that address an email that reminded the owner of any and all OpenID providers that they'd associated with their account. From there, it was up to the account owner to work with a provider to do things like reset passwords.
Skip forward a few years, and Stack Overflow had its own OpenID provider - now you could sign up with an email and password just like a normal site, except really you were creating an account on https://openid.stackexchange.com/ - so the recovery process remained pretty much the same, just with a new provider that happened to be run by the same company.
So far so good... Except, this was awkward to explain to folks. Really, that was what ended up killing OpenID: folks wanted a "Google" or "Facebook" button, not a whitepaper on fancy new authentication systems.
At this point Stack Overflow decided to try to streamline the login process, making signing up and logging in with their own provider seamless: no need to know anything about OpenID. Now recovery emails started including password reset links, and also reduced or removed information on other OpenID providers that were associated with the account in an effort to reduce confusion. The decision tree for generating those emails got complex.
And the decision tree for supporting users got complex as well. Support staff got frustrated; they'd been used to knowing what would and wouldn't be in a recovery email, and had a pile of templates ready to help folks navigate login issues based on that. But now they were getting replies back from folks who were confused and upset because their recovery email didn't contain information that the support person had asserted it would!
This was the genesis of the vulnerable route: a way for support staff to ensure that they were providing accurate information to users about how they could recover their accounts. By the time of this attack, it was already obsolete; the login system had been redesigned twice since the confusing and complex system that first required it. It was vestigial and forgotten... The ideal breeding ground for vulnerabilities.
(source: I worked at Stack Overflow through the time period described in this post, and was involved in support during the period when the relevant route was useful)
(I overlapped with Shog at Stack Overflow.)
Which is what makes this kind of stuff so insidious.
Still there is room for improvement. What confused me a lot recently was, that the reset link sent to a certain email is not necessarily for the login associated with that email.
I tried to to login at Stack Overflow after a long time. Entered my current mail and pw. Did not work, clicked pw recovery, received mail, reset pw, got logged in. So far so good.
Logged out, couldn't log in again. After a few password resets I realized that, while the mail was sent to my current address, the reset link actually was for the pw of a login associated with an old email.
At least for me that was not clear from the recovery email. Here is the full text with only email redacted:
> Account Recovery - Stack Overflow
> We received an account recovery request on Stack Overflow for new@example.com.
> If you initiated this request, reset your password here.
> You can use any of the following credentials to log in to your account:
> Email and Password (old@example.com)
> Email and Password (new@example.com)
> Once logged in, you can review existing credentials and add new ones. Simply visit your profile, click on Edit Profile & Settings and My Logins.
To be clear, "reset your password here." is a link and it changes the pw only for old@example.com.
So you end up with weird situations like this, where both the interface and the underlying system support multiple associated email addresses, but not in the same ways or with the same functionality exposed.
It is... A legacy system, with all that that entails.
> After attempting to access some URLs, to which this level of access does not allow, they use account recovery to attempt to recover access to a developer’s account (a higher privilege level again) but are unable to intercept the email that was sent. However, there is a route on dev that can show email content to CMs and they use this to obtain the magic link used to reset credentials.
Many of these debugging tools are great for devs to test things quickly but I've always felt very weary of having these exist in an app without some strict access control with 2FA. Ideally you'd not have them in the app at all, maybe just on local dev.
I'm imagining after a security issue is identified, the steps taken are roughly in the below order and close-ish for the date. I guess my question is, why does it take 20 months from start to blog post?
-Contain the issue (1wk) -Remove the threat (1wk) -Build up remedies (a few months) -Check and recheck what happened to make sure you're accurate when submitting final reports (a few months) -Release a blog post (1month)
The timeline is a cool day by day instance, but I just don't understand the larger timeline.
Discovery, immediate mitigation, deeper mitigation, general notice, notifying effected users - all these can happen pretty quickly once the ball is rolling. Once you're dealing with "the law" in any capacity you are constrained in what you details you can share broadly, and when.
I'm happy we were finally able to share this level of detail.
I have database connecting strings and password as ENV variables. But I still don't know what is the best practice. Lets say someone gets access to the server, they can still read the ENV vars, right? It definitely prevents from accidently checking in your code git repo. But still . Does anyone has good recommendation for storing credentials like database passwords in a way secured way.
Correct. Easiest way is to look at `/proc/$pid/environ`. It contains the \0 separated values for that process.
(FYI: I'm very familiar with BeyondCorp, as I was on an adjacent team when it was invented. Now I am an SRE at Stack Overflow when the incident happened.)
*> Hardening code paths that allow access into our dev tier. We cannot take our dev tier off of the internet because we have to be able to test integrations with third-party systems that send inbound webhooks, etc. Instead, we made sure that access can only be gained with access keys obtained by employees and that features such as impersonation only allow de-escalation—i.e. it only allows lower or equal privilege users to the currently authenticated user. We also removed functionality that allowed viewing emails, in particular account recovery emails.*
There was no "unauthenticated" access into dev - the access key here is what allows login at all to our dev environment, but the attacker was able to bypass that protection.
"2FA", as commonly implemented in many scenarios is weak and only helps address certain scenarios -- TOTP tokens, for example, are pretty trivial to compromise. Critical infrastructure needs hardened tokens and clients with more controls.
> Sunday May 5th
> ...a login request is crafted to our dev tier that is able to bypass the access controls limiting login to those users with an access key. The attacker is able to successfully log in to the development tier.
> Our dev tier was configured to allow impersonation of all users for testing purposes, and the attacker eventually finds a URL that allows them to elevate their privilege level to that of a Community Manager (CM). This level of access is a superset of the access available to site moderators.
EDIT: clarified that the report was held back
And affected users were notified once identified, which was shortly after the announcement: https://stackoverflow.blog/2019/05/17/update-to-security-inc...
This is an update with more details, which was held back for legal reasons.
2. Attacker was able to login to the dev environment with their credentials from prod (stackoverflow.com) by a replay attack based on logging in to prod.
3. The dev environments allows viewing outgoing emails, including password reset magic links. The attacker triggered a reset password on a dev account, and changed the credentials. This gives them access to "site settings."
4. Settings listed TeamCity credentials. The attacker logged into TeamCity.
5. Attacker spends a day or so getting up to speed with TeamCity, in part by reading StackOverflow questions.
6. Attacker browses the build server file system, which includes a plaintext SSH key for GitHub.
7. Attacker clones all the repos 8. Attacker alters build system to execute an SQL migration that escalates him to a super-moderator on production (Saturday May 11th).
9. Community members make security report on Sunday May 12th, stackoverflow response found the TeamCity account was compromised and moved it offline.
10. Stackoverflow determines the full extent of the attack over the next few days.
https://stackoverflow.com/legal/privacy-policy
GDPR anyone?
Clear as day that they're doing exactly that. You agree to this when you use the site.
However what makes it legal isn't if it is written in the TOS or in a cookie banner.
AFAIK what matters is either:
- if you have a specific, valid (according to the GDPR) reason,
- or if you have the users free and informed consent.
... and yes, I think a number of the things I still see on the web is not OK:
- dark pattern where if you click manage settings everything is opted out, but there's a big green "Accept everything" and a small bland "Confirm my choices"? Doesn't fly because the rule that it should be equally easy to opt out.
- Cookie banners with no real opt out? No way.
- Cookie banners where you have to deselect 927 "partners"? Also no way.
The only ones that seems legal are those who either uses a pure minimum of cookies for preserving state and those allow one to opt out directly but inform you that ads might become less relevant.