Slack's private GitHub code repositories stolen over holidays
bleepingcomputer.com
bleepingcomputer.com
Slack is selectively and deliberately limiting public access (discoverability) to the security breach announcements.
How do they even provide a plausible excuse for this? very bad look.
If you have a localised blog for multiple regions you will probably have your main blog set up to be indexed and your regional blogs set up to be no-indexed so that Google isn't indexing the same stuff twice.
From the linked article.
Placing meta tags at the bottom, out of sight ... an _interesting_ choice given the nature of the news.
Or, you know, to see if they change it in response to the article or the discussion here.
> In August 2022, Slack reset user passwords after accidentally exposing the password hashes in a separate incident. Unsurprisingly, that particular notice is also marked with a 'noindex' (both the U.S. and international versions).
Here all versions have the noindex. And it's 4 months after.
Best for what (except PR)? It's hostile towards users which in this case involves a lot of corporate ones.
As far as I can tell, Slack's not trying to hide this news.
Slack communicated about the breach here: https://slack.com/blog/news/slack-security-update
There is no `noindex` tag on that page (check the source).
There is a `noindex` tag on an alternate "en-gb" version of that page: https://slack.com/intl/en-gb/blog/news/slack-security-update
This is to be expected — it's presumably SEO equivalent to specifying a canonical URL.
It's a good practice, in all honesty. The time window is for the initial notification, after that there will be subsequent notifications when actual investigations complete.
Not gonna guess why, but if you play with the region in the footer:
English (US): it's listed
English (UK): not listed
English (India): not listed
You may argue it's not common, but I'd expect a standard to be determined by an authoritative and fairly representative body. Not by HN forum or a private company in isolation.
I think the accusation of foul play is somewhat overblown given there is a version that does not contain noindex.
[1] https://developers.google.com/search/docs/crawling-indexing/...
Using this query:
site:slack.com inurl:/blog/news/slack-security-update
https://www.google.com/search?q=site%3Aslack.com+inurl%3A%2F...
This is most definitely to burry the notices.
1. Slack
2. Okta (https://news.ycombinator.com/item?id=34081154)
3. Coa (https://github.com/veged/coa/issues/99#issuecomment-96169688...)
[0]: https://circleci.com/blog/january-4-2023-security-alert/
here's what it said:
We wanted to make you aware that we are currently investigating a security incident, and that our investigation is ongoing. We will provide you updates about this incident, and our response, as they become available. At this point, we are confident that there are no unauthorized actors active in our systems; however, out of an abundance of caution, we want to ensure that all customers take certain preventative measures to protect your data as well.
Action request:
Out of an abundance of caution, we strongly recommend that all customers take the following actions:
- Immediately rotate any and all secrets stored in CircleCI. These may be stored in project environment variables or in contexts.
- We also recommend customers review internal logs for their systems for any unauthorized access starting from December 21, 2022 through today, January 4, 2023, or upon completion of your secrets rotation.
Additionally, if your project uses Project API tokens, we have invalidated those and you will need to replace them. You can find more information on how to do that in our documentation here.
We apologize for any disruption to your work. We take the security of our systems and our customers’ systems extremely seriously. While we are actively investigating this incident, we are committed to sharing more details with customers in the coming days.
Thank you for your urgent attention to rotating your secrets.
Earlier last year.
I can see someone working at Slack try a nightly, they surely work with ML.
IMO the headline should be updated — much less than 1% of Slack’s private code can be accessed via github.com.
It is also worth nothing that other major breaches (LastPass) were preceded by source code breaches. The corresponding incident reports also included re-assurances that there was no impact on customer safety. The Slack incident report doesn't specify what type of Github repos were accessed, so it is hard to judge if any sensitive code has been leaked.
From the recent LastPass security incident report:
Based on our investigation to date, we have learned that an unknown threat actor accessed a cloud-based storage environment leveraging information obtained from the incident we previously disclosed in August of 2022 (source code breach)
There is no `noindex` tag on that page (check the source).
There is a `noindex` tag on an alternate "en-gb" version of that page: https://slack.com/intl/en-gb/blog/news/slack-security-update
This is to be expected — it's presumably SEO equivalent to specifying a canonical URL.
This is so important. Saying “less than 1% of code” is not very useful as a passwords or config repo being leaked may only be a few bytes. It’s also not customer data. But it’s extremely important.
I think the fact that they are being not forthcoming means they are clueless, or something really bad happened and they are weaseling and hoping nothing bad happens (spoiler: it will).
Is it wise to simply keep private repositories away from GitHub at this point? It seems the best way to avoid being drawn in with the rest as a target.
Internal bad actors? So location wouldn't matter?
> The incident involves threat actors gaining access to Slack's externally hosted GitHub repositories via a "limited" number of Slack employee tokens that were stolen.
> While some of Slack's private code repositories were breached, Slack’s primary codebase and customer data remains unaffected, according to the company.
There are "application credentials" or "API tokens" that get generated for various purposes and live long.
There are many programs that do not support "credential helpers", abd those would need you to input an API token that is generated usually just once and has almost always has enough permission to clone a repository (at a minimum).
The `hub` and `lab` CLI tools work exactly in this manner.
If you are familiar with intellij: certain plugins that integrate with github or gitlab also require the generation of an access token so that you can do API calls to see merge requests and such: https://docs.github.com/en/authentication/keeping-your-accou... & https://docs.gitlab.com/ee/user/profile/personal_access_toke...
Applications that support credential helpers are docker (for optionally authenticating with docker registries) and git itself.
But does it solve the problem? It prevents the keymatter from leaking, but the problem is that the API key was put somewhere sensitive and then stolen. Presumably you'd need access to be able to use the keymatter from any location where you'd normally need API keys: CI machines, developer workstations, production, etc.
Worse, if you have some machines (CI, HSM) that are accessed over network traffic and that traffic is secured with an API key, you are back to square one again, where a stolen API key allows for continuous access.
Would this help for auditing? No, not really. You can already have plenty of auditing and alerting to try to prevent exfiltration anywhere an API key is used.
In other words, probably not. It might help a little bit just by making exploitation harder, in optimal circumstances, but it doesn't fundamentally change the problem; if dev workstations or CI are compromised, you're going to have a bad time.
But then you lose convenience.
Slacked picked convenience and it led to the headline: "Slack's private GitHub code repositories stolen over holidays" and the top voted HN comment so far is: "Slack is selectively and deliberately limiting public access (discoverability) to the security breach announcements.".
The question is: was convenience worth it?
There are other tools. There are other ways to host. There are other ways to do CI/CD.
But it's less convenient.
It's a tradeoff.
* gitlab hosted internally * VPN needed to access gitlab * VPN requires a gsuite login, with 2FA * You can only login to a gsuite account on a employer-provided machine (so no access to anything on a non-employer machine, even email)
You'd have to steal a company laptop, or social engineer yourself into a building with a desktop machine, to even start to get near the code repository.
The "must login from a employer-provided machine" thing can be disabled on a per-user basis by remote policy (via an administrator), if needed.
How is this achieved?
This sounds like the equivalent of "Use MacOS because most viruses target Windows". Which some people do, and yes it probably lowers your risk, but there are ways to use Windows (and Github) securely.
If you're really paranoid about your source code, you should probably self-host your own instance of Github/Gitlab/etc (and lock it down through VPNs, IP white lists, etc) rather than using their cloud service.
This approach also gives you first hand access to logs that would expose internal bad actors, if that's included in your threat model.
No proprietary software you have no control over or ability to audit can ever be considered secure.
Going to respectfully elaborate on this specific point, with full awareness that it's not core to the discussion, and that my opinion is my own and highly subjective, and that I'm not fully disagreeing but wish to point out a nuance. There are too many ways to put hidden software on a Windows system, which makes it harder to use securely than my OS of choice, macOS.
It's more difficult to audit a Windows system than a macOS system. For example, background processes can be installed on Windows in dozens of ways. Go download AutoRuns (https://learn.microsoft.com/en-us/sysinternals/downloads/aut...) and look how many tabs it has. The docs even say "You'll probably be surprised at how many executables are launched automatically!". UAC mitigates this somewhat, but it lacks granularity; it's all-or-nothing so you'll click Allow without knowing what's actually going to happen. And then you have COM registration and win32 APIs that let applications get at other windows easily. Layer upon layer of legacy compatibility make the whole system feel like a mess - at least to me.
Using macOS securely is also quite involved of course - for example auditing Homebrew's behavior when it downloads half the open-source world just for one package. But when it comes to stuff running in the background, by comparison, macOS Ventura recently added a comprehensive list of launchd agents and other daemons that run at startup; each of which can be toggled (though the UI doesn't allow you to drill into them). It's all on one pane in System Settings, built into the OS, no need for Sysinternals' separate tools.
Using macOS securely is a lot easier, in my experience. There are ways to use Windows securely, but it's a lot harder, and (I think) that makes it not worth the trouble. Using macOS securely is a lot closer to just using macOS.
That said, for any OS, it goes without saying to only install trusted software. It's possible to cleanse both OSes fully of software you no longer want, but on Windows it always feels like surgery, whereas 90% of cleaning something from macOS is dragging the application bundle to the Trash.
The problem is that there are some resources where the benefit to the attacker goes up faster than the company can afford to cover, and the company has to cover all the attack avenues. You can imagine that the value of Slack's source code can be high to certain people, and it can be difficult to completely seal off something like that when source code by its nature pretty much has to be distributed to hundreds or thousands of people. (At least overall; any given source may not be that distributed, but there will be thousands with some sort of access.)
Major corporations basically have to act as if source compromise is inevitable. Best practices obviously include partitioning who can have access to what (perhaps the best practical argument against a one-company mono-repo, if de facto it has to be broken up by access permissions anyhow) and not including anything in the repo that is directly security catastrophic if it gets out (security secrets mostly), but this is still limiting blast radius rather than "solving" the problem.
It's a perfect storm of being extremely expensive to cover, in terms of money, internal process friction, and having a lot of heterogeneous vectors you need to cover, and for certain companies, being very high value to a number of different kinds of entities.
Being in GitHub specifically would only matter if GitHub was itself compromised. GitHub can not do very much about legit credentials being stolen through completely non-GitHub related means. (Most things, if not all things, that might leap to your mind will also block legitimate usage quite a bit. No matter what crazy thing you can imagine, someone's probably doing it somewhere and it's probably mission critical.)
I expect there have been a great deal more source code compromises than we've heard about.
Slack is more than its code. The product is really the aggregate of their engineer's knowledge and internal processes. It's not practical to steal code and build a business or spinoff product around it.
The only legitimate threat seems to be the potential for exploitation. I suppose this might also threaten any backend improvements (economic leverage) they made with proprietary algorithms.
It is bad because regardless of how infuriating and frustrating the user experience is, people are still gonna be forced to use it under the massive pressure of Microsofts weight.
They rewrote parts of it to React when they made the weird half client for Windows 11. Not sure why those two clients haven't merged yet.
They are actually per repo.
https://github.blog/2022-10-18-introducing-fine-grained-pers...
Seems like a good step forward; I'm still somewhat shocked it took this long, people have been pointing out the security issues for at least 5 years.
The cloud remains someone else’s shared computer.
A great time to self-host.
More in this podcast https://darknetdiaries.com/episode/19/
For smaller companies something like Cloudflare Teams and/or Tailscale would probably also add a good layer of extra security. I guess it's important to have some device whitelisting in addition to credentails, so that stolen credentials alone are not enough to access code or chat.
Sure, for some companies, it's really important to own their data completely. I get that (although I personally trust Salesforce more than most one-off companies when it comes to security). But I don't think it's fair to say that Slack is the HipChat of today at all... it's like iOS vs Android, both are just different flavors of a really good product.
The potential impact here is an attacker now has access to some of their code which could let them find and take advantage of vulnerabilities resulting in customer data being accessed.
Technically "potential impact" is correct but I think companies often underplay how severe a source code leak is. It's the exact blueprint of how their app is built.
No, but I don't think that's a fair comparison. An open source tool or app is open source by choice so they have an advantage of knowing what they're getting into.
In Slack's case maybe they have a bunch of undocumented APIs which are publicly accessible and now with access to the source code you know what they are, and when you hit them they result in customer data being returned. It runs in their production environment off the live site so it doesn't involve anything crazy like you needing VPN access to their DB to get production data.
That's just 1 basic example of what could happen when a private code base becomes public due to a leak. I'd like to think an open source site wouldn't do that because at a fundamental level the app is built in the open. Also, there's likely many sets of eyes from different folks looking at it from different angles.
Having undocumented publicly accessible API endpoints isn't an option in an open source world but a private code base could maybe get by with security through obscurity on a few things thinking "well, the code is private...".
Private/proprietary code base != secret code base, there is a great number of ways how a code base might leak - past employees etc..
Yeah, no, this is a security bug.
Meanwhile if you allow a closed code base to grow and mature for years and then expose it to the prying eyes of the public, it might contain a few bangers that would have been caught much earlier if the code had been developed in public.
Of course this assumes that vulnerabilities get fixed and dev teams learn to avoid them once they've found them. This somewhat breaks down in typical plugin-heavy open source stacks where any random plugin might be developed by a 13 year old in their bedroom and no longer maintained and your flagship product now depends on it because nobody noticed its glaring security flaws. Though this can be managed and avoided somewhat by being more deliberate when picking out plugins.
EDIT: A really straightforward example is committing unencrypted secrets (API keys, passwords, whatever). In an OSS product this will likely be exploited almost instantly but this creates cultural awareness to not do that (either by learning from mistakes or by being told horror stories about those mistakes). In a closed source product this could easily go on for years with no consequences until someone exposes the code to the public.
I've done contract work for a bunch of small businesses (1-50 dev team sizes) and the amount of bangers I've seen are quite high. It's a whole different world when the expectation is your source code is private and the code has been been around for 2-10+ years. Committing secrets is pretty common (sometimes accidental, sometimes on purpose) but it goes way beyond that type of thing. There's a whole culture around the app being private.
Survivorship bias is definitely real. Also "given enough eyeballs, all bugs are shallow": https://en.wiktionary.org/wiki/given_enough_eyeballs,_all_bu...
For a software company the code repo is the #1 asset.
if AWS/Amazon's entire source code were leaked tomorrow, how would competitors use it to their advantage?
Would it help Lyft at all if they gained access to Uber's source code? Probably not.
Is some startup going to be able to take Slack's source code and use it to build a competing service? Probably not, it would be easier to write the code from scratch and reverse engineer features.
Source code is definitely a company asset, but most companies wouldn't be incredibly damaged or threatened in any meaningful way if their source code were exposed... unless said source code exposes security flaws that are used for some attack on the service. But security through obscurity (e.g. keeping insecure code secret) is not a valid way to keep things secure.
As long as you do it outside the jurisdiction of the USA, you'll probably get away with it.
I have seen two instances of this in my professional career (can't talk about either I'm afraid!), but in both cases it had a pretty big impact on the original owner of the code, and in neither case did the company manage to get any compensation.
If I were given $50 million to build a Slack competitor, even if I had Slack's source code I can't imagine it would be that helpful. More than likely it would be easier to rewrite and re-architect in order to not inherit all of the technical debt of very old software.
Not to mention in most cases, access to the source code doesn't mean you can easily recreate the AWS / Google Cloud / Azure environments necessary to run the source code reliably with scalability.
Edit: There are obvious exceptions to this, but most apps these days don't require a whole lot of proprietary algorithms that can't be easily cloned without source code access.
We really needed a reaction against this nonsense which wasn't just adblockers.
Bummer.
Lack of real Firefox on iOS is the absolute deal breaker for me switching to iPhone
It's quite effective at defanging the annoying ads without breaking things I care about.