Slack Security Incident | Hacker News Reader

Slack Security Incident

keybase.io

308 pointsmalgorithms104 comments

Let's ignore the rather awkward self promotion, and the fact that 2FA would have prevented this specific incident.

This is the important part, which everyone should think about:

> What would have been way worse — immeasurably worse — is if our team had used Slack for anything other than what we did use it for, which was discussing outages of our own product. Had my cofounder and I discussed our company's cap table, or business partnerships, or compensation agreements, or ongoing legal matters over Slack; or had our team traded API keys, or security-sensitive matters; or had we controlled mission-critical infrastructure via Slack-powered "bots"; we'd be sweating bullets to this day that our important company secrets were out in the open, about to resurface at the worst possible time.

I see Slack being used for everything at a lot of companies.

There are a lot of interesting things in the chat history everywhere: SSH/API keys, logins and passwords, sensitive internal documents, chatops bots that would let one take control of the infrastructure, and worst of all - juicy office affair gossip.

Combine this with often lax user management (forgetting to disable old accounts, inviting people to channels they shouldn't be in, ...).

Most companies overlook this and don't even have a policy for what's OK to post on Slack, and what isn't. Not even to dream of any kind of enforcement.

I'ts a big security problem, even without Slack getting compromised, and should be on the radar for CTOs.

> the fact that 2FA would have prevented it.

Is that so? The article states: "If the attackers inject server code, 2FA or U2F or any Web-based security practice does little."

It won’t prevent an attacker from logging in as you at the moment, but it will prevent them from using stolen credentials later on.

If they can inject server code, they can bypass 2FA entirely. They don't need your 2FA code they'll just skip that part of the authentication.

The same goes for passwords, but with passwords there is the potential of additional value on other sites that haven't been compromised so those are always worth collecting.

This particular scenario involved injecting server code in 2015, then waiting until 2019 to use the credentials they collected. They would not be able to bypass 2FA here.

It’s true that 2FA wouldn’t protect you from having your account compromised immediately, but that’s not what happened.

I don’t quite understand that bit, the attackers could at the time steal plaintext passwords - however how does that affect 2FA benefits several years after the attack?

It sounds like he’s mixing a couple of things up there unless anyone can explain otherwise?

Slack has some options to delete all messages over N days old. Unless there is a really good reason not to, turning on this feature generally sounds like a good idea. At least you can drastically limit the length of the archive available to any attacker.

I strongly suggest not transmitting secure information or files via Slack or any other medium that retains or broadcasts (push notifications, email alerts) data.

Given the massive and escalating fines over customer data hacks and leaks I suspect we will see laws requiring data retention going head to head with consumer privacy laws. Given that even Jeff Bezos can’t keep his communications secure, the outlook for infosec consultants is fantastic. If anti-encryption laws start appearing on top of all of this, it’s going to be an absolute bonanza.

That's a terrible feature in an instant messenger client for work use? I cannot recall the precise details of my conversations with smart people who know things I do not. I can recall that I had the conversation and remind myself what I learned before.

Depends entirely on the conversation obviously, but perhaps important details should eventually be coalesced and moved to a more permanent document or project management system before the deletion deadline. It's a good discipline to have.

As with many things in life, it's a tradeoff. Do you want to accept a higher risk of security incidents or make it easier to recall information from a long ago IM conversation?

There are rational reasons for an organization to do either.

I don't think anyone would argue with the value side of the question. The tricky part is calculating the cost side. (The cost of not deleting old messages.)

Surely we can agree that the cost side is > 0.

I'd argue that the benefit side is generally at least a little lower than what we think it's going to be. And if the expiration limit is set to say, 1 year, the cost of deleting old messages goes down considerably.

Doesn't that have compliance implications? My understanding is that digital communications usually need to be stored for 3+ years in the case of a lawsuit and longer if you are in a regulated industry.

At my last job, I wrote a little tool to download hipchat chat logs and scrape them for anything that looks like a password. There were tons!! I raised it to the CTO/CIO, and while they sounded interested in it, they never enacted any policies, etc, to prevent the sort of password sharing that anyone can apparently easily take advantage of.

In fact, the CIO asked me to send him a spreadsheet of when/where passwords were shared -- and then never did anything with it, despite the file essentially being a password map of the infrastructure. When I asked him if he deleted it, he told me he couldn't because the company's lawyers wouldn't allow him to. Lol.

It's insane out there.

The correct answer is routine password rotation, but of course that's stupidly hard to do.

I don’t agree with that. Regular password rotation increases the opportunity for phishing attacks because people become used to sleep walking through it. And users are generally just adding one or two characters to the password anyway.

Instead, that energy is better spent on requiring strong passwords and people using password managers and two-factor.

Password rotation in this context is for the infrastructure, not normal user accounts.

There are other options for securing/managing infrastructure access (e.g. PKI, Hashicorp Vault), but if you're using passwords, it's a good idea to rotate them if only to encourage good practices around automation.

I'm not referring to user accounts, I'm referring to password management for things such as database and/or internal APIs.

What did you use for "looks like a password"?

I use a python library called 'passwordmeter' to check the password 'strength' of tokens. If the token's password strength is high, it's either going to be a password, a hash, a key, or perl (half joking). It's then just a matter of ignoring the false positives - about 95% - but that's not as time consuming as it sounds considering the reward.

The precursor to the hipchat scraper: https://github.com/red-bin/password_finder

Cool, thanks!

Do Keybase folks use Keybase for literally all their communications, document sharing? That's a bit hard to imagine. I'd wager that some of those important comms were on email and pretty much all the scary arguments about server/account hacking would apply to the involved email accounts too?

And a naive interpretation of the posting also raises the question of what secret deals, etc could Keybase be doing that would be disastrous if it were to come out later, that, as a Keybase user, I should be concerned about? A secret backdoor in the keys embedded for the NSA?

(Keybase user here)

Ive been considering building a product that would help eradicate "lax user management" of Slack, and like platforms. It would basically keep an audit trail of managers approval for their specific team members they manage access to specific platforms and products. Basically, helping scale that one IT guy whom should somehow know every person at the company in detail.

If I succeeded in making the process easy & less time consuming than it is now, do you think it's a product you would pay for?

> or had our team traded API keys, or security-sensitive matters; or had we controlled mission-critical infrastructure via Slack-powered "bots"

That's why I always strongly recommend having a https://privatebin.info/ . In fact, in every startup I have been, I setup an instance of that forcing "burn after reading" and as a policy all credentials and sensitive info goes there when sharing.

The author would have done well by refraining from using this as an opportunity to make a sales pitch for their startup, as it detracts from an otherwise important message.

Let me see if I have this right:

Slack had a major security breach in 2015. Apparently someone installed malicious code that could even read password inputs in plaintext. They waited 4 years, after growing large and going public, to inform affected users. And in the interim they blamed their users for any related security problems.

Do I have this correct? If so, how is anyone going to defend this situation? And how can anyone put any sensitive data on Slack, or tell their company to do so, and feel good about it now?

I expected some stupid apology note from the CEO on their website if this turns into a bigger issue, which is sort of an anti-pattern at this point...

Would shareholders have a case to make for fraud here? Slack clearly didn't want this information getting out pre-IPO, as a security disclosure in this case would certainly impact public confidence in the company.

bloomberg.com/opinion/articles/2019-06-26/everything-everywhere-is-securities-fraud

Every tech IPO filing has a generic statement saying something like "our software may contain bugs, including bugs that we cannot fix blah blah blah." So unless Slack has made fraudulent statements about this specific breach, I doubt they've done anything illegal WRT securities fraud.

I'm sure shareholders will sue, because they try to turn everything into a violation of securities law. But securities law shouldn't be the only way to regulate companies imo

This is large companies must notify the authorities within days of such a data breach in the EU now - no room for "guessing" whether or not the companies should be sued over this type of fraud.

I think the point was a sales pitch. The post worked well enough to get to the top of the Hacker News front page. I think the effort was rewarded.

I don't even dislike the pitch, tbh. Rarely do I view these posts as innocent / without-pitch. So seeing the pitch is almost appreciated by me, as at least he's trying not to hide it.

With that said, I wish he'd have gone into more of a pitch - specifically why Keybase might be unable to suffer from this specific problem. Or at least, why the attack surface area is smaller with Keybase, etc etc. He goes into a bit, but not a ton, I could have used more.

Most things related to Keybase go to the top of Hacker News. The company has many HNers.

I like their products a lot; I wish they were a bit less self-promoting, as it is in fact the case here.

The issue looks to be that they thought they had informed all the affected users back in 2015, but underestimated the set of affected users. The breach certainly wasn’t secret until today, they posted it publicly at that time: https://slackhq.com/march-2015-security-incident-and-the-lau....

they failed to mention that malicious code was reading plaintext passwords though

I also believe they were aware of this problem in March of 2019 as I was forced to do a password reset on my affected Slack account from my password manager last updated timestamp.

This is the "lite" version of his sales pitch. The real sales pitch was the email that Keybase users just received. It said, basically, that Slack was compromised but if you would have used Keybase instead, you wouldn't have been compromised.

Awkward, indeed.

I cringe when I see companies try to use a competitor's misfortune to their advantage.

It's not using competitors misfortune if they didn't reasonably disclose the breach

What makes this even more sad is the extreme difficulty you'll have if you attempt to remove your company data off of the Slack platform. (Disclaimer: I stopped using Slack 2 years ago)

Our company used Slack extensively for multiple years. A couple years ago, we decided to stop using Slack for official company communication. After switching to alternative communication tools, we tried to delete the data in our Slack account and found it to be nearly impossible.

I recall using 3rd party python scripts (opensource on Github) that took hours to run - the script used an API key to fetch and delete messages individually.

We also tried using Slack's Admin panel to delete messages. At the time, I believe it required clicking a checkbox next to every chat message we wanted to delete. Clearly not a realistic way to scrub an account.

The sad reality is that with many services like Slack, once a provider has your data, there's often no easy way to remove it. IMO, this is a major downfall of our reliance on modern SaaS services (companies have no incentive to prioritize features for deleting account data - the only users who would find those features useful are already churned customers).

Sounds like an opportunity for a third party tool!

I don't understand this. They say that if you delete your workspace, all messages are gone.

https://get.slack.help/hc/en-us/articles/204067366-Delete-a-...

It says they're "irretrievable", which isn't necessarily the same thing. That you can't get that data back doesn't mean it's no longer stored by Slack in a way that a sufficiently severe compromise of their infrastructure might reveal.

I doubt they keep customer data after you're not a customer anymore. Sure, they could be lying but we don't have information pointing to that, do we?

They are subject (or have subjected themselves) to various security standards: https://slack.com/intl/en-ie/security

And their security whitepaper [0] mentions:

Customer data is removed immediately upon deletion by the end user or upon expiration of message retention as configured by the customer administrator

It would be a stretch to say a message is deleted permanently but a workspace is kept forever, for unknown reasons. Occam's razor and stuff.

0 - https://a.slack-edge.com/78b2/marketing/downloads/security/S...

It probably means the data is encrypted at rest and they discard the encryption key. Of course I doubt we'll ever know because they won't give us that level of detail unless they are compelled to.

At the time, there were reasons we couldn't completely delete the workspace. I believe there was certain data we wanted retained, and other data we wanted removed.

Wow - for a sales pitch fantastic. Many of these security issues leave you little to actually do. This write up provides an alternative.

What’s super bad here is slack misleading about the cause wasting all the users time.

Quick question, anyone use key base - can u give a quick review? Team currently use slack

I have used it in conjunction with Slack; a channel for infrastructure team members to share sensitive details, or to direct message secrets to people. It works well for that light usage.

I cannot imagine replacing Slack with it for all company or team communication though. It's not remotely polished enough.

I use Keybase. I like it, the thing to keep in mind though is 99.9% of the time the biggest threat vector to your company is going to be preventing your employees from sexually harassing each other, not preventing the Russians or whoever from reading your internal messages.

99.9% of the time the biggest threat vector to your company is going to be preventing your employees from sexually harassing each other, not preventing the Russians or whoever from reading your internal messages.

Yes, but:

1. We always have to combine the likelihood and incidence rate of threats with the magnitude of damage. I do not personally subscribe to this thinking, but some bloodless financial types will suggest that you can compensate harassment victims, but a major security incident could cost you the entire company.

2. Insert animated gif of a little girl asking, "Why not both?" Companies can and must address threats like sexual harassment AND exfiltration of sensitive data.

I’ve been running my 100% remote team on it the last few months, it works very well for us. Never been a fan of Slack, and never use any bots, so just comms.

I use keybase and promote it around the office, but I've had trouble getting people to join me. Maybe if it had an edgy dark mode...

Keybase has a Teams feature so it's built more towards your use-case. I use it to trade Stellar (XLM), file storage/management, and sharing/proving all of my platform identities. I haven't had a chance to get too social on it yet so I can't comment on the chatting experience, but overall it's a neat platform. It also has a command-line interface.

If you end up signing-up, follow me @ https://keybase.io/jbales

The post says "If the attackers inject server code, 2FA or U2F or any Web-based security practice does little." Understandably if the attacker was actively scraping passwords with 2FAs and was using the credentials immediately, that would be an issue.

However, for the issue mentioned in the post, wouldn't 2FA have saved him? Supposedly, the compromised credential is from 2015.

U2F saves you, most of the other options potentially make it worse.

If the bad guys are dumb and just were there to passively steal passwords then almost anything works. But if the bad guys are paying attention and stealing authentication secrets then it breaks down like this:

Passwords: Stolen if used while bad guys could watch

TOTP/MOTP/ similar software one time codes: Seed gets stolen by bad guys, they can generate the same codes as you now.

SecurID/ similar key fob type one time codes: Seed gets stolen by bad guys, they can generate the same codes as you now.

SMS: Now bad guys know your phone number. In a targeted attack they will "jack" the number and cause even more havoc, but you might be safe against script kiddies

U2F (or WebAuthn): The server doesn't know any secrets. It knows a cookie (meaningless gibberish except to one FIDO device) and a public key (public). Stealing these is futile and cannot be used to impersonate the user.

TOTP and SecurID surprise people because they forget that although the credential transmitted is just a one-time code, the _server_ (which bad guys broke into here) needs to know the same seed as the user does to get the same codes.

Not only does Keybase not automatically update its client, there is no way to even figure out if your client is out of date and in need of security updates. Even if you look up the exact version of your installed client, which you can find, there is nothing on the website that says what the most recent version is. The only way to even get a hint is to look on GitHub, and even that isn't accurate; version 4.2.1 is the latest release, but when I download the mac app it's still version 4.2.0.

For whatever problems Slack has, at least I know if there is a new version that I need to install.

You know what's better than installing slack? Not installing it and using it in the browser.

If there's a website option for any tool, I recommend using that over native. It's usually more performant and is less of a security risk. And it's always up to date.

> It's usually more performant and is less of a security risk.

Source, particularly for the "less of a security risk"? (It might well be now, I'm not an expert, but a few years ago I'd have thought "no way").

I expect he's referring to the web page running in a sandbox vs running an app locally with no sandbox.

What's the mystery here? It doesn't have access to your file system or the ability to request root access.

Which do you think is more potentially damaging?

1) Using a website which has had its server code compromised (slack).

2) Installing and using an application which has had its code compromised (maybe also slack).

The installed application is going to have more access and potential to damage your system and to compromise your data. There's not really anything more to it. One's in a browser sandbox and limited by browser capability, the other can do literally anything it wants.

Makes sense, thanks. I think my mental threat model was different:

you're talking about dodgy (potentially compromised) software: better run in the browser sandbox (albeit imperfect) than natively.

I was thinking of sensitive software (eg secret chats) I want to protect from attack: better run natively than in the (imperfect) browser sandbox.

In the context of this discussion (Slack), your threat model probably makes more sense.

Software running in a browser is probably going to heavily sandboxed and have tight restrictions on what it can access since the purpose of a browser is to run untrusted code. Native software will usually have minimal sandboxing because it's supposed to be trusted, and can do anything the user running it could do.

As far as performance, I think that only applies to apps like Slack and Spotify that use Electron to pretend to be a native app.

> You know what's better than installing slack? Not installing it and using it in the browser.

What's sometimes even better than that, is not using Slack at all.

That is why you use a system with a package manager. Maybe you could follow the releases feed on Github. It was what I do when I maintain a package on AUR.

Keybase developer here. Keybase does automatically update on Mac, and you can check if you're out of date on the CLI with `keybase update check`. Additionally, the "widget" popup from the system tray will display an out of date banner automatically.

4.2.1 was a bugfix release that only affected Linux and BSD, so we didn't push out Mac or Windows releases for it.

> Keybase developer here. Keybase does automatically update on Mac

Interesting, is this relatively recent? It didn't update for me, but I hadn't opened it in at least six months.

Also, that CLI command should probably be wrapped in a "Check for update" menu option.

>By contrast, Keybase currently runs all of its mission critical chat applications over Keybase itself.

Reminds me of the time Amazon learned that using S3 to host status indicators for S3 is a bad idea if S3 goes down.

> Were our Dutch friends sifting through our messages for four years before Slack notified us of a suspicious login?

The author might know this already but the hackers aren't Dutch. They simply bought a cheap NL server from LeaseWeb to mask their real IP.

> bought a cheap NL server

.. or exploited/hacked a random cheap server to use to proxy their requests and have access to high speed 100Mbit/1Gbit downloads/uploads of Slack data.

slack in 2015: "We have no indication that the hackers were able to decrypt stored passwords, as Slack uses a one-way encryption technique called hashing."

slack in 2019: "In 2015, (...) The attackers also inserted code that allowed them to capture plaintext passwords as they were entered by users at the time. "

This reflects poorly on them. Unless they discovered only now that the 2015 hack included capturing plaintext passwords.

Scary, and certainly doesn't reflect well on Slack. But, do keep in mind that the author runs a company that does compete with Slack in some ways.

I don't think that's relevant.

Poor security practices are poor security practices despite conflicts of interests, and Slack's are certainly extremely poor.

From the blog posts slack has released we know nothing about their security practices.

They have a lot of high quality security features and you can see they actually work because they alerted Max that his account was compromised.

Saying their security practices are extremely poor based on an incident they had in 2015 when their company was 1/20th the size it is today is ridiculous

It's relevant because as a security minded CEO...he did not even enable 2FA on his account.

Regardless of the issue, that still reflects very poorly on him.

I do not run a company that competes with Slack but still agree with the author.

Encryption for a business chat app limits the potential users rather significantly, as I understand it. A number of sectors (like banking) have strict rules which require keeping a record of company communications. How does Keybase deal with this, or do they choose not to play in that market?

Well, the bank could still record from their end of the conversation, capturing the chat logs from the employee's terminal and shipping them somewhere.

That's a nice sales pitch! A little on the nose, but I think that's ok.

I think one issue keybase still had is it's minimal web presence that sounds to focus too little on what keybase can do for regular users. People need more explaining of the day to day benefits.

I wonder if this affected companies like IBM... I know for a fact IBM uses slack to talk internally about client billing, meaning if IBM was compromised the attacker knows what most of IBM clients bought from IBM.

Same applies to a bunch of other companies...

> Unlike Slack, it is free. And without ads.

How does Keybase make money?

It's supported by the foundation that maintains Stellar, a popular cryptocurrency

Thank goodness. I was worried it was something unsustainable.

> Keybase messages are end-to-end encrypted, and only our users control their decryption keys.

End-to-end encryption isn't just good for the users because the service can't access the messages and sell them, it's also good for the users as it provides good protection if the service gets hacked: message content can't leak unless the attackers can change client code.

As people are discussing Keybase for teams and whatnot - could anyone comment on Keybase for individuals, families, etc?

My family are debating moving to Matrix (and away from iMessage). I had briefly debates Keybase due to some interesting features. Anyone have experience with Keybase for families and individuals?

Why move away from iMessage? It's end-to-end encrypted.

Either one of:

- iOS only

- an extremely-crypto/privacy-aware family who is concerned that although end-to-end encrypted, there is no out-of-band user-validated key exchange (e.g through a digit code or QR-Code) or prior validation (e.g per-key TOFU) of the other party's identity or update to their keys. Which means that theoretically Apple could insert a new key in the user's set (there is a key per user's device for perfect backward/forward secrecy†) anytime to MITM/intercept messages and you would not be able to monitor it††.

† This is obviously defeated by Messages in iCloud (which you can disable) but the iMessage protocol has this built in.

†† Of course if there was some feature like that, Apple could also make such a key hidden in some way because they also control the Messages app code.

Because I'm planning on moving to Linux, and I dislike typing a lot on my iPhone. So I need a desktop client to talk with my family/etc.

I'd prefer to keep iMessage, but you know, walled garden. Lol.

It's OK, notifications collapsing is somewhat wonky for some people and some of my friends complain about the app's performance vs something like Facebook but given the security model it will never be as fast.

disclaimer: I work for Keybase

Appreciate your response. After finding out how Keybase is funded, the model has me worried about relying on the future of a magically[1] funded platform compared to something I could self-host like Matrix.

Do you have any insight on why I shouldn't be concerned?

[1]: since I don't have clear funding motivations I'm referring to it as magical.

Have you considered Wire? What are your thoughts? Runs on many platforms, sign up with phone number or email, and of course e2ee.

The second I read this, I immediately signed up to Keybase today and deleted my Slack account and switched to the Keybase chat system instead which I am setting up right now.

I am confident to say that this incident was the final nail in the coffin for me to abandon Slack. Choosing Keybase was a no brainer.

From the slack post:

> In other words, if you’re one of the approximately 99% who joined Slack after March 2015 or changed your password since then, this announcement does not apply to you.

Hackers compromising plaintext password would seem to apply to everyone using Slack, whether their account was compromised or not??

I’m a little confused here. Isn’t Keybase a slack replacement product? Why is the CEO of Keybase using Slack for any company communication instead of his own service? Am I missing something obvious here?

It's described in the article:

> what we did use it for, which was discussing outages of our own product

You can't use Keybase to communicate about why Keybase is down :)

Slack gates a lot of its exfiltration security features behind enterprise licensing. Expect to double your spend if you want to try to prevent sensitive data floating around in slack.

Is there any indication why did this post drop from the front page?

Does Keybase support chat message search yet? Otherwise claiming to support all important features of Slack is stretching the truth. I use search extensively at work.

Yes, it does

Yeah, this is basically marketing material.

Does Keybase have any kind of revenue stream? I didn't see any mention of pricing on their website.

this needs to go up on HN!

So why was the Keybase CEO using Slack in the first place?

it seems to be a backup for when they break keybase

> We basically just use a #breaking channel in there in case we have Keybase downtime.

The problem wasn't they were using Foo and Foo was compromised.

The problem was that he couldn't be 100% sure whether Foo was compromised or not (when asked, they claimed they weren't).

Now, if Foo is a "random-cheezburger-meme-generator" startup you just try out with a throw-away password, then you can probably assume that it's likely they have been hacked and they are lying to you.

That leaves one alternative scenario: you have been hacked.

You'd expect that Slack (in 2019), while it's still possible they lie to you, has decent security practices and thus makes it less likely to be hacked and thus increasing the odds that you might indeed be hacked.

In a way, it acted as a honeypot. But one that you don't control so you can't be sure whether it caught a hacker or whether the bees are drunk.

Reply on news.ycombinator.com