HipChat security notice
blog.hipchat.com
blog.hipchat.com
I've seen companies posting root passwords, ssh keys, salaries, internal financial details, etc in Slack and HipChat. Just waiting for a disaster to strike, adding value for every additional company to the target. Maybe this breach won't be the last straw, but it's a consistent risk.
You can run your own MatterMost or XMPP server quite easily and even lock it down to behind VPN only to minimize security risks almost completely.
And if they're too lazy/dumb/entitled to download Adium/Pidgin and enter their email address+password; well, you should probably find better employees.
Your job as IT is to deliver business value. It's certainly possible that people not communicating is better for the business than people communicating over a hackable service, but it's not the conclusion most people have come to.
The same open source community (IgniteRealtime.org[1]) that maintains the Spark[2] XMPP client, also maintains OpenFire[3], a very good and easy to setup XMPP server.
[1] http://igniterealtime.org/
I think they probably just bought a license for a commercial fork of OpenFire.
That's very possible. Cisco bundled/bundles OpenFire into several of their enterprise appliances, including the Cisco Finesse product. Other companies do similar things. OpenFire is licensed under the Apache license.
There's also the possibility that your friend bought an enterprise license to OpenFire back when it was a commercial product under the name WildFire (Spark was commercial back then too). That would have been many, many years ago, back before Jive Software open sourced WildFire/OpenFire, Spark, Smack (XMPP Java Library), and several other pieces of software for real time communications.
It's not remotely comparable to re-developing the whole system. If your developers can maintain their own machines, they can probably handle this.
Security is just as much (if not more) responding to a breach effectively and quickly as it is preventing one.
You claim "risk," of using 3rd party services but can you quantify it with actual data?
Slack's entire business is secure business communication. Are we to think that our teams are better than Slack's when Slack's core competency is secure communication?
Should companies install their own phone lines because the 3rd party phone companies can't be trusted? Is there not risk when your internal teams who aren't necessarily domain experts, are building and maintaining systems that are outside of the company's core competency?
The security value of doing it yourself is nothing but anecdotal and not based on any actual data.
Not really, if I recall correctly Sony's whole Windows network was compromised via trojans in a PDF attachment exploit. Nothing to do with local vs cloud storage. They certainly couldn't have replaced their desktops with Dropbox.
> Are we to think that our teams are better than Slack's when Slack's core competency is secure communication?
Not necessarily - but they are much better able to restrict things to only your employees by applying VPNs and HTTPS+LDAP auth only proxies. Preventing you from being affected by public breaches like these. The value there is well documented - look what happened when the world moved from exposed to behind NAT routers.
Big public breaches happen all the time - you mentioned dropbox, I've gotten several reset emails from dropbox due to compromise, I'm guessing the attackers didn't walk away empty handed in such cases. A quick search reveals that just last year 68 million dropbox accounts were compromised.
On the other hand, when was the last time an even moderately well maintained SMB file server behind a LAN was compromised directly? Unique zero-day attacks are much more likely to be used on public services too due to the nature of their value.
What about Matrix?
"Hey, you know what might be a good idea? Let's email all of the accounts at the same time using an Appriver blast!"
Atlassian. I hate to hate you.
Only emailing a rolling amount of your customers becomes a shit show of support, who do you email first? Who do you email last? How long do you wait between groups? For who is security important, your biggest customers, highest paying, most security conscious? It's a real shit show to know, and one you'd absolutely get wrong, letting everyone know as fast as possible is the only acceptable solution to a security breach.
Truth to be told I can only assume this was done in a short burst, given my limited sample of (hopefully ever narrowing) circle of people who use Atlassian products. But would distributing the bulk-mail over an hour (two, three) using a randomized sample of their customer base really made a significant impact to security or their support?
I wonder how I'd do it, really, if let's say, beefing up my infrastructure for some reason isn't an option.
Warranties are not included. So it's a bit lame to blame "a popular third party library".
The OP was trying to say a company of Atlassians size should dedicate the resources to vet (and fix) those libraries if they use them for these purposes.
Many have always argued that it has.
>Do we say the same thing about any large company that uses openssl or any other open source libs that people use or depend on?
Many do.
>That doesn't seem fair or reasonable.
Many argue that any company that failing to contribute to the OSS projects they depend upon isn't fair or reasonable.
Alright, I'm arguing that it doesn't.
> Many do.
shrug I don't.
> Many argue that any company that failing to contribute to the OSS projects they depend upon isn't fair or reasonable.
This sort of attitude bothers me. At this point the software is not really free in my opinion. I am not a lawyer :P Just my $0.02
Quality software doesn't create itself out of thin air (yet, if ever). That means someone has to make an investment.
You don't have to invest in the upkeep of the foundation of your house, but if some bugs, say termites, were to sneak in you can't blame the original builders for the donated foundation.
Please downvote for the bad analogy.
I worked for a company that needed to push out an out-of-cycle patch for Heartbleed. We were building a virtualization product that included a OpenSSL and other free-software libraries in the core product, plus an entire Linux distro to support our install-this-on-dedicated-hardware product. We made the business decision that we could reuse Ubuntu and not develop our own operating system and control plane. Others, like Microsoft, made the decision to implement it all themselves. Others, like VMware, took a decision sort of in the middle.
We got a significant amount of functionality for free - and a significant amount of risk for free. Whatever code worked for our needs, we could profit from. Whatever code introduced security vulnerabilities in our application (and it was not all upstream security vulnerabilities, since we intentionally designed our system to anticipate that local root exploits would be easy), we took responsibility for. That was part of saying that this was our product, and not just a shell script for building a similar product on your own.
Yes, and it always has and it can't be discharged. Pay-it-forward is the right thing to do.
> Do we say the same thing about any large company that uses openssl or any other open source libs that people use or depend on?
I certainly do. A red line, I-will-quit condition is and always has been "I won't participate in the development of private forks of open-source software" and I have at multiple employers gotten checks straight-up cut to open-source software maintainers. I have also entreated (and in two cases succeeded in convincing) maintainers to start up maintenance programs so we could pay them a yearly fee--because donations are way harder to push than support plans.
And, in turn, I open-source useful tools[1][2][3], including major parts of my consulting business, because it, too, is the right thing to do.
You should do likewise, because it is the decent and human thing to do.
> That doesn't seem fair or reasonable.
I consider not paying forward kindnesses paid to you way, way more unfair and unreasonable.
[1] - https://github.com/bossmodecg
Interesting - it may be a nice thing to do, but I don't agree that there is any sort of obligation to the project just for using the project.
> I certainly do. A red line, I-will-quit condition is and always has been "I won't participate in the development of private forks of open-source software" and I have at multiple employers gotten checks straight-up cut to open-source software maintainers. I have also entreated (and in two cases succeeded in convincing) maintainers to start up maintenance programs so we could pay them a yearly fee--because donations are way harder to push than support plans.
I have also worked at companies that funded open source projects through donations or maintenance, but there was never a moral obligation there. It was more a method of risk management than altruism.
> I consider not paying forward kindnesses paid to you way, way more unfair and unreasonable.
Paying forward kindness and being morally obligated to maintain an open source library just because you use it are different things in my mind. It seems like being paid a kindness creates an obligation, which is not that nice.
---
Interesting perspective even though I strongly disagree. I'll continue to use open source projects 'AS IS'[0] and I still wont feel morally obligated to maintain them. Similar to how I use linux/BSD and don't feel obligated to maintain the kernel. I'm certainly grateful of course, but I don't feel like there was any sort of contract/exchange between myself and the maintainers that creates an obligation on my part.
[0] https://github.com/eropple/auster/blob/master/LICENSE.txt
There's really no way around this; it's a straightforward practical problem.
If you use someone else's code, especially if you're not paying anything for it, you get what you put into it: nothing.
The liability for this breach is ultimately owned by Atlassian, not the third party library writer. To quote the most permissive license out there:
"THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED."
Obviously that is Google's problem, but I haven't seen Google (nor Atlassian in this case) claim anyone else is to blame.
"We have one rule in the kernel: don't break userspace. Everything else is kind of a guideline. The whole security thing? It's a guideline that we shouldn't do stupid shit. But people do stupid shit all the time and I don't get that upset." https://www.youtube.com/watch?v=1Mg5_gxNXTo#t=8m28
Imagine, Torvalds said, that terrorists exploited a flaw in the Linux kernel to cause a meltdown at a nuclear power plant, killing millions of people. “There is no way in hell the problem there is the kernel,” Torvalds said. “If you run a nuclear power plant that can kill millions of people, you don’t connect it to the Internet.” Or if you do, he continued, you build robust defenses such as firewalls and other protections beyond the operating system so that a bug in the Linux kernel is not enough to create a catastrophe. http://www.washingtonpost.com/sf/business/2015/11/05/net-of-...
And the Linux kernel has a track record for not being the world's most secure piece of software. Which is fine, it's a project he started for fun.
It's certainly possible to pay people for a kernel that they'll stand behind commercially. If you're using, say, a RHEL kernel and a bug or vulnerability is discovered that impacts you, by all means get upset at Red Hat. If you're using a Fedora kernel, though, you made the choice to use it. It's completely unfair for you to get the benefits of running a kernel you put neither time nor money into, and not also the risk of running that kernel.
The problem is absolutely owned by Atlassian, but they actually did do something to fix it.
I don't believe anyone (apart from maybe Daniel J Bernstein) can claim any piece of software is bug/hole free, and neither does anyone need to!
Why, when I use someone else's software without so much as telling them, let alone signing a support contract with them, does the blame shift to them? My responsibility is to deliver a service, not to write software. If writing software is the easiest way to do it, great. If using someone else's software is the easiest way to do it, great. But I'm the one running the service, either way.
I don't know the details of the vulnerability - I would say they were at fault if they did not update/patch a fixed vulnerability.
I can't speak to this case, obviously.
If I use the library, and it is non-essential for my business, then I should know what it is so that I can remove it.
Well, I didn't receive anything, but couldn't log in either. I had to google this to find out what the hell is going on - error messages on the login page are not helpful either, they just refuse login even after resetting the password.
Fine, but I wonder why they didn't reset the API tokens while resetting password immediately. Are they managed by the different servers/services?"Security Intelligence Team" ... yeah, because that team actually exists.
You might find the below numbers interesting. Note that this performance is only one workstation with 8x gtx980. Even the mighty bcrypt (sidebar, look at the sha512 #s) won't save you if your password is bad. Now consider social media mining to enhance the word list. Now consider that (anecdotally) I have never done a hashcat audit and not had to have a conversation with someone about choosing better passwords:
Hashtype: bcrypt, Blowfish(OpenBSD) Workload: 32 loops, 2 accel
Speed.GPU.#1.: 6398 H/s Speed.GPU.#2.: 6507 H/s Speed.GPU.#3.: 6513 H/s Speed.GPU.#4.: 6643 H/s Speed.GPU.#5.: 6534 H/s Speed.GPU.#6.: 6512 H/s Speed.GPU.#7.: 6689 H/s Speed.GPU.#8.: 6542 H/s Speed.GPU.#*.: 52338 H/s
"HipChat hashes passwords using bcrypt with a random salt."
This is a good example of how not to do mass e-mails targeting the general population.
I'm happy they say this.. now i'd like some sort of proof :-)
Poor UX is my point of this
blah blah...We take your privacy very seriously...blah blah...
[1] Technically, we use bcrypt with a random salt.
Security is hard. Describing security is even harder.
I'm for transparency in security for review, but you can achieve the same via a 3rd party audit.
Just my 2c.
If they'd explained both ways, somebody would accuse them of the notification being too long as part of a scheme to bury the details below the fold.
As a company reporting an issue, no matter what you do, the internet's gonna nit pick