Zendesk was hacked
zendesk.com
zendesk.com
For the last 2.5 years, we've used a popular service called Zendesk to store, organize, and answer emails to Tumblr Support. We've learned that a security breach at Zendesk has affected Tumblr and two other companies. We are sending this notification to all email addresses that we believe may have been affected by this breach.
This has potentially exposed records of subject lines and, in some cases, email addresses of messages sent to Tumblr Support. While much of this information is innocuous, please take some time today to consider the following:
The subject lines of your emails to Tumblr Support may have included the address of your blog which could potentially allow your blog to be unwillingly associated with your email address. Any other information included in the subject lines of emails you’ve sent to Tumblr Support may be exposed. We recommend you review any correspondence you've addressed to support@tumblr.com, abuse@tumblr.com, dmca@tumblr.com, legal@tumblr.com, enquiries@tumblr.com, or lawenforcement@tumblr.com. Tumblr will never ask you for your password by email. Emails are easy to fake, and you should be suspicious of unexpected emails you receive.
Your safety is our highest priority. We're working with law enforcement and Zendesk to better understand this attack. Please monitor your email and Tumblr accounts for suspicious behavior, and notify us immediately if you have any concerns.
Second - and any incident response team will tell you this - patching and removing the backdoor is not enough. You have to wipe that machine.
It's not uncommon for an attacker to leave multiple backdoors. Even if you don't think they got root, you have to wipe it completely.
Modern servers have firmware running on internal devices, including Linux runnning on full ARM cores, that can be 1) imaged from the base OS, and 2) have full control over hardware or other important data.
This includes:
- IPMI Modules: http://fish2.com/ipmi/itrain.html
- HDD Controllers (in your HDD/SSD drives): http://www.arm.com/markets/enterprise/hdd-ssd.php
... as well as ethernet controllers, any devices with PCI option ROMs (they can control boot), and quite a few other things.
This essentially means that you have a bunch of fixed, poorly documented, runtime reconfigurable and re-imageable hardware, with full access to data and system resources.
Typically when a breach-in happens you do two things:
1. backup the entire disks for forensics purposes
2. wipe everything clean, re-install the latest version of the OS + all the latest patches, re-install the latest version of the apps.
Nowhere did parent suggest to "destroy evidence".
Ok, so striking out so far. The machine is still running? With the same software (patched) and user accounts? How do you know only 3 users were exposed?
I'm hoping for automated tools, services to install on our servers, or security auditors who have an out of the box package.
This is the only automated tool I know about: http://brakemanscanner.org/
We don't do code analysis, but we think it's a useful service.
They are in beta.
But since the question was one of a live monitor that detects intrusion, I've never heard of such a thing. There's always the possibility of aliasing `mysqldump' or `pg_dump' to another command that emails your admins, or other manual commands that shouldn't be run throughout the course of the day. My personal boxes run such an email script anytime someone logs in as root, and emails the logfile anytime someone uses sudo. That won't help against SQLi, but might against RCE that's allowed someone to tunnel into your box.
But, in the long run, there's nothing that won't beat subscribing the the security lists of all the software you run to get immediate notice of any vulnerabilities, hiring a pen tester, and stopping every day to read the code you've written to discover what kind of edge cases might help an attacker compromise your system.
http://www.wired.com/threatlevel/2013/02/twitter-tumblr-pint...
Excellent question. The answer really is, if you are running a "web app" with any kind of sensitive information, you need to have a security expert configure and administer your systems, or become one yourself, and you need to stay on top of every update.
Now, there are basic things you can do that will eliminate most of the casual script kiddie attackers. Firewalls. Defense in depth. Keeping up with patches. Google or any good book on computer security will provide this information. But the sad truth is, if a smart, determined attacker has you in his sights, you will most likely lose. And even the automated tools are getting better and better.
I think we are starting to see a tipping point in the web. More and more high profile sites are getting compromised. We have learned that China has been inside US government, utility, and industrial systems for years. Right now I would not trust any sensitive, personal information to any website or cloud service. I think we are going to see some major, consequential attacks on government, banks, and other commercial entities in the coming years. At some point I believe we are going to need to rethink the cost/benefit equation of having everything connected to everything.
A friend of mine put together an open source project that uses cryptography to store your data in the cloud securely, so it's certainly possible. [1]
It's also possible to write complex software and not be vulnerable, though 99.999% of the time companies (start-ups and otherwise) seem more concerned with an MVP and new features than security. If you design a system from the ground up with security as a core feature, then you have a CHANCE of having a system that won't be vulnerable to script kiddies every other week. On top of that you need to be sure to protect against social engineering, but that's another discussion.
I don't even know if it's possible to use something like Rails (or Ruby, even) and be secure for the long term without having to deal with constant updates and patches. On the other hand, I HAVE used complex systems that were designed from the ground up to be secure and that simply NEVER turned out to have a security vulnerability after the first few releases. (Anything by DJB, for example. [2] Some of those tools have gone 15+ years with no vulnerabilities. Compare the constant sendmail or bind security exploits, numbering in the hundreds at this point, to DJB's qmail and djbdns.)
Until it's a priority, it's always going to be an afterthought, by definition. People will use Rails or the framework du jour, despite the fact that such frameworks are designed with the same "get it done and release ASAP" philosophy that most commercial sites are developed with, and then everyone wonders at security holes. Sigh.
[1] https://tahoe-lafs.org/trac/tahoe-lafs
[2] http://cr.yp.to/
After just apt-get'ing that, the system is of course always vulnerable to the next as-of-yet-undisclosed 0day vulnerability in the base software.
As soon as you start adding useful scripts of the "P" variety (as in "LAMP") then you are at the mercy of those scripts not having any vulnerabilities.
If you wrote those scripts yourself, then yes, xss, sqli, rfi etc are all issues you need to consider. The OWASP pages could be a useful place to start getting a feeling of the most common pitfalls.
So, how to keep safe:
1. The app itself:
If you have a small web app, you have an incredible advantage that makes your web app potentially way more secure than that of a big corporation: You write the code and you know the inside outs of the system. And you are probably the only one having passwords etc. to the system.
In a big corp, sometimes the interns write some code and they have no clue of web app security. From an attacker's perspective, one security hole is enough.
So, if you know about web app security, you're probably better off than any other big corp.
2. The server infrastructure:
If you worry about the server architecture, get a managed dedicated server. There's probably not really a guarantee that this system is set up perfectly, but there might be trustable hosters that know what they're doing.
- Keep everything up-to-date (apt-get update, apt-get upgrade). - Don't install software you don't know or can't trust - use SSH only with public key authentication, remap port 22 to something else - use SFTP + SSL for your website - don't include any third party JS-software that messes with your website. Personally, I don't even include Google Analytics. I self-host everything and install only stuff that is trustable. - don't use the root user if not necessary
There's this "myth" going on that every system is hackable. This is only partly true. You can't do anything against 0-day hacks, but if you protected your web app against XSS, SQLI etc. and your software is uptodate, there's hardly any chance that you get hacked.
What makes every system "hackable" is the human factor. The secretary that gives out her password to some guy on the telephone who pretends to be the technical administrator of server X. Are you likely to get social engineered? Probably not.
Also, it is a big deal because (as a former support person I know this), users often send in sensitive info with their support requests: SSNs, full credit card info with CVV data, date of birth (yes, sometimes all in the same message).
And two, the body of the emails was not exposed, only the subject line. People typically don't but their SSN in the subject line...
Obviously you can also take the opinion that this should have never happened and question their competence and security. I personally weigh their response and transparency more than the issue itself, but it may seem easier since the impact to overall customers was relatively small.
Their response seems to have been handled well and may even generate some positive PR. That may change if it turns out to have been one of the recent Rails security flaws.
Well done Zendesk.
For the record, when I wrote the above comment, the headline of this thread was "Zendesk was hacked, 3 customers affected". It has been changed since, without due notice of course. I wonder why...
Now, my comment looks like I'm a nutjob and have a personal gripe with Zendesk or something.
That the three customers were Twitter, Pinterest and Tumblr didn't come out until later as well.
Carry on.
Leaking my personal info is one thing, loosing customers personal info is simply far too boneheaded to begin with.
Now a serious question: What does this say for storing everything, including tax filings, in the Cloud?
Here in Finland tax information is public and it doesn't seem to be that bad. As you guess, yellow papers embarrasingly make yearly rankings of rich people and they lose some anonymity. But anybody that was in any way interested in their life likely knew that they were rich.
I don't say that this Finnish practice of having public tax information is a good one, but it just shows that it's not the end of world.
Leaked Tumblr support information that allows associating Tumblr blogs to email addresses can cause a similar level of agony that leaked tax information.
I'm assuming that the original poster is from U.S. One reason why tax filling are considered sensitive is that they include the individual(s) social security number (SSN). This number is meant to be private and, since it's often used as an identifier, it's frequently used for identity theft. See the wikipedia article below.
http://en.wikipedia.org/wiki/Social_Security_number#Identity...
On balance, I think its actually a positive thing. Thinking about it and the few occasions I've seen it referred to, I think its a positive thing.
It's very convenient before you start contracting for someone to see if they look like a legit business and to see if they do any kind of volume (and are in the business since a few years).