Heroku: Resolved Security Vulnerability | Hacker News Reader

Heroku: Resolved Security Vulnerability

pastebin.com

49 pointsbjonathan25 comments

Summary of the vuln, from the original post: Heroku was relying on unguessable filenames to keep processes associated from one app from reading files containing the config files and source code of another app. However, the random portion of those filenames was showing up in the output of 'ps' as run from (I think) 'heroku console'.

The upshot was that the filenames were easily guessable, and their contents (config files and source code of the app) easily accessible.

So, if you've had a heroku app running between December 28 and January 17, your config files are possibly compromised.

(The scary part, to me, is what a malicious party could have done with access to somebody else's database.yml, but I haven't seen any writeup specifically address that point.)

The filenames weren't so much easily guessable as a change Heroku made last month accidentally exposed files. Relying on unpredictable strings isn't a fundamentally bad architecture; it's not an exaggeration to say that the whole web depends on that design.

My recommendation to Heroku might start with auditing every Heroku tool to make sure it calls "setproctitle".

I meant to say that the filenames were easily guessable after the random strings were exposed by 'ps'.

Relying on unpredictable strings works fine, so long as you keep them away from prying eyes. But that requires attention to detail, and can get harder over time (witness, say, Firesheep, which made it a whole lot easier to hide "secret" login cookies than it used to be).

Can I just say that "secrets" sent over bare HTTP aren't secret, and that even before Firesheep, nobody really thought they were hard to find? I love what Firesheep did to popularize that problem, but it was an old, very widely-exploited problem.

Sure, but Firesheep made it possible to exploit it without even "script-kiddie" level skills. No easier in principle, perhaps, but easier in practice.

The point is the type of security scenario being talked about here is wildly different than the one made simpler to exploit by Firesheep. This was not security through obscurity. If your servers are configured correctly, unguessible filenames are just fine from a security perspective, at least as secure as passwords.

The discoverer of the vulnerability also wrote a blog post about it.

Blog post: http://daverecycles.com/post/2858880862/heroku-hacked-dissec...

HN Discussion: http://news.ycombinator.com/item?id=2128175

Disclaimer: I know David E. Chen personally from college.

Official Status Blog Post: http://status.heroku.com/incident/115

Seems like something like this could have been easily prevented early on using bsd jails or linux vservers for proper sandboxes.

What I really don't understand - why aren't they using virtualization? (e.g. OpenVZ)

Because it's very slow compared to application-layer segregation.

There are different kinds of virtualization. OS-level virtualization like Linux-VServer and FreeBSD jails have little to no overhead while being very secure.

Jails aren't virtualization. I don't know why they don't use jails, but the answer on "virtualization" seems straightforward.

Yes, I was surprised too. I've never used the word "virtualization" when refering to jails and VServer, but Wikipedia apparently refers to it as "Operating system-level virtualization".

http://en.wikipedia.org/wiki/OS-level_virtualization

Remember this?

http://news.ycombinator.com/item?id=362082

Do you still think that's a good idea?

I don't understand what point you're trying to make (also, how'd you come up with THIS particular comment, from 800 days ago?).

It was a serious question, I didn't mean to sound snarky.

I'm asking if you think it'd be a better idea (from a security perspective) to keep customers on separate VMs. My understanding of this vulnerability is that it relies on several customers sharing a filesystem; if Heroku simply spawned a new instance (they're already running on EC2) for each customer, wouldn't this be mitigated?

Also, I was looking up info on Pivotal Tracker yesterday (after the announcement that they were going paid) and saw your comment.

It really depends. Like most pentesters, I'm looking at two factors when I consider a stack like this: first, how big/complex is the exposed attack surface of the stack, and second, how well tested is that attack surface.

For many applications it seems entirely possible to do better with a single multi-tenant app stack than you could with OS virtualization with the same service model. In particular, if, to get the same service model, you had to customize your OS virtualization stack, I'd tack sharply towards preferring a multi-tenant app stack.

I can easily get my head around auditing the Ruby interpreter and its deployment on a Unix host. I've done projects assessing custom virtualization solutions and they are gigantic and complex to test. I'm pretty sure interrupt timing isn't going to screw up Heroku, but that's a flavor of the kind of stuff you have to test with full virtualization.

Finally, I have to add: everything is going to have bugs. You can fully virtualize every customer and every app and you will still eventually be exposed to something bad. The measure of the team is how they handle the exposure, how they fix it, and what they learn from the experience to make the next one harder to find.

"if Heroku simply spawned a new instance (they're already running on EC2) for each customer, wouldn't this be mitigated?"

Yes, but that'd cost a ton more than what they're doing right now. Consider the number of apps that they are hosting at the free level right now. Offering private instances as an upsell might make some folks happy though.

I don't agree with your "very slow" sentiment -- can you provide a source that shows that linux-vservers are very slow compared to running on bare metal? To my knowledge, vservers have almost no overhead vs running directly on the machine.

Heroku needs to be able to spin up and spin down server processes ("dynos") very quickly. Their process right now, so far as anyone can tell, is basically to untar a "slug" and start a ruby process. Booting a VM would be a lot more cumbersome (and there might be technical issues doing it on top of EC2 instances which are already virtualized).

If you're using a fairly recent kernel, 2.6.26 supports linux containers natively. You can "boot" a VM in less than 200ms, then start whatever process you need to start. I think when people think of virtualized instances, they think that the VM boots all kinds of other unnecessary processes.

Given that one could compile custom kernels on EC2, I think it's quite possible to run LXC on top of amazon instances. When I run "top" on my vserver running on top of vmware fusion, running on my laptop, I really only see my sandboxed process and nothing more.

Also, I see that heroku keeps these "disks" in a squashfs filesystem file. It seems like you might gain better performance if you just had all the files on your filesystem, used vservers, then use something like vserver hashify to prevent file duplication (to reduce space usage).

Hmm, I thought Heroku used chroot jails to prevent apps from seeing each other. No?

Holy crap i have a pile of passwords to change now.

not cool, but figures.

Reply on news.ycombinator.com