Use Rails? Check yourself for the YAML exploit | Hacker News Reader

Use Rails? Check yourself for the YAML exploit

tinfoilsecurity.com

122 pointsborski56 comments

It might be worth mentioning "We determine this by sending a harmless request to your web server. The request doesn't do anything -- it is designed to be rejected with an error, much as if someone typed a URL incorrectly and requested a web page which doesn't exist. If your application is vulnerable, it will respond with a particular error code, whereas apps that are not vulnerable will not. [bold]This vulnerability allows attackers to execute code on your server, but we don't actually do this.[/bold]"

I mention this because I have lately had a lot of experience dealing with Rails devs whose attitudes with regards to security are akin to "If you don't do voodoo then voodoo can't hurt you." We should take special care when communicating with them to a) emphasize that we're not the bad guys and b) explain, as many times as required, that ignoring the bad guys does not make the bad guys go away.

Mind if we copy that verbatim? :)

Go for it.

You should mention that this will give false negatives for people who actually are vulnerable but happen to be running Ruby 1.8.7, by the way. (I'm distilling that from this part of the announcement blog post, plus the knowledge that Syck is the default 1.8.X parser. It is indeed difficult to exploit -- I banged my head into a wall for 12 hours attempting to do so -- but people who are highly credible to me on this issue report having successful exploits for it.)

Also, this check is for websites using the Psych YAML Engine and not the older Syck. All of the proof of concepts we've seen so far are for Psych. That doesn't mean Syck isn't vulnerable, but that our checker will only work for Psych. In all likelihood, Syck is vulnerable too and you should upgrade your Rails all the same.

Thanks. Yeah, we have some friends who have some working sploits for it now too. We'll work that in.

The exploit gist was actually pretty easy to modify in order to correctly identify if a Rails 2.3.x / Ruby 1.8.7 app was vulnerable. If you accept the fact that "payload being interpreted as object deserialized from YAML" is bad, then you don't necessarily need a malicious object to see if your app is vulnerable. If you have access to your Rails logs, then you just need to see a params value in your logs that's an object.

Here is my modified gist that can help you identify if your app is vulnerable, which works with Rails 2.3.x / Ruby 1.8.7 apps. If you run this and see in your Rails log something like "params = {#<ActionController... => {default => :action}...}", then your app is vulnerable: https://gist.github.com/4694948/26d9b6d46c2c60aeb5abde9ca2c2...

We also have a couple apps we know of that are really old Rails 2.2.x apps. This doesn't work with them, because the passed YAML doesn't work in the old YAML parser. But then, if you just trust that deserializing an object from YAML in parameters is bad, then we can just make a super simple hash object and check our rails logs to see if the params contain an actual hash object. Here's an even-more-simplified-harmless proof-of-concept to test older Rails 2.2.x apps: https://gist.github.com/4694948/1e8a94f7c768bf95863288005039...

I'm no security expert, but I was able to use these PoCs last night to verify that the patch for Rails 2.3.x worked, and also that my reverse-engineered-patch for Rails 2.2.x worked as well.

Edit: Just to clarify and reiterate, the above modified PoCs require access to the Rails app logs to check if they worked or not, since the older Rails apps could error out for YAML-parsing reasons (which are just as bad, because it just means an attacker needs to get more creative with their YAML payload), instead of being a result of rejecting the attack. That's why I don't think a web-based attack mechanism like this could really know 100% whether or not an app is vulnerable unless it really tried doing something semi-malicious, like obtaining secret information.

You're completely right. We call out that it isn't 100% certain if we return a 'not vulnerable'. Our full scan will soon include some code execution modules for these vulnerabilities which will provide a more certain response.

After the fantastic post by patio11, I spent all afternoon emailing every person in my network. Letting them know of the huge security risk. The answer I got from most people? "We have backups, and plan to deal with it soon. Thanks for the heads up." Really? This is the kind of stuff that made me delete every simple rails project I had on my workstations, and just push them to git. No sense in risking it.

Oh, and best of all was the response from my webhost (which is well known here):

We've added updated Rails installers to our control panel so that new Rails apps will be created with the latest secure versions. However, it's our customers' own responsibility to upgrade their existing Rails apps.

Note that customer Rails apps run as unprivileged users, with resources managed by the kernel cgroups feature, so its very unlikely that you'd see any issues on your own sites if another customer's app was compromised.

Hmm, very unlikely but still possible. Not something I want to hear from the people who Im trusting with my projects. I'm canceling my account tomorrow and moving away from them. I know they are not responsible for their client stuff, but this is a real threat to their servers.

FWIW, some hosts were more proactive --- Heroku in particular sent out email to the owners of vulnerable apps within a day or two of the announcement.

EngineYard did this too, FWIW.

In fairness, if other users apps are able to hurt yours then that is not very good security anyway.

Their only other option would be to push the update to everyone and risk it breaking things.

>Its very unlikely that you'd see any issues on your own sites if another customer's app was compromised.

>Hmm, very unlikely but still possible.

I think this is a fair statement as far as I can tell: If it were possible for a compromised customer's app to affect your app, then you're not safe anyway without this exploit: a malicious customer could affect you too just as easily.

anyone have an open source command line tool to do this? call me paranoid (and probably dumb) but I really dont want to enter this into another website

I can tell you we're trustworthy, but I hear where you're coming from. :)

With that said, you can use Metasploit for this too - we just wanted to make a really easy checker for those that wanted a quick OK/NO GO and didn't want to deal with setting Metasploit up.

Metasploit docs here: https://community.rapid7.com/community/metasploit/blog/2013/...

You are providing a valuable service, but is Metasploit really necessary for this? I think this is a simpler command:

'grep rails Gemfile.lock'

If you don't see a rails version of 3.2.1, 3.1.10, 3.0.19, or 2.3.15 then you are vulnerable. Yes?

I guess in theory an attacker could have compromised your server and changed the rails version number...

There are other workarounds, too. You could disable XML parameter parsing, for example (as seen here: https://groups.google.com/forum/?fromgroups=#!topic/rubyonra...).

Thus, you might be running an old version, but still actually be safe by disabling the vulnerable bits.

We use old rails, but not the affected activerecord components. Also, there are more yaml bugs in existence than the ones publicly talked about, and this would detect those issues as well.

    3.2.1 is not safe (you probably meant 3.2.11)
    3.0.19 is not safe (3.0.20 was released to fix CVE-2013-0333)
    2.3.15 is not safe (2.3.16 was released to fix CVE-2013-0333)

Good point. I just copied the version numbers from the linked website - I guess it was outdated. Although I checked again and they are correct now.

A smart attacker would install a backdoor and update rails.

Rails 2.x apps aren't necessarily using Bundler at all, and some folks may have disabled the vulnerable parsers, or installed the security patches only into an earlier version to avoid other breaking changes or new deprecations. (Rails-core has gotten better at avoiding this kind of collateral damage from upgrades, but some folks are still gun-shy.)

Actually probing the running app really is the only way to be sure.

Your scanner says 3 issues on my domain, but rails_xml_yaml_scanner instantly says none. Is your scanner scanning for more than the yaml thing? I don't really have access to just modify a production system on the fly to go through your verification process, and I don't want to go through all the trouble if it's just some nessus nag. It's confusing because it says three issues, but low impact. I thought I was only scanning for a yaml flaw, and yaml flaws typically lead to remote code exec.

edit : I somehow stumbled into the full scanner on the main site rather than using the yaml scanner, my bad.

Sorry about the confusion.

If you run a scan from our homepage, you're actually looking for a lot more than just the YAML vulnerability (XSS, Mixed Resource, etc.) as our product isn't limited to just the YAML vulnerability.

If you run the scan from https://www.tinfoilsecurity.com/railscheck, then you'll get a quick check for just the YAML vulnerability.

Does that clarify it a bit?

Ah, not sure how I got turned around, but yes I was using the scanner from the main page. Thanks for the clarification, and nice work. This is going to help out a lot of people.

If you don't do it, someone else will. And likely with less pure intentions.

While it won't cover all situations, Brakeman has been updated to warn about arbitrary YAML loads:

http://brakemanscanner.org/

[edit] This is a code analyzer.

Does this work, at all? I get nothing but "There was an error - please double check the URL and try again". There is no-one in the support chat provided. As far as I can tell it simply does not work.

And, to head off the inevitable comments, yes, it's free, I know: I'm just letting people know not to waste their time with something of zero value.

Sorry about that! Can you email support@tinfoilsecurity.com and I'll take a look? It's 11PM here and we've taken a few hours to rest. :)

I think this page is hoax ... just an attempt to gather a list of rails urls. looking at my logs I see no such request.

I think it can't deal with the traffic it's getting. I saw a couple of hits to my server and the page's progress bar moved a little, but it's been stalled for quite a while now.

Are you referring to the scan from our homepage or the Rails checker? Either way, could you get in touch with us at support@tinfoilsecurity.com or http://www.tinfoilsecurity.com/chat ? I'd love to see what went wrong.

We shouldn't be having issues with load.

It's not a hoax - could you get in touch with us at support@tinfoilsecurity.com or http://www.tinfoilsecurity.com/chat ? I'd love to see what went wrong.

I just did it and got this in my logs:

184.73.36.14/ec2-184-73-36-14.compute-1.amazonaws.com - - [02/Feb/2013:06:12:19 +0000] "GET / HTTP/1.1" 200 11310 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-us) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.3 Safari/5 Powered by Spider-Pig by tinfoilsecurity.com"

They're definitely hitting the URLs, I just think they have a backlog.

I just tried this on one of my side project sites and it's reporting it as being vulnerable. However I upgraded the site to Rails 3.2.1 last week (and just confirmed that's the version in the bundle). Is there something that would cause a false positive or is my app really still vulnerable?

There is the potential for false positives, but I'd be happy to chat about it - feel free to get in touch either via email at support@tinfoilsecurity.com or in our support chat: http://www.tinfoilsecurity.com/chat

ITYM 3.2.11? 3.2.1 is definitely vulnerable... We had a typo in ours that said 3.2.1 was safe - so sorry about that! Fixing that now. You should upgrade to 3.2.11.

Aha, well apparently I never merged that commit into my deploy branch, so it was still actually on 3.2.1. That combined with the fact that your page had said 3.2.1 was safe caused my brain to short circuit and not realize that 3.2.11 is what it needed to be on. :) I'll redeploy and check it again, thanks guys!

Yeah, that was totally our bad. I'm deploying the typo fix now.

Let me know if you have any other issues! Happy to help.

See, in a very circular way I was just reporting the typo. :)

(Looks good now that I have the right release deployed, by the way.)

Ha, fair point. :)

Glad you got it fixed!

We also are seeing a small group of apps with vulnerable applications even after upgrading to Rails 3.2.11, possibly due to a rogue middleware or other library. Disabling XML parsing entirely is one approach (see http://news.ycombinator.com/item?id=5035389) but we'd love to track it down further for everyone's good. Feel free to join us at https://www.tinfoilsecurity.com/chat if you'd like.

Rails 3.2.1 or Rails 3.2.11. It has to be Rails 3.2.11 to get the patch

'bensedat seems to be hellbanned for some reason (Can a mod unhellban? He's definitely not a troll.) Anyway, echoing his comment:

Out of technical curiosity, would it be reasonable to remove YAML from the source tree and check for the lack of exploit via "find $APP_ROOT -name "*.rb" | xargs grep YAML"?

You cannot conveniently remove YAML from Rails. It is used in dozens of places in the framework, including some code paths which your app is virtually guaranteed to exercise.

This doesn't seem to work. It returns the OK for apps I know I have not patched (simple stuff I've thrown up on heroku).

It's possible heroku is filtering the headers...

We know it's not supposed to be 100% reliable, but we haven't seen any false positives so far, actually; only false negatives.

Are you using Syck instead of Psych? If so, that would cause it to return the OK instead, since our check only works for Psych.

So what is the exploit, and how do we fix it?

I've posted a little bit more on our blog at: http://blog.tinfoilsecurity.com/use-rails-check-yourself-for...

There have been multiple vulnerabilities discovered with semi-related causes. These all have (several) exploits which are widely available. The upshot is that they permit remote code execution (i.e. "root the server") on virtually all Rails applications accessible on the public Internet and, perhaps, some which are not.

Rails has been updated to close these particular vulnerabilities. The proper versions are listed in these two advisories. There are also work-arounds if your app is not in a state to just upgrade Rails. If your app is in that state, I strongly advise you that your only priority for the next several days is fixing that, because one should have high confidence that there will be more vulnerabilities announced soonish.

https://groups.google.com/forum/?fromgroups=#!topic/rubyonra...

This is about the equivalent of putting a single quote in a GET variable to detect sql injection. It tries to detect any bugs in the bug class, rather than a specific bug. The solution is different depending on what component is broken.

Single location to aggregate a list of all vulnerable apps.

See my response here: http://news.ycombinator.com/item?id=5153615

You are more than welcome to use Metasploit - a lot of our customers didn't want to go through the trouble of setting up their own Metasploit config, etc., so we built this to give them a simple quick check.

Why bother with something this elaborate? I ran a Metasploit scan on the /24 network that my Linode is located in and turned up 4 vulnerable servers in a matter of minutes. If someone were hunting for vulnerable hosts, this form would be a really inefficient way of finding them when you consider how easy this is to discover using simple network scans.

Rails Security is apparently "omakase" - meaning 'leave it to someone else', apparently.

Do you pose any interesting facts or thought to these discussions? I've seen you post in nearly every single one of these threads with nothing constructive. I understand that "you used to use Rails" and are much happier now that you're leaving the "insecure framework" but do you need to post it in every single thread?

Disclaimer: I'm a java programmer.

Not true, and I'm not a fan boy either. Did you read the security section of the guide? The edge one is more up to date (not just because its edge- the 3.x one needs love). Are you subscribed to the Rails security mailing list and core list?

There is plenty to security that is not in the Rails documentation, though, and it is by no means bulletproof by any stretch of the imagination. If you don't take security seriously, then you increase the chance of getting stung. From exploits that you have to watch out for in your code, to setting up the server(s) and infrastructure in the most secure way that makes sense, it is a lot of work.

Reply on news.ycombinator.com