Easy, realtime, system-wide Shellshock monitoring | Hacker News Reader

Easy, realtime, system-wide Shellshock monitoring

draios.com

91 pointskristopolous32 comments

Has anybody looked for historical Shellshock exploit traffic (in old/backed-up logfiles)?

Though none of the exploits I have seen thus far look very obfuscatible and therefore probably would've been discovered already.

Yeah I did. I run a niche site that gets about a thousand hits a day. It has a cgi that uses bash. I've got logs going back to January, and the first (and so far only) attempted exploit is from shellshock-scan yesterday. It came after the first bash patch, which I had applied, and did not succeed.

How complete is your log? Does it log every header, or just User-Agent?

Now that the exploit is public, people don't care anymore, but anyone who knew about this bug and tried to exploit it before it was public would be careful to avoid using a commonly-logged HTTP header.

Is this log public available ? I would like to dig into it.

This Google Doc has a list of URLs tested by the detectify.com scanner:

https://docs.google.com/document/d/1vN2QOG2OZIAHGXDmd5wB8FPi...

I have some truly old (disk) images going back to around 2003. I've looked through them and found nothing.

What freaks me out though is that these systems were vulnerable. All this time; it was right there. Its like finding out you just drove from LA to NYC with nothing but a single loose lug nut on your left front tire.

I'll give props to hackers clever enough to poke holes in a massive beast of a system that enjoys boasting itself as "impenetrable", but these, I'm not too happy about - http://www.pressreader.com/profile/Media_Mentions/shellshock. The potential effects are much too close for comfort.

Just so I understand fully, this doesn't block attempts, just logs them?

Correct. It will log time, process name/pid, and what's going to be executed.

Why would you prefer this over the LD_PRELOAD workaround provided by redhat?

The LD_PRELOAD workaround is meant to "fix" the shellshock bug, sysdig doesn't do that, it just passively monitors every injection attempt up to the point where the injected function is actually read by a newly spawn bash.

With the LD_PRELOAD trick you can definitely secure your system, but you won't be able to see if there's a new service that is being used as an attack vector (for your own curiosity). With sysdig, you can, and if you capture a trace file you can also follow the exact process chain that caused the propagation of the environment variable.

Why not actually also stop them?

That can get tricky as a general case, but would probably work OK for most people in this specific case. But, then again, hopefully you've patched bash already anyway, so blocking hosts that are scanning for a vulnerability that you've already fixed probably won't accomplish much.

Generally speaking, if you use something like iptables to block abusive hosts, you dive head-first into a very deep rabbit hole. Usually sysadmins don't want hosts blocked forever or iptables with 30k+ lines in them, so now you have to also add some kind of automated ban-clearing feature. Then you want to make sure you don't ban certain networks, so now you have to have some kind of whitelist feature. Then sysadmins will want to be able to tune which networks are trusted and which aren't, so now you have to add some configuration options for it. And so on.

I've written some software for my servers that does this for several different annoyances, and I spend almost as much time tuning the software as I spent dealing with the annoyances in the first place.

If sysadmins really want to auto-ban abusive hosts, you're probably better off letting them do it with something like Fail2Ban, and then all that muckety-muck becomes their problem, and not yours.

I wouldn't use a program B to circumvent a bug of another program A if not in a exceptional case in which patch of program A cannot be created.

This is exactly the case. If bash has a bug. that must be fixed. And it has been done already. Just update bash in major distributions and the bug is gone.

But still you want to have a tool like sysdig to detect if your system has already been compromised previously and it is out of sysadmin control.

This looks cool but I can't get it running on Ubuntu 14.04. I just installed sysdig but I don't have the shellshock_detect chisel :/ Is it available yet through apt?

What do you get if you run "sysdig --version"?

If you used the official Ubuntu packages, those are a few versions behind upstream (currently at 0.1.87 while we are at 0.1.89): http://packages.ubuntu.com/trusty-backports/sysdig.

What we recommend is uninstalling those ones (sysdig and sysdig-dkms) and just use the binaries that we, Draios, provide, following this: https://github.com/draios/sysdig/wiki/How-to-Install-Sysdig-...

Should be very easy, and sysdig --version should show 0.1.89

Ah ok, mine says "sysdig version 0.1.87". Thanks, I'll give that a shot instead.

version on brew is 0.1.88. What's the best way for mac users to get this?

Yeah, bummer, I can submit a PR to Homebrew but it would take a few hours/days, and we don't ship OSX binaries from Draios, so why don't you go with:

https://github.com/draios/sysdig/wiki/How-to-Install-Sysdig-...

Assuming you have a C/C++ compiler installed (comes via XCode) it really takes like 2 minutes.

Lazy alternative, in a couple days maximum Homebrew should be updated, unfortunately it doesn't depend on us.

Also, notice that sysdig for OSX doesn't (yet) have live capture, so you'll just be able to run the chisel on a trace file that you previously created on a Linux host.

If you publish the updated brew, can't it manually be added by users?

Yes, we most definitely can publish a private brew tap. I'm no expert as I mainly use Linux, but my understanding is that I'd need to create a brand new draios/homebrew-sysdig repo. I'll try to find the time to look into this and update the documentation. If, instead, just a PR to our main repo would suffice, feel free to send it over and we'll merge it in no time :)

Why is the Ubuntu package version behind the upstream version?

We just released 0.1.89 (special release to include the shellshock chisel) a few hours ago, so distribution maintainers aren't that fast: https://github.com/draios/sysdig/releases

Debian is currently at 0.1.88: https://packages.debian.org/sid/sysdig

And Ubuntu periodically merges all the unstable packages from Debian, so that's why they're lagging one version behind at this moment.

I guess there are already worms around, exploiting this bug.

Thus, has somebody thought of exploiting and patching the attackers in response?

I haven't studied a worm in years, but historically it's been common practice to close the door you came in through upon entry, for exactly this reason.

Common, but by no means ubiquitous.

shellshock_detector.lua seems to let this more obscure exploit slip by, undetected:

http://seclists.org/oss-sec/2014/q3/696 http://seclists.org/oss-sec/2014/q3/734

that has now been fixed: https://github.com/draios/sysdig/pull/252

Seems like an interesting article, however with the super thin font and the grey-text-on-grey-background it's just too hard to read.

The authors need a visit to http://contrastrebellion.com/

http://rdd.me/ei7jpz02

well, installing a LKM, just that =p

Now sysdig aint bad per se but id like to see it mainlined or using mainline code

Fair point, even though:

- At this point sysdig is estimated to have tens of thousands of users, and we haven't gotten a kernel bug in a while, with people (us included) regularly using it a lot in production. Of course, I see the irony of mentioning this in a "shellshock" thread

- the dkms packaging should completely hide all the complexities required in maintaining a kernel module

- Part of the kernel code, if you look at the contributors, has been written/reviewed by gregkh, so we like to think the quality is "high enough"

- There might be plans at some point to try and propose a merge of the code to mainline

its respectable but in the end it doesnt matter. when you have to run this on thousand of systems that have not been tested with that LKM, the LKM can potentially destroy everything.

its not like if grekh code was bug free - theres a lot of bugs being fixed daily in the kernel as well.

additionally, the kernel distribution path has better verifications than sysdig's and sorry, ill trust that more than a few guys. It doesnt make your work any less, its just the way it is.

Reply on news.ycombinator.com