Analysis and Exploitation of a Linux Kernel Vulnerability | Hacker News Reader

Analysis and Exploitation of a Linux Kernel Vulnerability

perception-point.io

218 pointsddb65 comments

This vulnerability is a great advertisement for grsecurity/PaX, which mitigates it at the source of the issue thanks to PAX_REFCOUNT.

Unfortunately no major distributions ship those patches by default.

Why don't they? There are not that much performance downsides to PaX/grsecurity AFAIK. And it doesn't break the ABI monthly like OpenBSD does. (Don't get me wrong, I like OpenBSD personally, but this is not suitable for enterprises)

I don't agree with Torvalds, but at least I can understand him. I don't understand why distros don't implement it.

http://www.washingtonpost.com/sf/business/2015/11/05/net-of-...

Distributions prefer to minimize divergence from upstream, and therefore tend to avoid non-upstreamed patches.

While the fact that since last fall grsecurity only ships the stable branch of the patchset for sponsors (because of persistent trademarks violations) doesn't help integration in smaller distributions (also Debian, I would guess), I am always baffled as to why grsecurity/Pax were never chosen by a distro like Suse to differentiate itself from Red Hat.

FYI: If you're comfortable with Gentoo, the jump to Hardened Gentoo[1] is an easy one[2] and gives TrustedBSD[3] a run for its money w/r/t out-of-the-box security. It has the SElinux MAC, role based access controlled stuff from GRsec alternatively [you can mix and match GRsec + SElinux except for RBAC and SElinux's ACL system), grsec's ASLR at runtime, modified versions of musl/uclibc, stack protection support (why anyone would write software that could possibly be EIP exploited with countless ASan damn well integrated into some compilers, I haven't a clue), and a whole host more.

Some interesting academic work: http://hmarco.org/data/OnTheEffectivness-NX-SSP-RenewSSP-and...

While SSP is not enough, (use PIE too): http://cybersecurity.upv.es/attacks/offset2lib/offset2lib.ht...

A fun toy: https://github.com/JhetoX/VectorAttackScanner

[1]https://wiki.gentoo.org/wiki/Project:Hardened

[2]https://wiki.gentoo.org/wiki/Hardened/Introduction_to_Harden...

[3]Granted I haven't used it in ages, it's still the benchmark I use in my mind, back when Linux 2.2 was rife with out-of-the-box ROP vulns.

Freedom of the Press Foundation (disclosure: I am an employee) ships grsecurity kernels for SecureDrop. You have to build them yourself, but it really isn't that hard to do. We're working on making this easier for everybody by publishing guides [0] to building your own grsecurity kernel, as well as developing an Ansible playbook to automate building hardened kernels [1].

This is all still a work in progress, but expect a blog post once what we have is more polished.

A relatively easy-to-follow blog post was recently published that guides you through building a grsecurity kernel for a Debian desktop installation: [2].

grsecurity was operating a service last year that would allow you to request custom built kernels for a small price, although I can't find it now so it may have been taken offline. This is an ideal solution, and is actually better than packaging the kernels in an distribution's repository, because some of grsecurity's security features are more effective when everybody is running a unique kernel (e.g. RANDSTRUCT).

Finally, the linux-hardening group is working on upstreaming select PaX features into the mainline kernel [3], including PAX_REFCOUNT, which is the protection that would've mitigated this particular vulnerability.

[0]: https://gist.github.com/ageis/a91f36ca99c252291a00 [1]: https://github.com/freedomofpress/grsec/ [2]: https://micahflee.com/2016/01/debian-grsecurity/ [3]: http://lwn.net/Articles/668876/

Copperhead-os is an open-source android ROM that includes the security features of grsecurity. It runs on nexus-5 and Galaxy-S4.

https://copperhead.co/android/

Which of course ,raises the question: Why doesn't Google integrate this into android, or at the very least for a secure/business version of android ?

And it's not vulnerable: https://twitter.com/CopperheadSec/status/689553753423216641

Arch Linux does: https://www.archlinux.org/packages/?name=linux-grsec

> by default

Arch don't even have coreutils "by default". Yes, I know, you're supposed to install them anyway, but if you follow the standard install guide, there is no GRsec.

Oh, I interpreted the "by default" as "being installable without compiling it from source". But I guess more distros are able to do that.

Insomnihack just had a teaser CTF[1] where one of the challenges was to find an exploit in a poorly configured grsecurity-based Linux installation from an archive of the filesystem that was exposing a CGI based web server. It was a really fun challenge!

[1] https://ctftime.org/event/258

Can anyone identify a patch to apply to existing kernels to fix this? Are there any released 3.8+ versions that are not vulnerable? The "mitigations" section at the end is not forthcoming about this. Vulnerable systems remain vulnerable without such information.

This does not seem to be a responsible disclosure of the vulnerability (keep in mind it was posted on their blog 5 days ago).

Well, the exploit doesn't seem to work on my kernel 4.1.0 system, so that's good.

Did you update the addresses of commit_creds() and prepare_kernel_cred() to match your running kernel before you compiled/ran it?

How do you find those addresses?

Grep for commit_creds and prepare_kernel_cred in /proc/kallsyms. The address is in the first column.

https://gist.github.com/gcmurphy/1c91644718d28695da2d

^-- this version should do that automatically

By looking at /proc/kallsyms:

  grep commit_creds /proc/kallsyms
  grep prepare_kernel_cred /proc/kallsyms

Then update addresses as shown in one of the code snippets:

  _commit_creds commit_creds = 0xffffffff81094250;
  _prepare_kernel_cred prepare_kernel_cred = 0xffffffff81094550;

Thanks, that worked.

Question for you.

As a normal user, grepping these values I actually get 0000000000000000. I can't imagine these being the actual values. Is it possible that because I remount my /proc with the hidepid=2 option the values are not visible for normal non-root accounts?

That only hides the pid directories of others users, and indeed on my system remounting /proc with hidepid=2 I'm still able to see the same values for kallsyms. Maybe your kernel is compiled without the CONFIG_KEYS=y option? (I'm spitballing here).

It is indeed compiled with CONFIG_KEYS=y. Does this protect me against this issue? I'm not sure what this means.

No, the bug is in the kernel keyring facility, so if I'm not mistaken compiling with CONFIG_KEYS=n option should protect you (I haven't tested though). As for the /proc/kallsyms, I honestly don't know how come you only get zeroes.

EDIT: The obvious question I should have asked is which distro you are running. Also, as others have pointed out, hoping that the attacker can't read kallsyms from the machine he's attacking is not really a good defense plan.

I'm running Ubuntu 14.04 which should be affected. I just hoped it would be harder without having the correct kallsyms version. It seems I will have no options except to reboot my cluster :)

make sure you have privileges to read that file. I ran grep without sudo and got that address on Ubuntu 15.10, but with sudo it will display an actual address.

As I understand it the whole idea of this exploit is getting sudo access from a normal user.

Exploiting from root to root doesn't make a lot of sense. If the values are not retrievable as normal user this exploit can't be used.. Right?

The file /proc/kallsyms contains the addresses of various kernel functions: it's useful when you've already exploited a vulnerability in the kernel, and you want to do something useful with it, e.g. escalate your userspace process to root. In theory, since you're running code in kernelspace, you can do whatever; in practice it's much easier to start with known functions that can manipulate the data structures for you.

Think of it like the manual of a nuclear missile launcher. Obviously they don't want to give that out to everyone. But if you can break into the room with the launcher's control panel, you've still caused a major security breach even if you have no idea what to do next. Having the manual just makes your job easier, but the attack was that you broke in in the first place; they can't count on you being unable to figure out how to work the panel.

Also, if you're running a binary kernel (for instance, a kernel built by your Linux distro), the addresses in kallsyms are going to be constant for everyone running that distro. Hiding kallsyms for non-root users is primarily useful for people who build custom kernels.

Production-level malware often also has code that doesn't depend on having access to kallsyms and is more advanced (e.g., starting from the system call interrupt vector and disassembling it and seeing what addresses are jumped to). Using kallsyms is good for keeping a simple proof-of-concept exploit readable.

The /proc/kallsyms values are identical across identical kernel compilations.

Suppose you have a vulnerable CentOS 7 system that you want to exploit - you could get the proper addresses from your own CentOS 7 VM running the same kernel, apply those to a modified exploit compilation, and run that compilation on your target host.

Latest versions of Linux have Kernel ASLR for text but I think lots of distributions have it disabled because they set CONFIG_HIBERNATION: https://www.kernel.org/doc/Documentation/kernel-parameters.t...

Also, it is not bullet proof because apparently there are lots of info leaks in linux and I think linux also does not reboot after a panic (http://www.cyberciti.biz/tips/reboot-linux-box-after-a-kerne...) so if the entropy for KASLR is small enough you can retry very aggressively. Though in this particular instance if you have to wait 30 minutes between each try that would kill brute forcing.

No, it would seem (I'm not sure though) that this happens for you because your distro is smart enough to have a nice security setting be on by default. The setting / sysctl can be changed at:

  /proc/sys/kernel/kptr_restrict

For me, this gave me 0, but if you cat this it should give you 1. So essentially it's a setting but I'd wager it's not related to you remounting /proc.

Just for curiosity's sake, from searching around it seems that this particular security feature (hide kernel pointers for unprivileged users) seems to have originated from this commit (which in itself implements a broader and more general security feature): http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.g... - but it's often not enabled.

So it's not ASLR or anything to do with filesystem options (per se), but rather a custom "kptr_restrict" kernel vs. userspace security feature provided by the kernel:

"[...] The %pK format specifier is designed to hide exposed kernel pointers, specifically via /proc interfaces. Exposing these pointers provides an easy target for kernel write vulnerabilities, since they reveal the locations of writable structures containing easily triggerable function pointers." (Dan Rosenberg, 2011-01-13 - nice.) (Of course if there's no ASLR and one knows which distro is being targeted, it's possible to find those values on another machine and to just bake them into the exploit, as others have pointed out. Also, other channels exist for finding these addresses, etc.)

Arch seems to be kosher

  $ uname -r
  4.3.3-2-ARCH

No root:

  $ ./cve_2016_0728 PP_KEY
  uid=1000, euid=1000
  Increfing...
  finished increfing
  forking...
  finished forking
  caling revoke...
  uid=1000, euid=1000

Only took about 25 minutes on a Lenovo core i7. That's kind of scary.

EDIT: Actually it seems that atomic_t is 64 bits long with x86_64 these days, so no hope of overflowing.

Arch is not kosher. You need to do this: https://news.ycombinator.com/item?id=10931954

I did. I still maintain that the reason why the exploit doesn't work is this:

include/asm-generic/atomic-long.h:

  #if BITS_PER_LONG == 64
  typedef atomic64_t atomic_long_t;
  ...
  #else
  typedef atomic_t atomic_long_t;

Just checking it with a 32-bit VM with the latest upstream kernel.

EDIT: No, I was wrong. atomic_t is 32 bits also with 64 bit architecture. Must be something else then, let's see.

Is there a reason that there's no overflow check on these refcounts? I suppose it's something like, these are hot paths in general and there's no good hardware support for atomic-increment-if-not-overflow?

The path can only be so 'hot'... any given cryptographic operation using the artifacts maintained by this `keyring' system will be many orders of magnitude greater than an overflow check on allocation, hardware support or not. I think we're left to conclude this is just another instance of naive C programming.

The blog post specifically thanks the Red Hat Security Team, but according to the Red Hat Bugzilla, no patch has been released yet for RHEL/CentOS:

This issue affects the Linux kernels as shipped with Red Hat Enterprise Linux 7 and will be addressed in a future update.

https://bugzilla.redhat.com/show_bug.cgi?id=1297475

Premature blog post?

Tested on CentOS 7, fully patched.

    [ohadmin@localhost shm]$ ./cve_2016_0728 PP_KEY
    uid=99990, euid=99990
    Increfing...

This is taking a long time. I disabled SELinux and it has been cranking away for a while now.

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    1140 ohadmin   20   0    8428    388    296 R 100.0  0.0   9:25.17 cve_2016_0728

No need to test on CentOS 6. Forgot how ancient that kernel is.

Update: I am not having any luck getting this to work on CentOS 7. I even completely disabled SELinux (selinux=0 vs setenforce 0) Anyone else getting this to work?

That's not how you execute binaries, just "./cve_2016_0728 PP_KEY" is the correct syntax

Yes sorry. I will make the cheap excuse that I am recovering from food poisoning and don't quite have it all together. Thankfully I don't manage nuclear weapons, so we are all safe for now.

You need to update the addresses of commit_creds() and prepare_kernel_cred() as per this: https://news.ycombinator.com/item?id=10931954

The best I can get is an oops panic on CentOS 7.

It appears smaps may be preventing the exploit from working.

Given that it looks like commits were just made 12 days ago (https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux....) or possibly as recently as 38 hours ago (https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux....), perhaps only 4.4 has the fix?

Would a properly setup SELinux able to capture a local program that exploit this kind of bug?

Is there any other other of Linux/Mac/Windows/Android dist that have some type of security framework capture this kind of issue?

As others said, using a kernel with the Grsecurity patchset would prevent the issue (I believe the configuration of SElinux on Android should be sufficient, but the default config in RHEL7/Fedora is insufficiently strict).

Yes.

Well, Android and most RHEL-derivated distros use SELinux and it can mitigate this kind of problems (assuming it is well configured) and Ubuntu has "AppArmor" but I don't know if it mitigates this issue as well.

Fedora is currently building a 4.3.3-X for F22 and F23 that includes a fix.

To me it's kinda baffling that an open source project so intensively used had a root privilege exploit for years. It is often said that open source is inherently more secure because "everyone can look at the source" yet quite a few exploits in in the last few years alone kindly proved the opposite.

This is not a jab at open source software, just at the general statement which people like to throw around when referring to the security of open source software; "It's open so it must be safe!"

i guess learning that unqualified opinions don't really convey useful information is a valuable lesson no matter how you learn it.

Open source is inherently more secure then closed source, not because of how the code is written and reviewed, but because of how the code is published and distributed.

Suppose we have two projects with 100% identical source code, which are mathematically proven to both contain 0 bugs, and which have been extensively audited by third parties.

Despite being identical, the open source version will be much more secure for end users, because the source code and machine code can be obtained, compiled and distributed by a much larger number of competing parties.

This allows users to compare compiler output and run-time behavior, to verify that software being run is actually the software being written, and to ensure that the software is not surreptitiously modified by the publisher during its distribution.

With closed source software, even if developers write a 100% perfect codebase, the end users have no way of knowing whether they are actually getting that specific code base in their binaries, as there are legal and technical barriers in place preventing them from reliably making that verification for every change.

What about the corollary to that? Open source code is more accessible than closed source code, not only to reviewers, but also to vulnerability hunters. For every honest reviewer, there can be a malicious one. Also consider the profit incentives are stronger for malicious reviewers than they are for friendly ones.

The National Institute of Standards and Technology (NIST) in the United States specifically recommends against this practice

https://en.wikipedia.org/wiki/Security_through_obscurity

Free software allows users to get access to fixes for systems not supported by vendors. Saying "free software" is more secure is not correct (just look at libpurple, a free software library riddled with bugs). What is correct is that you have a much better shot at white hats and security researchers finding bugs in free software, and the fixes for those bugs being freely available.

Black hats (trust me, I've known a few) will gladly attack proprietary software (usually with more glee, because there's far less review of proprietary software in general). Using proprietary software doesn't save you from black hats. But it does give you more chances of white hats finding the bugs sooner.

Security isn't a binary state, it's a lifestyle. All code has bugs, and almost all bugs can be security bugs. Taking steps to add more layers to your security is what you should be doing, not arguing that "because the software is free someone found it!".

Has the demo code actually worked for anyone?

I'd be surprised if this code actually gives #. it would be like winning a lottery :)

I'm not big on patching needlessly. Guess I'll have to wait for a better tester.

Did not get root on my test box (OpenSuSE 13.2 64bit) with kernel 3.16.6-2-default

  uid=10000, euid=10000
  Increfing...
  finished increfing
  forking...
  finished forking
  caling revoke...
  uid=10000, euid=10000
  sh-4.2$

If you have SMAP, i.e. an Haswell or newer Intel CPU, you should not be vulnerable, so that could be an explanation.

Is SMAP required for mitigation, or is SMEP enough?

IIUC SMEP is on Sandy Bridge processors too.

According the lwn comments it should be sufficient (and the post by perception-point suggests that it would at least make things more difficult), but I haven't the hardware to test for myself.

I have Sandy Bridge, i7-2820QM. The exploit code has been running for nearly an hour, still "Increfing..."

EDIT:

    [chris@f23m cve20160728]$ ./cve_2016_0728 PP_KEY
    uid=1000, euid=1000
    Increfing...
    finished increfing
    forking...
    finished forking
    caling revoke...
    uid=1000, euid=1000
    sh-4.3$

SMEP would stop this particular exploit because it returns into usermode but SMEP is trivial to bypass on linux if there is no KASLR or other mitigation (apparently there are compiler plugins that remove popular stack pivot gadgets).

Did you update the addresses of the kernel functions?

    #define COMMIT_CREDS_ADDR (0xffffffff81094250)
    #define PREPARE_KERNEL_CREDS_ADDR (0xffffffff81094550)

even if you don't update the addresses you would at least get an oops if the exploit is working. I've explained the problem with rcu calls in my post https://cyseclabs.com/page?n=02012016 reply but did not describe the technique for ordering these calls. The way they do things in the poc, you'd be lucky if it succeeds once out of 100.

Reply on news.ycombinator.com