Userland rootkits are lame (2022) | Hacker News Reader

Userland rootkits are lame (2022)

grugq.substack.com

45 pointsqweqwe1445 comments

Author here.

The context of this post is somewhat important. It is a direct response to a post titled: Symbiote Deep-Dive: Analysis of a New, Nearly-Impossible-to-Detect Linux Threat

Userland rootkits are not “nearly-impossible-to-detect.” They are not novel, they are not impossible to detect, and they are not the pinnacle of hacker techniques.

I felt that it was worth pointing out that the history of userland rootkits goes back a ways and that they were very easy to detect because they rely on proxying all access to the system. If you bypass the hook they use to enter their proxy, they you evade them entirely.

Forensic and incident response guides used to advise using static linked binaries for exactly this reason. There are guides from the 1990s telling people to do this because userland rootkits were an issue (before kernel rootkits everyone used userland rootkits.)

Here is an example from 2013 which points out that you can’t trust any binaries/libraries on the potentially compromised machine and should use statically linked tools. [0]

LD_PRELOAD rootkits are not new and they are not nearly-impossible-to-detect to detect. My post listed a number of ways to detect them, all of which have been known for decades.

[0] https://www.forensicfocus.com/forums/general/trusted-static-...

> they rely on proxying all access to the system

It’s worth noting that this has a measurable and enormous impact on system performance, because they usually are adding a bunch of strcmp or similar to every incantation of a bunch of different libc calls.

Adversaries live by the saying "if it works it isn't stupid."

What they actually did was to implement kernel-level rootkits.

This is why things 'secure boot' and requirement for signed drivers are a thing.

The downside with kernel-level rootkits is you essentially have to compile it for many different kernel versions if you want it to work everywhere. I think I've read about some malware that literally contacted a server, sent the kernel version, and the server would compile the rootkit on demand.

And that's service right there - they tried to distribute the most compatible malicious kernel exploits for you, but sometimes you need a bespoke compilation for your system -- and these guys step up and make sure you get one!

That's customer service, I think a lot of (all of) our trillion dollar overlords could do a thing or to and learn about providing reliable service.

Kernel level code injection doesn't require code signing.

ld_preload is the lazy way of doing userland rootkits. there are other ways.

For example, you can patch libc, and by patch I mean there are different ways of patching. a simply binary patch is unreliable because it'd be pinned to a specific version but you can parse the elf symbols dynamically and patch them, even when libc is updated (assuming certain critical symbols are unlikely to ever change) so it becomes reliable. a lot of "static" programs still dynamically load libc.

There is also a more aggressive but easily detectable approach, just patch every elf file on the system so that that the _start/entry always does something to prevent discovery of the malware like messing with the plt so that all function calls are proxied through rootkit code, where the rootkit filters for parameters and return values it doesn't like.

Kernel rookits are great but secure boot is also a thing, if implemented correctly it would be very difficult to beat. you don't even have to go full on rootkit, just a stealthy implant running as root via a hijacked dynamic library (doesn't have to be libc) will do. this isn't windows, it's not like there is a reliable/common way of making sure system libraries are legit (like authenticode/pe signing). The package managers make sure the correct binaries are being installed but they don't have something like sfc on windows (system file checker) that refuses to let the system boot if files are corrupt and/or restore tampered system executables.

Bringing your own static linked busybox will still evade that rootkit.

If the attacker has modified the environment to present a specific view of system state, bringing your own environment defeats it.

There are tricks which are better than modifying things to hide. For example, there is a race condition between opendir() and readdir() which you can win by using inotify(). Then you can unlink() whatever, wait a while, then link() it back in. During that time it will be deleted and thus invisible to any detection. (I saw a demo of this 12 years ago, so I might be misremembering a bit. I know it used inotify() and unlink())

Not if your running processes infect running userspace processes.

like a sibling comment mentioned, process injections can also happen. but besides that, if your busybox wasn't already on the system then what's the value of bringing it when you suspect a rootkit? userland or not, a memory acquisition for the system for off-box analysis (volatility) would be ideal and most reliable in my opinion.

Is it really practical to just statically compile everything so you don't get a rootkit? Is that really the solution?

I don't think that's what the article is suggesting. It's saying if you suspect a system to be infected you can install some statically-linked tools to diagnose it.

Some of the rootkits i've written attack statically compiled executables, so shrug.

Have you written about this? I'm fascinated by exploit dev.

In case you're interested, it's usually done via ptrace by attaching to every process and modifiying the syscall arguments every time it makes one. There are performance issues associated with that (ptracing is rather expensive) and IMO it's more complex than LD_PRELOAD. Furthermore, ptrace may be disabled altogether on some installations, even for root. See yama_ptrace_scope

I wouldn't say it's a practical approach. Works for a cool demo, sure, but as an adversary I would be hesitant to use this widely.

And you can detect when you are being ptrace()d because a process cannot be ptrace()d twice. Unless they changed Linux again.

There are also timing issues that show up, and you can do any number of anti-debugging tricks which would reveal that the environment is being manipulated. Which is an instant red flag.

In general if the attacker is running at the same privilege level, you can probably evade it or at least detect it. I’m somewhat surprised there isn’t a basic forensics tool that automates all of these tests already.

“sus: [-h] [-v] [-V] [-o file] [-O format]

Sus tests for common indicators of compromise using on generic tests for common methods of implementing userland rootkits. It will check for LD_PRELOAD, ptrace(), inotify() and verify the system binaries match the upstream distribution hashsums. It can be used to dump the file system directly (warning, slow!) for comparison against the output of `find`. See EXAMPLES for more.”

Implementation is left as an exercise for the reader.

I almost got nerd sniped by this comment.

There's also TracerPid field in /proc/PID/status, which is non-zero when a process is being ptraced

> Sus tests for common indicators of compromise

There's a lot of stuff that Linux malware tends to do that almost no legitimate program does, this can be incorporated into the tool. Just off the top of my head, some botnet clients delete their executable after launch, in addition to being statically linked, which is an almost 100% guarantee that it's malware.

Check for deleted executables: ls -l /proc/*/task/*/exe 2>/dev/null | grep ' $deleted$$'

(not really reliable) check for statically linked running programs: wc -c $(grep -L libc /proc/*/task/*/maps 2>dev/null) 2>/dev/null | grep -v '^0 '

Although a malicious process can just mmap libc for giggles, and also theoretically libc can be named in a way that doesn't contain "libc". A more reliable method is parsing the ELF header in /proc/PID/exe to determine if there's an ELF interpreter defined.

You can also check for processes that trace themselves (TracerPid in status == process id), this is a common anti-debug tactic.

You can also hide the process by mounting a tmpfs on top of it's proc directory, tools like ps ignore empty proc directories due to the possibility that the process has terminated but it's proc directory is still around. This is obviously easily detectable by checking /proc/mounts or just listing empty directories with numeric names in /proc

Another heuristic can be checking /proc/PID/cmdline for two NUL bytes in a row, some malware tries to change it's process name and arguments by modifying the argv array, however they are unable to change the size of cmdline, hence having multiple NUL bytes is a viable detection mechanism. Legitimate programs do this too, but it's rather uncommon.

You can obviously combine these heuristics to make a decision whether the process is malicious, as by themselves they aren't very reliable

I am reasonably sure that the intended behaviour of Linux is that a process can only be ptraced by one other process level at a time.

However, a few years ago, I discovered that a process inside a container being ptraced could be ptraced by a second process running as root at the host level.[1][2] I don't know if that's been patched away since then, but my assumption at the time was that it meant that the "there can be only one" aspect of ptrace was more of an arbitrary decision, not a hard limit.

[1] https://assets.bishopfox.com/prod-1437/Documents/Guides/2022...

[2] I'm not sure if the "double ptrace" scenario made it into the final document, but it's the same techniques discussed in there, just attach a tracer to the containerized process from inside the container before you attach gdb or asminject.py from outside of the container.

I cannot find any reference to sus project, care to link some?

It is not real, OP is suggesting that it be written. I was so very tempted to write it. The only downside to writing it is that once you have it in circulation, it moves the goal posts and rootkit authors with some skill will use less obvious techniques to hide their software.

If you have root access, you may be able to re-enable ptracing:

https://github.com/BishopFox/asminject/tree/main/ptrace_scop...

That specific technique only works if root can still load kernel modules, but if I could throw that together with minimal knowledge of the Linux kernel's inner workings, there's probably a sneakier way.

Static linking does get used as a hardening measure in fact, but mostly at the level of individual applications from what I’ve seen.

I'm of the opinion that all systems should at least have a static busybox, it's useful for more than just rootkit hunting, for example if you somehow break your installation because of glibc shenanigans (rare but happens)

If it’s weak, I’d expect a standard distribution to detect it by itself and inform the user.

Only revealing itself to a defender who looks for it sounds like it’ll be working as intended in almost all cases.

What are uses of LD_PRELOAD that make us keep it?

Fixing bugs in a function in a library. Overriding libraries.

Patching the binary seems like a better way to do this.

Better long term,but also much harder. Some anti cheat software will be tripped if you do this as they checksum the binaries on disk, that of course can be worked around so its just a matter of understanding your adversary.

Patching libraries is error-prone and annoying.

Replacing allocators https://github.com/jemalloc/jemalloc/wiki/Getting-Started

Pretending you're running on a remote system https://github.com/viraptor/libremotec

Cheat in games https://github.com/ProfessionallyEvil/LD_PRELOAD-rand-Hijack...

Fixing bugs where the upstream can't do it.

Some uses of LD_PRELOAD off the top of my head:

- GNU stdbuf (BSDs use an env-var instead of hijacking libc)

- fakeroot

- gprofng's data collector

- For a long time it was necessary to use LD_PRELOADto hack around a bug in flashplayer because flashplayer used memcpy when it should have used memmove.

Do you mean the functionality itself (inherent in the capabilities of a dynamic ELF loader) or specifically that it uses an environment variable? Maybe moving it to static configuration in ld.so.conf would cut down on abuse.

tsocks is quite useful for proxying a program's network access over ssh using SOCKS. I use it for accessing geo-blocked servers using command line tools without a full VPN.

This sounds like a food youtuber trying to explain how easy cooking a Turkey is. Yeah sure in a vaccum maybe, but I don't spend all day cooking food. Ive got other stuff to do.

Maybe its easy to find rootkits if you know what you are doing and use a statically compiled tcpdump. Do you know how many times I've done a TCP dump on my computer to check for root kits statically compiled or not? 0. I don't spend my time doing that. All it has to do is use less CPU and RAM than my corporate anti virus already does ( which is a lot ) and I'm never going to find it, cause im never gonna look.

Yeah but this is more from the perspective of a blue teamer, someone actively looking for/protecting against malware or performing forensics. If you're not in that position, just have some good AV and don't do anything that might make you a target.

> don't do anything that might make you a target.

Like "don't get connected to the Internet"? That's a tall order.

You may not have noticed, everyone is on the internet. From the penthouses in new york to small villages in the savannah. Just being connected to the internet isn't noteworthy or targeted.

Being targeted by an adversary is not the same as being hit by random drive-by attacks hitting every IP in the relatively limited IPv4 address space.

That is what the antivirus is for. If you are not a high value target, then your risk is virus that are being widely distributed; and those will quickly find their way into an AV database.

If you are a high value target, AV is useless, since the attackers you care about will target you with bespoke mallare, that will not be in an AV database.

Agreed that it still seems decently hard to find if you aren't looking for it. Though the tone of the blog post made a lot more sense to me when I saw the over-top-top title of the article[0] it's responding to: 'Analysis of a New, Nearly-Impossible-to-Detect Linux Threat':

[0] https://intezer.com/blog/research/new-linux-threat-symbiote/

In retrospect I really should have included the title with the link.

The target here is not a random YouTube watcher though. If you're doing actual rootkit hunting, you're in a larger org with lots of other indicators in place. You'll have DNS and traffic logs (hopefully), some local agents, maybe managed devices.

You're not going to find rootkits randomly by deciding to check with tcpdump. You're looking for them in the first place, because they tripped some other detector. That's a big prerequisite.

Cooking turkey in a vacuum… sous vide is your idea of easy? Props, dude!

Put meat in bag, put bag in water, receive protein?

A turkey might be hard by size, but sous vide is basically the easy-bake of cooking, takes no effort or creativity during the cook and you can only really impact it before and after.

Reply on news.ycombinator.com