Hacker News Reader

Comment by xorcist | Hacker News Reader

Challenge accepted. "All files loaded" is probably not what you want to do however. It is much easier to just ask rpm directly which files under your library directory has been modified, and treat any files outside known library directories as suspicious.

Anyway, this is how you check which open files match ".so" and see if they are modified since installation:

  lsof | grep -o "/[^ ]*\.so[^ ]*" | while read path ; do
    pkg=$(rpm -qf "$path" 2>/dev/null)
    if [ $? != 0 ] ; then
      echo "$path does not belong to a package"
    else
      rpm -V $pkg | grep -F "$path"
    fi
  done

View on HN

I hope you know that there was a reason I wrote that challenge.

Your solution failed because you installed the rootkit that was aliased via the lsof command in the .bashrc.

Additionally, lsof like so many other tools rely on procfs, which allows processes to rewrite their own process names (comm) and arguments (cmdline).

Even if the malware of the article would run only in userspace (as non-root and "only a wheel user"), you certainly would have executed it.

My point being that you also forgot to check any processes against environment variables like LD_PRELOAD that the malware uses before executing any command (meaning even syntax programs like "if" as a program can be hijacked).

Again, this is a conceptual problem because there is a lot of programs in $PATH that can be executed by the same user, meaning only a kernel hook or ebpf module can audit/grant access to these kind of things to prevent that.

There is no trusted execution in Linux because of so many things down the line. Glibc, the $PATH mess, aliases, .local overrides etc.

Obviously you can't check for the presence of a rootkit while under a rootkit, in the general case.

Checking environment variables wasn't part of the challenge. The challenge was likely not intended to check for rootkits, because there are a thousands of other ways to place a rootkit apart from already loaded libraries. (Why check only open files? Closed files can also contain unwanted things.)

If the purpose is to check system integrity, just check all packages. That is much easier and faster.

If there is even the slightest possibility that the system is already compromised, do it from rescue media.

Why so?

Couldn't we have a nice overview of what kind of signed modules are valid in their integrity and authenticity based on cryptography?

(Also I wanted to point out that LD_PRELOAD was specifically mentioned in my comment, but it doesn't really matter, it's the lack of integrity checks across the /usr folders that are part of the problem. Glibc, $PATH, sideloaded .so files, kernel hooks...it's such a vast problem space of insecure development practices that by now we need a better OS architecture because all (old) tools down the stack rely on 100% trustable programs being installed, which after the invention of the internet is not a reality anymore.)

Here's a .deb version; only running debsums once per package name. Errors will go to stderr:

    sudo lsof | grep -o '/[^ ]*\.so[^ ]*' | awk '!seen[$0]++' | while read -r path; do
        if p=$(dpkg -S "$path");then
            cut -f1 -d: <<<"$p"
        fi
    done | awk '!seen[$0]++' | xargs -n1 debsums -s

(But is there a hard rule that says a loaded library has to be named .so, or show up as .so for lsof? I'm sure there's ways to avoid the above detection.)

> But is there a hard rule that says a loaded library has to be named .so, or show up as .so for lsof?

No, yes(-ish)

Filenames are just a convention and not necessarily enforced.

lsof will really list every file the kernel thinks has a handle held by an active process, but depending on your threat model I think you could get around this. For example you could copy the malicious code into memory and close the file, or use your preload to modify the behaviour of lsof itself (or debsums).

Anyway debsums is a great tool. I'd have used a command similar to yours, though maybe run debsums first, use `file` to filter dynamic libraries and then check which of those have been recently accessed from disk.

This didn't quite work for me - dpkg complained about "no path found matching..." for every library. I replaced the "$path" in the dpkg command with "${path##*/}" to just match the library name. Further inspection showed my package manager installed libraries to /lib (a soft link to /usr/lib/) but lsof returns files in /usr/lib.

Other than that it seems to work - i.e. no alarming output. :-)

Didn't check the code, but lsof seems a good approach.

How does that work with the various namespaces? From root namespace you should see everything. But in a mount namespace you could bind mount under a different name. How would that confuse things? With a SELinux module even root cannot do everything. If /proc is mounted privately does it change anything?

Not sure, just starting to think. Linux has become incredibly complex since the old days...

Edit: orc routinely loads executable sections not belonging to any package.

Besides namespaces, if one were malicious, one could also hide loaded libraries from lsof another way: open the library file, memcpy() the contents into RAM somewhere, mprotect(...,PROT_EXEC) that contents and then close the library file. You'll have to do your own linking, but then no open file will appear except for a very brief moment.

Seems like you assumed none of your tools got backdoored. I'd start bootstrapping from busybox.

If the system is backdoored, do none of these things. Boot from rescue media. Save only non-executable files and wipe the rest.

Do not trust key material, sensitive data or remote logins that the backdoored system have had control over. Repeat the same operation for them.

To check for backdoors, again boot from rescue media and do a full integrity check. Do not limit the check to open files.

Not even that is enough if the malware has loaded a kernel module.

Reply on news.ycombinator.com