Comment by woodruffw | Hacker News Reader

Comment by woodruffw | Hacker News Reader

Awesome post! `LD_PRELOAD` is a powerful tool for program instrumentation.

It's worth noting, though, that using `LD_PRELOAD` to intercept syscalls doesn't actually intercept the syscalls themselves -- it intercepts the (g)libc wrappers for those calls. As such, an `LD_PRELOAD`ed function for `open(3)` may actually end up wrapping `openat(2)`. This can produce annoying-to-debug situations where one function in the target program calls a wrapped libc function and another doesn't, leaving us to dig through `strace` for who used `exit(2)` vs. `exit_group(2)` or `fork(2)` vs. `clone(2)` vs. `vfork(2)`.

Similarly, there are myriad cases where `LD_PRELOAD` won't work: statically linked binaries aren't affected, and any program that uses `syscall(3)` or the `asm` compiler intrinsic to make direct syscalls will happily do so without any indication at the loader level. If these are cases that matter to you (and they might not be!), check out this recent blog post I did on intercepting all system calls from within a kernel module[1].

[1]: https://blog.trailofbits.com/2019/01/17/how-to-write-a-rootk...

There is a very recent development in this area -- there is now a way to do it without ptrace and instead entirely using seccomp[1]. It's somewhat more complicated (then again, ptrace is far from simple to get right) but gives you the benefit that you don't need to use ptrace (which means that debuggers and upstart will work).

It's going to see a lot of use in container runtimes like LXC for faking mounts and kernel module loading (and in tools like remainroot for rootless containers), but it will likely also replace lots of uses of LD_PRELOAD.

[1]: https://youtube.com/watch?v=sqvF_Mdtzgg

There's another way to intercept syscalls without going as far as a kernel module, using debugging API ptrace. There's a pretty neat article about how to implement custom syscalls using ptrace: https://nullprogram.com/blog/2018/06/23/

Yup! I discuss the pros and cons of using `ptrace` within that post.

It's all about the use case: if being constrained to inferior processes and adding 2-3x overhead per syscall doesn't matter, then `ptrace` is an excellent option. OTOH, if you want to instrument all processes and want to keep instrumentation overhead to a bare minimum, you more or less have to go into the kernel.

I've been looking for a while for a way to capture all file opens and network ops to profile unknown production workloads similar to proc. explorer on Windows, which I believe is implemented using ETW. Unfortunately strace seems to be out of the question purely because of the performance impact. Is the performance impact due to strace or ptrace itself?

It's ptrace itself: every traced syscall requires at least one (but usually 3-4) ptrace(2) calls, plus scattered wait(2)/waitpid(2) calls depending on the operation.

If you want to capture events like file opens and network traffic, I'd take a look at eBPF or the Linux Audit Framework.

I recommend bpftrace as an entry point to working with bpf

https://github.com/iovisor/bpftrace

This is really cool. Unfortunately the 4.x kernel requirement wouldn't work for the majority of my work since RHEL is still on 3 :|

If you have RHEL 7.6 or later, you have bpf

Most infamously, Golang programs use their own syscall wrappers, so you can't hook that with LD_PRELOAD.

You can use [1] from Intel to successfully intercept all syscalls from a given library, by default libc only. This library actually works by disassembling libc and replacing all `syscall` instructions with a jump to a global intercept function that you can write yourself. Incidentally it's also an LD_PRELOADed library.

[1] https://github.com/pmem/syscall_intercept

Yep, I give them a shout-out at the bottom of that post -- using capstone to instrument system call sites at runtime is great (if a little crazy).

I didn't actually realize it was an Intel project. I wonder how it stacks up against Pin.

Reply on news.ycombinator.com