Bpftime: Userspace eBPF runtime for fast Uprobe and Syscall hook and Plugins

We talked about it in LPC 23 today, and the slides: https://lpc.events/event/17/contributions/1639

Stream: https://youtube.com/watch?v=zDNZY0HQOMw&t=1765s

The draft paper: https://arxiv.org/abs/2311.07923

I have been talking about someone needing to do this recently, uprobes are largely quite slow for tracing (e.g. with bpftrace) because it has to trap into the kernel every single time, unlike tracing anything in the kernel because you're already in the kernel - so no context switch is needed.

It was always possible to do something like this, to inject the bpf code to run in-process and only export data occasionally just like the kernel side version - and the pieces already existed but no one had put all the pieces together, now someone has. Awesome!

Just need someone to implement this seamlessly into bpftrace :) That would be amazing!

If you want to understand why it's slow without this, I recently did a talk explaining the difference in tracing speed between strace and bpftrace (when tracing things in the kernel) that may help! In this case (which I didn't really cover), most function uprobes have to trap into the kernel and back twice (once on function entry and once on return, I explain why) - not as bad as the full strace example I explain in detail in the video - but still comparatively expensive.

https://www.youtube.com/watch?v=ZDTfcrp9pJI "bpftrace recipes: 5 real problems solved"

I re-ran this talk (mostly the same, a few minor updates) at the Riga Ubuntu Summit a couple of weeks ago but the videos aren't up yet. But the version above is mostly the same and was a pretty good recording :)

A full version of the preliminary descriptions and results can be found here http://arxiv.org/abs/2311.07923. Thanks for the advise for cross boundary communication brought on the LPC, we will integrate!

This looks really cool. Like all eBPF though, what sort of eBPF verification is it doing and is it possible to do the verification in CI?

With eBPF the common gotcha is you can compile totally invalid eBPF code, with no compile errors, only to find out at go time (and only to be able to find out at go time).

Sounds a lot like DTrace user defined probe tracing.