Does the limited flexibility it provides really justify the added kernel space complexity? I can understand it for packet filtering but some of the other stuff it's used for like sandboxing just isn't convincing.
You may not use it much, but some people use it all day. I think FAANG engineers have said that they run tens (hundreds?) of these things on all servers, all the time. And that's excluding one-offs. And FAANG has full time kernel coders on staff, so they're also funding this complexity that they use.
But also yes, I've solved problems by using eBPF. Problems that are basically unsolvable by non-kernel-gurus without eBPF. I rarely need it. But when I need it, there's nothing else that does the trick.
In some cases, even for kernel gurus, it's a choice between eBPF or maintaining a custom kernel patch forever.
I look forward to seeing that patch on Google Graveyard in a couple years' time.
You may not use it at your smaller scale. But there are millions of machines out there that do use it, and the alternative for the same functionality is much worse.
I bet you never use SCTP sockets either. eBPF is used much more than SCTP.
And its users "fund" its development, so it's not a burden to those who don't use it.
But are you sure your systems don't use it? Run "bpftool prog" to see. Whatever you see there someone thought was better than the alternative.
Would you mind giving some examples? I recently started learning about ebpf's from Liz Rice's book and is curious about what makes ebpf the correct choice in a particular scenario.
To add on this point: I successfully used SystemTap a few years ago to debug an issue i was having.
Before going further: keep in mind that my point of view (at the time) was the one of somebody working as a devops engineer, debugging some annoyances with containers (managed by Kubernetes) going OOM. I'm no kernel developer and I have a basic-good understanding of the C language based on first-years university course and geekyness/nerdyness. So in this context I'm a glorified hobbyist.
Learning SystemTap is easier in my opinion. I followed a tutorial by RedHat to get the hang of the manual parts but after that I remember being fairly easy:
1. Try to reproduce the issue you're having (fairly easy for me)
2. Skim the source code of the linux about the part that you think might be relevant (for me it was the oom killer)
3. Add probes in there, see if they fire when you reproduce the issue
4. Look back at the source code of the kernel and see what chain of data structures and fields you can follow to reach the piece of information you need
5. Improve your probes
6. If successful, you're done
7. Goto 4
I think it took like one or two days between following the tutorial and getting a working probe.
It was a pleasant couple of days.
The additional flexibility eBPF gets from this is amazing really. While dtrace is a more-targeted (and for its intended usecases, in some situations still superior to eBPF) but also less-general tool.
(citrus vs. stone fruit ...)
[0] https://bcantrill.dtrace.org/2005/07/19/dtrace-safety/
[1] https://www.usenix.org/legacy/publications/library/proceedin..., §3.3
- it cannot branch backwards (this is also true of eBPF)
- it can only do ternary operator branches
- it cannot define functions
- functions it can call are limited to some builtin ones
- it can only scribble on the one pre-allocated probe buffer
- it can only access the probe's defined parametersThere's also a need to make sure that even trusted users don't accidentally cause too much observability load. That's why DTrace has a circular probe buffer pool, it's why it drops probes under load, it's why it pre-allocates each probe's buffer by computing how much the probe's actions will write to it, it's why it doesn't allow looping (since that would make the probe's effect less predictable), etc.
Bryan, Adam, and Mike designed it this way two plus decades ago, and Linux still hasn't caught up.
This working model significantly increases the attack surface of the kernel, since it allows executing arbitrary code at a high privilege level. Because of this risk, programs have to be verified before they can be loaded. This ensures that all eBPF security assumptions are met. The verifier, which consists of complex code, is responsible for this task.
Given how difficult the task of validating that a program is safe to execute is, there have been many vulnerabilities found within the eBPF verifier. When one of these vulnerabilities is exploited, the result is usually a local privilege escalation exploit (or container escape in containerized environments). While the verifier’s code has been audited extensively, this task also becomes harder as new features are added to eBPF and the complexity of the verifier grows
DTrace was developed over 20 years ago; there have not been "many vulnerabilities" found in the verifier -- and we have not grown the complexity of the verifier over time. You can dismiss these as implementation details, but these details reflect different views of the problem and its contraints.
That's not to say that security researchers couldn't find DTrace vulnerabilities if they, for instance, built DIF/DOF fuzzers of 2023 levels of sophistication for them. I don't know that anyone's doing that, because DTrace is more or less a dead letter.
I appreciate the rest of tptacek's comment which is informative. I also acknowledge that there may not be fuzzers written that have been disclosed.
I'd also be open to an argument that the code quality in DTrace is higher! I spent a week trying to unwind the verifier so I could port a facsimile of it to userland. It is a lot. My point about fuzzers and stuff isn't that I'm concerned DTrace is full of bugs; I'd be surprised if it was. My thing is just that everything written in memory unsafe kernel code falls against Google Project Zero-grade vulnerability research, at some point.
That's true of the rest of the kernel, too! So from a threat perspective, maybe it doesn't matter. I think my bias here --- that's all it is --- is that neither of these instrumentation schemes are things I'd want to expose to a shared-kernel cotenant.
Thanks for helping me clarify this.
Until eBPF came around and said we can now prove it to be secure.
Until the sidechannel hackers came around to prove the opposite.