Attempts to reinvent Dtrace continue unabated thanks to stupid Sun making up a new OSS license. Such a shame.
It'd have been great to have GPLed, in-kernel Dtrace and ZFS for Linux.
It'd have been great to have GPLed, in-kernel Dtrace and ZFS for Linux.
Many people I know haven't been aware that Linux has had in-built dynamic tracing capabilities for years, with ftrace, kprobes and later uprobes. These are much more difficult to use (which is why I wrote some front-ends: https://github.com/brendangregg/perf-tools). But if you really cared about performance, you could use them. So it's not that Linux has been completely missing out; It's been missing out on a some specific features (eg, in-kernel aggregations and variables), and a nice interface.
As for attempts unabated to reinvent DTrace: FWIW, that's not really the genesis in the case of bcc/eBPF. Extended Berkeley Packet Filtering (eBPF) was developed to create virtual network infrastructures. Things like this: https://www.iovisor.org/sites/cpstandard/files/pages/images/... . It provides a kernel virtual machine for executing sandboxed bytecode. As a bonus, it can be used for system tracing as well, to do custom capabilities that ftrace/perf_events was missing.
There are too many tracers for Linux already, so I'm pretty glad that eBPF is getting integrated into the Linux kernel, since it should stop discourage anyone inventing yet-another-linux-tracer, since the kernel will already have a powerful tracer in-built. We'll no longer need new tracers. We'll want front-ends, like bcc.
Also: Thank you for working on Linux tracing - your blog posts have been very valuable to keep track of where it's all headed.
Another example of duplication is BPF and iptables, but I suppose iptables isn't going away soon simply because it's so popular (and, compared to BPF, simple to use)?
- perf_events (the "perf" command), which is intended as the official end-user profiler/tracer. It's great at PMCs and sampling. It can do dynamic tracing and tracepoints.
- ftrace (currently being renamed to "trace", to bring its ambiguity on-par with doing an internet search for "perf"), which is really a collection of custom lightweight tracing capabilities, developed to aid the real time kernel work.
- eBPF, which is an engine that both perf_events and trace could make use of.
I talked about the more here: http://www.brendangregg.com/blog/2015-07-08/choosing-a-linux...
So you could say that the plan is there is one: perf_events. There's already work to bring eBPF to perf. perf already has a Python scripting interface, so it's not inconceivable that one day bcc will become part of perf.
Some (f)trace capabilities could be rewritten/improved in eBPF, which could mean some cleanup. But the ftrace implementation wasn't bulky to start with.
It's somewhat slow, the UI is clunky and I had to comment out a rather anal sanity check related to module timestamps from the source, but it works. By far the biggest drawback is it has no formal API, instead you have to wrangle with subprocesses to automate it. It's pretty ugly to see things like the SystemTap initscript or SystemTap toolkits which execute scripts as interpolated strings with a template engine, though that's reality.
It has certainly improved from its former reputation as a routine kernel panic trigger, though fiddling with guru mode haphazardly will still hang or crash your system perhaps a bit too easily.
SystemTap has a grammar and extensive tapsets, so one interesting possibility would be for SystemTap to use eBPF as a backend: compile to eBPF bytecode. You'd then get safety, a grammar, and all the tapsets, which include many tracing odds and ends that you sometimes run into, which SystemTap has solved years ago.
Although at times I think the tapsets have gone too far, and become too big and untested (one of the reasons I started writing lightweight tools https://github.com/brendangregg/systemtap-lwtools).