BCC: Dynamic Tracing Tools for Linux | Hacker News Reader

BCC: Dynamic Tracing Tools for Linux

iovisor.github.io

118 pointsingve14 comments

For "Enterprise" environments it could take another 7 years to get to this kernel level. Rather depressing. Many of my clients are still on RH 5.

I hope you're working on moving them off it, since RHEL 5 reaches end-of-life March 31, 2017.

I suspect a lot of them would pull the same sort of thing as the IRS did with Microsoft and pay millions for continued support specifically for them. Few organizations would stay so far behind the curve without having a business justification worth a substantial amount... or their IT costs are so outrageous for the most basic levels of operation it's not too bad to pay half your IT budget to avoid having to upgrade and retrain your likely demoralized IT workforce and employees to handle an already-archaic OS like Windows 7.

These will be awesome once the newer kernels and LLVM versions that they need are widely available. I am really looking forward to it.

Hopefully you don't have to wait!

We're really hoping that this set of tools has only the minimal requirements, that being the kernel.

Since Clang/LLVM can be linked as a static or dynamic library, we only have a build-time dependency on those, and opt for the static library. The build scripts have steps to build clang+llvm from source if your distro doesn't include it. The binaries that we ship hide this and should (if we've done things right) only have a libc and few misc .so dependencies. A change under test even adds a fully static lua version.

On the kernel side, we don't conditionally compile anything based on kernel features, and instead try/fail at runtime. You can just upgrade your kernel and the tools will start to take advantage of the new features immediately.

Naive questions to people in the know: is it the case that ebpf/bcc are the standard for Linux tracing? Are they the Future(TM)? Or is the long-term vision a pluralistic world of Linux tracers?

BPF (formally eBPF) provides the programmable tracing capabilities that Linux previously lacked. Things like custom latency histograms. It works with other built in features: kprobes, uprobes, tracepoints, etc.

So yes, BPF is becoming the standard and the future. It made it into the kernel (well, the enhancements needed for tracing did; BPF has been around for a long time as just a packet filter). In other words, BPF won. bcc is likely the future, but we'll see other front ends as well (eg, ply).

I should add that Linux has two tracing front ends built in: ftrace and perf_events. And perf_events is building BPF support (see lkml). bcc is another front end for BPF. If this is confusing, I should categorize them like this:

back end technologies: ftrace, kprobes, uprobes, tracepoints, BPF

front ends: ftrace, perf_events, bcc

ftrace is hard to categorize: it is a collection of both back end and front end capabilities (although the front end is /sys). Also, it's being renamed to just "trace".

Thanks Brendan! A more navel-gazey question I believe you're well positioned to answer:

For a long time, there seemed to be a conventional wisdom that Linux tracing was mediocre, because it doesn't have DTrace. Has Linux approached or passed the "caught up" point?

It hasn't caught up yet, but it's getting pretty close. Although I'd think about it in terms of three dimensions:

- technical capability: we should pass it this year. We'll be technical capable of solving pretty much the same issues and then some.

- ease of use for script developers: bcc is currently ~10x more verbose to program. ply is much easier, but that project is in its early stages. This won't matter that much for Netflix, since much of the time staff will use this from a GUI (eg, Vector). It also won't matter that much for people who use the many provided tools, like opensnoop, tcpconnect, ext4slower, etc. It matters most for whoever codes the GUI and those tools (eg, me).

- marketing, awareness, brand: Sun marketing, sales, trainers, and evangelists, did an awesome job - everyone knows about DTrace, even if they've never used it. In comparison, bcc/BPF (and many other Linux kernel technologies, like ftrace) have virtually zero marketing budget or professional marketing staff promoting them. Awareness will take the longest to catch up.

I also summarized the current state of bcc/BPF in http://www.brendangregg.com/blog/2016-03-28/linux-bpf-bcc-ro...

... I should add, a tipping point for awareness may be when a commercial performance monitoring company adds BPF support (and latency heat maps, etc), and then promotes this feature. There's a lot of performance monitoring companies out there, many are well funded, and have professional marketing/sales departments.

New account, just to ask this.

I understand how much effort is involved in writing / publishing.

Are you planning a Second edition of your "Systems Performance, Enterprise and the Cloud" text, especially in light of the pace of development in the Linux performance / profiling space?

Thank you for all your countless contributions.

Yes, I'll be writing more books, more details later; as for a 2nd edition, well, the 1st edition is probably more useful in the meantime than you might think. Lets say you install bcc/BPF tomorrow. Ok, now what? The hardest part is to know what to do with these superpowers. Apart from the provided scripts in bcc, the sysperf book has other DTrace examples that you might need, and can then port over. The hardest part of DTrace was knowing what to do with it, so all the prior DTrace documentation will be good material to browse and be inspired from, including my DTrace book. Many of my bcc/tools originated as versions I wrote using DTrace.

I couldn't find a debian repo, and I wonder what to do with the bcc nameclash for the source package. bcc is an existing tiny c compiler in debian.

Reply on news.ycombinator.com