LTTng: An open-source tracing framework for Linux

Thanks, we worked quite hard on this new redesign!

edit: Yes, EfficiOS [1] is the open source business behind the LTTng project :).

[1] - https://www.efficios.com/

In the event LGPL is too restrictive, there's also concurrencykit (http://concurrencykit.org/), which provides similar mechanisms (both for concurrent memory reclamation and various datastructures built on top).

You can also find all the presentations/publications that we did through the years here [1].

[1] - http://lttng.org/publications/

"Excluding proprietary solutions, a few competing software tracers exist for Linux. ftrace is the de facto function tracer of the Linux kernel. strace is able to record all system calls made by a user process. SystemTap is a Linux kernel and user space tracer which uses custom user scripts to produce plain text traces. sysdig also uses scripts, written in Lua, to trace and analyze the Linux kernel.

The main distinctive features of LTTng is that it produces correlated kernel and user space traces, as well as doing so with the lowest overhead amongst other solutions. It produces trace files in the CTF format, an optimized file format for production and analyses of multi-gigabyte data. LTTng is the result of close to 10 years of active development by a community of passionate developers. It is currently available on all major desktop and embedded Linux distributions."

Further down you'll discover that your application(s) are linked with a library provided by the project.

Most of them rely on kprobes [1], or like ftrace on [2]. But there is also oprofile [3], and almighty perf [4], which rely on CPU extensions to implement tracing, (also implemented and by the HW manufacturer as well -- but in a OS-agnostic way).

[1] http://lxr.free-electrons.com/source/Documentation/kprobes.t...

[2] http://lxr.freeelectrons.com/source/Documentation/filesystem...

[3] http://oprofile.sourceforge.net/doc/internals/index.html

[4] https://perf.wiki.kernel.org/index.php/Main_Page

Well, I've never used OProfile myself, so please take this answer with a grain of salt, but from a quick perusal of their website, and as the project's name seems to suggest, OProfile is a sampling profiler, which usually means it will collect data from counters at given intervals of time, producing a number of statistics over time. This generally implies a somewhat significant overhead, and consequent performance drop.

LTTng, on the other hand, is a tracer, which means it collects events from the kernel (using the built-in tracepoint facilities) as they happen. It's also possible to trace userspace apps (with lttng-ust), or define your own tracepoints. This has the benefit of being much more detailed, and also has a much smaller overhead.

You might be interested in reading the "What is Tracing?" section of the lttng docs [0], which does a far better job at explaining this than I do.

[0] - http://lttng.org/docs/#doc-nuts-and-bolts

oprofile has been abandoned, and no longer works on recent distributions. 'perf', I think, is the usual replacement, though the tooling is not as nice.