Perforator – Open-source tool for cluster-wide continuous profiling

Repo: https://github.com/yandex/perforator

We use this for GWP-style[0] continous-profiling over O(10k) nodes.

It does sampling through an eBPF program in the kernel, and supports generating a bunch of different output formats (classical flamegraphs, but also things like PGO profiles). One example where we use it daily is flamegraph diffs for prestable releases of services.

It's also worth mentioning that while the Github repo has an example for a Kubernetes setup, this isn't a Kubernetes-specific tool. We don't actually run Kubernetes ourselves. It works wherever (well, if you have Linux, that is!)

(bias note: I work at Yandex, but not in the Perforator team)

[0]: https://research.google/pubs/google-wide-profiling-a-continu...