milianw

I still think we should find a way to integrate the two somehow - i.e. analyze locally and then send the pre-analyzed data for visualization purposes to the remote firefox profiler. Does anyone know a good format we could use for that purpose? It needs to be non-verbose to not hit the infamous 2GB/4GB JSON heap size limitation on import e.g. Similarly, we also need something that can deal with the various cost types we support in hotspot, most notably off-CPU time.

You are welcome :)

And to people using other IDE/editors - you can configure which one gets opened when you click on a source line from the hotspot settings. QtCreator is just the default (when that is installed).

initially yes but then the local power supplier will come and install a bidirectional meter

C++/Qt GUI applications (embedded, desktop), mostly for a large customer in the medical sector, i.e. microscopy/imaging/cancer research.

True, that's another good point. But again, this reasoning is very different to the one from the linked article and website - if you have oprofile or valgrind's cachegrind available, you clearly could get perf setup instead.

I'm not debating that manual GDB sampling has its place and value. I'm debating that perf is "lying" or that it's impossible to get hold of off-CPU samples, or profiling of multithreaded code in general.

perf is easily available through yocto and buildroot (and probably other embedded linux image builders). hotspot can be downloaded as an appimage. It should not take 30-60min to set this up, but granted, learning the tools the first time always has some cost.

Furthermore, note how your reasoning is quite different from what the website you linked to says - it basically says "there are no good tools" (which is untrue) whereas you are saying "manual GDB sampling might be good enough and is easier to setup than a good tool" (which is certainly true).

The premise of this website and articles like https://yosefk.com/blog/how-profilers-lie-the-cases-of-gprof... just show that the authors are using the wrong tools. It is nowadays relatively easy to also look at off-CPU time when profiling with perf (e.g. https://github.com/KDAB/hotspot/?tab=readme-ov-file#off-cpu-...). The idea is to use sampling for the on-CPU periods and then combine that with the off-CPU time measured between context switches. VTune also supported this mode for many years.

With `-z` (zstd compression) you can bring down the disk-space cost of dwarf unwinding by a factor of ~100 based on my personal experience.

to GP: What you describe sounds like https://github.com/koute/not-perf to me

I don't know exactly what these BSD things did, but there is a super easy way nowadays to get the stack for any process:

    eu-stack -i -p $(pidof ...)

Thanks to debuginfod this will even give you good backtraces right away (at the cost of some initial delay to load the data from the web, consecutive runs are fast). If you get a "permission denied" error, you probably need to tweak kernel.yama.ptrace_scope=0

Sad to see people are still unaware of modern perf profilers like hotspot, tracy, vtune, ...

they are easy to setup, work extremely efficient and can nowadays also catch off-cpu time. ctrl+c debugging finds you trivial stuff, but dismissing the real tools as unusable or inferior is simply ignorant.

> Unlike perf, gdb will give you a callstack even if the program was compiled without frame pointer support.

There is `--call-graph dwarf` and it exists since many years, and with `-z` it is pretty efficient and just works too - unless the stack gets too long, but even then it's good enough for profiling purposes...

> Also, sampling profilers are bad at tail latency - if something is usually fast but occasionally slow, you won’t be there to Ctrl-C it when it’s slow.

Very true, thankfully the flight recorder modes for profilers can help with that to a certain degree, with a potentially large sampling frequency.

I managed to read some books rented via the Berlin library system (voebb onleihe) using this software, really great work!

MTuner exists for Windows which is similar but different. Porting heaptrack is not straight forward, as it heavily relies on libraries that do not work on Windows or do not support Windows executables and debugging information, such as libunwind and elfutils.

IIUC then you might be looking for https://github.com/facebookexperimental/object-introspection - very new and not yet easily usable outside of the Meta sphere, but goes into that direction, or?

So more something like dhat from valgrind? see https://valgrind.org/docs/manual/dh-manual.html ?

Heaptrack can handle millions of allocations per second, and thanks to the deduplication built into the trace file format combined with zstd compression, the overhead is pretty manageable. It's pretty hard to get "hundreds of GB" of data recorded.

Another thing: you can always just runtime attach and profile a partial time to reduce the amount of data recorded.

Finally: if your tools do so many allocations, maybe you should consider optimizing them...

What is not decent about heaptrack or similar tools?

I applaud this initiative. yet I wonder - the ideal situation would be to not throw away the debug info (ever), but rather it should be put into an external file. even Linux supports this for many years and nowadays we even have direct support with dwarf5/dwo. is trust taken any action towards that direction?

after all: without debug info (which you do not want to ship to customers necessarily), you cannot do profiling or debugging in any meaningful way...

well said! doing large scale refactorings or invasive performance optimizations without a sound testing infrastructure is a huge pain. the more often you run into this, the more you'll appreciate tests going forward. but as always, one must not forget the pareto principle and related mantras - don't aim for 100% test coverage, roughly 80% coverage in 20% of the time will often give the biggest ROI.

I'm very happy about the port to elfutils, it's such a great project!

sadly there's also a lot of QNX :(

The Qt framework has excellent documentation which made it very easy for me to learn it back then!

author here, happy to answer any questions you may have about heaptrack :)

I still remember optimizing customer code bases that suffered from horrible startup performance because they instantiated thousands of debug rects unconditionally all the time...

You should look into GammaRay which allows you to such debugging without having to change your code. You can literally just pick the element on the screen. See: https://github.com/KDAB/GammaRay/

No, but you can use the docker script I use to generate the AppImage as a basis:

https://github.com/KDE/heaptrack/blob/master/tools/Dockerfil...