Profiling with Ctrl-C
yosefk.com
yosefk.com
You can fix a lot of stupid problems that way. (And most problems are stupid.) Yes, yes, a real profiler would be better, but if you don't have the fancy tools because your employer doesn't buy you such things, and it's a primitive and cruddy embedded system so there's no obvious better way to do it, and you built this horrible hack right now and... hey, the hack solved the problem, and what do you know? it keeps on solving things....
As someone who wrote several profilers for a living... that is a real profiler.
Did the same, but added a stack canary to tell if I overflowed, and wrote my call-stack results to a hardcoded address at the very end of memory (last 32 bytes, IIRC). When the chip reset, in addition to the reset flags (brownout, etc), I could peek that memory to see what the last addresses in the callstack were.
Helped immensely in figuring out a transient bug (device resets) which was hard to repro.
but my motivation there isn't to keep people from calling me an idiot. that's a lost cause. it's to save me and them time rediscovering problems i already know about
I've made a thing[2] that can display that within a visual timeline (interpolated between ticks of the nearest syscalls/events, which do have known real time), essentially giving a sampling flamegraph that can be arbitrarily zoomed-in, with the ability to interact with it at any point in gdb.
Though this is not without its issues - rr's ticks count retired conditional branches, so a loop with a 200-instruction body takes up the same amount of ticks, and thus visual space, as one with a 5-instruction body; and of course more low-level things like mispredicts/stalls/IPC are entirely lost.
Will try it next time when I have such issue. Thank you!
to GP: What you describe sounds like https://github.com/koute/not-perf to me
eu-stack -i -p $(pidof ...)
from your reference to kernel.yama.ptrace_scope (and your apparent belief that bsd belongs to the distant past) i infer that eu-stack is a linux thing? this looks pretty awesome, thanks for the tip!
https://stackoverflow.com/questions/12394935/getting-stacktr...
lol, it's a story as old as time. The infinite loop of ego entrenched developers not wanting to change something out of some trivial inconsequential disagreement. The bike shed will be built my way!
Sometimes I do so but add something to the stderr output referencing the issue number I'm compensating for - that has a surprisingly good rate of getting somebody who knows what they're doing looking at the issue in the other project and submitting a patch.
Most of the time I don't event need to reach for a profiler proper.
My guess would be that it's because tail-call optimisation only happens in -O2 and above.
Parsing recursively is frequently the cleanest way to implement a parser of tree-structured input, after all.
If you're doing anything recursively, it makes sense to slightly restructure the recursive call to be the last call in the scope, so that TCO can be applied.
There's also https://github.com/pietro/gcc-musttail-plugin to ensure it does happen (and clang has musttail support built in these days).
I think, firstly, that spending 15s trying the CTRL-c approach is a worthwhile tradeoff. If you don't find anything, then sure, spend another 30m - 60m setting up perf, KDAB, etc. Maybe more if you're on an embedded device.
Secondly, the author seems to say that he's used this on embedded devices with no output but a serial line for the debugger. This is also a 15s effort[1].
It's basically a very low effort task, takes seconds to determine if it worked or not, and if it doesn't work you've only lost a few seconds.
[1] I'm assuming that if you're developing on a device supporting a serial GDB connection, you've already got the debugger working.
Furthermore, note how your reasoning is quite different from what the website you linked to says - it basically says "there are no good tools" (which is untrue) whereas you are saying "manual GDB sampling might be good enough and is easier to setup than a good tool" (which is certainly true).
but they all support gdb
I'm not debating that manual GDB sampling has its place and value. I'm debating that perf is "lying" or that it's impossible to get hold of off-CPU samples, or profiling of multithreaded code in general.