> The ability to monitor go programs with a very high sampling frequency — up to 10s of microseconds
I'm curious about the overhead of this and will probably try to measure. This work certainly overcomes the limitations of setitimer(2) for sampling rate, but faster sampling is going to increase the performance overhead significantly.
In particular Go's built-in stack unwinding APIs use gopclntab (which is Go's Plan9 inspired version of eh_frame). This is rather slow compared to frame pointer unwinding (up to 55x in my testing). I think on average you can expect to spend at least 1 microsecond on unwinding, and then some additional time hashing the stack and updating the in-memory profile (perhaps another microsecond? I haven't measured this part yet). I've started writing some of this up here [1] but it's still WIP.
Anyway, based on the above, sampling every 10usec will likely slow the target program by ~20%. This is probably a very optimistic estimate because it doesn't take into account all the CPU cache thrashing that will happen.
As I said, I'll try to take a closer look later, but if anybody has some insights/comments here I'd be very interested.
[1] https://github.com/DataDog/go-profiler-notes/blob/main/stack...
Disclaimer: I work at Datadog on Continuous Go Profiling, but I'm here speaking for myself.