All my favorite tracing tools
thume.ca
thume.ca
Thankfully, eytzinger-ordered 4-ary trees work totally fine at 165+ fps, even at 3+ billion functions, but I like to read back through that post once in a while just in case I hit that perf wall someday.
Working on timestamp delta-compression at the moment to pack events into much smaller spaces, and hopefully get to 10 billion in 128 GB RAM sometime soon (at least for native builds of Spall).
Thanks for the kick to keep on pushing!
The other tool I didn’t mention is WinDbg. In my opinion, it’s the greatest debugger on any platform.
That’s a great blog post. Thanks for sharing it.
Missing out on one of the most useful areas for tracing which is time travel debugging. There are a number of interesting solutions there taking advantage of hardware trace, instrumentation, and deterministic replay. Even better when you get full visualization integration so you can do something like zoom in from a multiple minute trace onto a suspicious 200 ns function and then double click on it which will then backstep to that exact point in your program with the full reconstruction of memory at that time so you can debug from that point.
No particularly good publicly visible documentation of the functionality, but it does that and is a publicly purchasable product.
They also had TimeMachine + PathAnalyzer from the early 2000s which was a time travel debug with visualization solution, but they were only about as integrated as most of the solutions you see floating around today.
I just read your post and don't think it would take much to integrate with some of the visualisations you posted about, as a first step.
We've played around in the past with a sampling profiler (code here, requires a copy of our product to be useful though it could easily port to rr): https://github.com/undoio/addons/tree/master/sample_function... which can output in a format understood by Brendan Gregg's flame frames (https://www.brendangregg.com/flamegraphs.html)
But that's not quite the kind of tracing you're talking about. We also built a printf-style interface to our recording files, which seems closer: https://docs.undo.io/PostFailureLogging.html
Something like that but outputting trace events that can be consumed by Perfetto (say) would not be so hard to add. If we considered modifying the core record/replay engine then even more powerful things become possible.
I'm still waiting on the keyserver to be able to run in Kubernetes though
I'm glad you liked it - and that's useful feedback for other demos we give!
> I'm still waiting on the keyserver to be able to run in Kubernetes though
I believe support for that is on the way. If you're already in touch then I imagine you'll get an announcement soon.
If anybody wants to try it, they should get in touch with us.
Our Java tech is based on an underlying record/replay engine that works at the level of machine instructions / syscalls to record the entire process. On top of that we've added the necessary cleverness to show what that means at Java level (so normal source-level debugging works).
That's different to e.g. Chronon, which I think was a pure Java solution: https://blog.jetbrains.com/idea/2014/03/try-chronon-debugger... It had some flexibility (e.g. only record certain classes) but at the cost of quite considerable slowdown and very large storage requirements.
When we need to reconstruct state we always have the option of time travelling the process and re-executing to drill down on the details, though that's only required when you're replaying a recording.
We have some ability to walk data structures and the re-compute the program's behaviour by other means, which I probably shouldn't get into here. I think we could fall back on that more-or-less completely if we couldn't retrieve the bytecode pointer directly.
The fact the JVM introduces Safe Points to help it transition between optimisation levels is quite helpful!
Our original intention was to always fork a copy of the JVM back in time to handle Java debug protocol requests but that turned out to be painful and, thankfully, also unnecessary.
is this something like https://www.reddit.com/r/ruby/comments/15o9hc1/timetraveling... ?
But Time Travel Debugging applies that to everything in the program, not just log statements - all function calls, variables, memory locations, etc can be reconstructed after the fact without having to log them explicitly.
Serious compute bound workloads can run days with a gigabyte of non deterministic event log. Serious IO bound workloads burn it much faster.
For a rule of thumb, think of it consuming a few MB per second, so the length of the time travel is limited by how much of that you can store.
>eBPF and DTrace exist on orthogonal systems
That was true 15 years ago. eBPF and DTrace exist on some of the same systems now, Linux and Windows.
>and most using eBPF have never even used DTrace, let alone "moved on" from it
The performance and tracing groups at Microsoft certainty have. Same with Oracle, Netflix, among others.
>The systems are really quite different, and have different design centers; for the use case of instrumenting the system for purposes of understanding it
True, but unfortunately for DTrace, it is too late. Oracle should have done this years ago. Now Linux has a more powerful tracer builtin, eBPF, and it would be a backwards step to switch the kernel code to DTrace. [0]
[0] From the man that wrote the books: https://news.ycombinator.com/item?id=16377141
Wow. This is some great engineering. Obviously that's what you'd do, but I'd never think of it in a thousand years!
p.s. I think the blog post could use more screengrabs of the traces. Great first pass at it though, and screengrabs can be added over time!
If anyone has any tips on how to trace JavaScript (not just profile by time, but deterministically measure the cost of it in CI), I'd love to hear tips!