Using SystemTap

He's going through the same exercise I went through recently where your customer is running your software on Linux and you want to try to make use of the facilities offered by that OS so he can reproduce and diagnose the same issues.

The problem is that Linux just doesn't have good diagnostic tools. Most people don't recognize it because they've never had good tools. It's like being a C programmer who's never heard of a debugger. You can keep adding stuff to your code to try to isolate a bug and keep rerunning tests hoping it hits it where you've instrumented. That seems to be fine. But then you get a debugger and you can suddenly do significantly more on a running system gathering even more information on exactly the part that you care about.

This is what dtrace is. Linux needs something like it. If SystemTap will be that someday, then great. It has to work, though. My attempts to get it functional on Ubuntu and debian left me not feeling too great about the current state of thing (at least on debian, you can get kernel probes in without adding new and poorly documented package repository locations in). In neither case could I get SDTs working, though. For example, with dtrace, I can pretty trivially tell you the exact code paths in any running application on your system that are leading to IOPS. Then maybe I can exclude "known" paths and focus on only areas we don't think there should be any. This will include, for example, paging information. I can decide whether I want that when I'm drilling down. I can then look at another process that's running on the system and see when one is competing with the other for IOPS, CPU time, etc... I do this while the software is running and with minimal impact to the performance of the system.

When SystemTap does these things on my customer's Linux boxes, then I'd call it a dtrace competitor.