strace is the Daddy of debugging especially when you can attach it to an already existing program using it's PID. I have solved so many permissions errors problems with strace where the application itself just failed without logging anything.
For many of the systems applications I design for Illumos, I've used DTrace probes as a way of logging very frequent events, on demand (to avoid frequent IO). All of the events that _must_ be logged for the application to function properly, are logged and fsync'd.
I think that in most systems dynamic tracing will eventually replace a significant portion of the logging functionality that people code into their applications.
Either way, if you think strace is sweet, give DTrace a spin.
Some helpful links:
[0] A video demostration of DTrace, by the creator of DTrace.
[0] http://www.youtube.com/watch?v=6chLw2aodYQ
[1] A post I wrote, demonstrating DTrace. Similar posts can be found on the blog's dtrace-addict page.
[1] http://nickziv.wordpress.com/2011/04/08/adventures-of-a-dtra...
[2] A wiki that contains DTrace examples for various languages and system facilities. Some examples may be Illumos-centric.
[2] http://www.solarisinternals.com/wiki/index.php/DTrace_Topics
[3] DTrace's home. Contains blogs by the engineers behind DTrace.
UPDATE: Meant to reply to parent's parent.
On one system, I couldn't even invoke kernel functions.
On another system, the kernel panicked as soon as I executed `stap -e ...`.
Basically unusable, at this point.
I hope it improves because developers on linux could really benefit from dynamic tracing.
also, in developer > applications there is an app called 'Instruments' which is an excellent GUI front-end to DTrace