"Tracer bullets". Have a flag that can be put on an action that causes it to be logged very verbosely. In a way that travels through your system. And then aggregate those logs, process them, and make them available. Then randomly tag a small fraction of actions with that flag. Like 0.1%. Because you do this relatively rarely, you can make verbose logs without overwhelming your system.

When you receive a complaint about foo occasionally being slow your first stop is to see if you've got a concrete action that was slow, look at the picture from the logs, and try to figure out why. This will greatly help in reproducing rare problems or, if you can't, at least identifying where they come from deep in your system.