One of my favorite conference talks is semi related to problems like this. https://www.youtube.com/watch?v=fE2KDzZaxvE Brian Cantrill - Zebras all the way down