I worked at a place doing network appliance stuff. We managed and developed the machines, and product to end users. A simple dashboard increased our effectiveness by an order of magnitude. Some simple graphs and deep knowledge of the code really really helped. Some data from other departments combined with this was amazing. A simple color coded call queue count told us there was a problem before the front-line tech support people could even report it. Simple analysis of that queue told us where to look for problems: a predominance of one area code told us to examine specific machines (and which subset!), while a wide array of area codes told us that it was probably in billing or comms infrastructure. A couple of graphs of network traffic and system load told me what what the problem could be. I don't know how, I couldn't write code to do it, but I certainly could get to the problem much faster with that minimal input from graphs.

My working theory on dashboards is this: the human mind is amazing at pattern recognition. We're wired for it. We can make intuitive leaps see patterns that may be very hard to describe in math/stats or code. Particularly visual ones. So if you provide that data to your brain to crunch, you are enabling and augmenting your natural tooling. The graphs and stats should be as specific an pre-thought out as possible, but they aren't perfect. Fortunately, as time progresses you learn what "looks right" and what "looks like a problem in subsystem Foo".

This isn't a silver bullet, but certainly it is a great tool. Since then, I've tried to never do work without some sort of visual feedback I can background my innate pattern matching on. Even if it is just scrolling logs -- these patterns emerge and provide clues even if you can't express what they are.

(Anecdote: I had built a demo a while back, and it hiccuped during the live presentation. Fortunately I was in the back of the room with my logs scrolling, and I noticed the logs looked wrong, so I found out a script had died. I restarted it, causing a weird blip in one of our display graphs, but the presenter noticed it before calling attention to that graph in the course of presentation. He glanced at me and I gave him the thumbs up and the audience never even noticed. What was the pattern? The scrolling in one of my log windows slowed down....)