All Dashboards Should be Feeds

Let me propose a maturity scale and suggest where things are...

Level 1) Are you recording metrics?

That's the first basic level of maturity, without recording data you cannot have data from which to support any decisions. Without data you are blind. The assumption here is that you are able to record everything that you need to... recording 3 metrics isn't enough, you need to be able to get to all your data and record it over time.

Level 2) Are you able to visualise your raw data?

This is the basic dashboard. Whether you build something pretty or use an existing software package it's just the ability to see the data that you are collecting. This relies on people to figure out the inter-connectedness of data and to make inferences from it. Basic graphs and trend analysis fit in here.

Level 3) Are you able to visualise insights in your data?

This is about changing your dashboard or reports such that you can start to consider metrics that are abstractions from the raw data but important to you. Think of cohort charts, revenue relationships to user behaviour, UX funnels, engagement charts, etc.

Level 4) Are you able to record exceptions in your reporting?

As in, can you create thresholds against a metric such that if the metric goes above or below the threshold an event is triggered to record it. Exception tracking coupled with the insights in level 3 should give you the ability to do things like release a new feature and see the resulting effect on certain metrics and be alerted when those effects are outside of some normal range.

Level 5) Are you able to visualise your exceptional events?

And here we are at the newsfeed of data, the events which stand-out are highlighted and all of the other noise drops away. This is great for producing a list of things you might want to know or react to... but you can't get here without going through (and still having) levels 1-4 and it's important to understand that different people in your organisation still need all those prior levels (try telling your sysadmin that he can't have log files or munin and that instead he's got a news feed).

The article is comparing fundamentally different things, which serve different purposes.

Charts and gauges display "raw data". The downside of this approach is that a human has to process that data to turn it into useful information. The upside is that humans are _very_ good at this.

The feed displays higher level information extracted from the raw data. On the upside, you don't have to have someone sitting around watching it. The downsides are that it requires a higher up-front investment to program or train the system to extract this information, and the system will never (given current technology) be as good as a human for any substantial correlation.

The correct balance depends on your situation (resources, uptime requirements, implementation details of your system), but dashboards displaying raw or summary data that enable you to see unusual patterns, monitoring (which is very similar to a "feed"), and comprehensive stats gathering (which you won't look at on a day to day basis, but is tremendously powerful when diagnosing issues) are all tools that any team should be looking at using.

Please Both. You need to be able to see real-time and long-term.

All Dashboards should show goals you set or something that's unusual compared to your past data. THAT is the reason why everyone's confused by most dashboards - they're visualization for the sake of looking good first. When they're done right they let you see the state of things, whereas feeds are like alarms - too many or too few can make you stop giving a damn.

Be like a scientist - ask every question you can think of. Then try to answer it with data and if that answer changes over time, keep it on a dashboard. Then set alerts and snapshots to the feed, so only what's really important will bother you.

What we need to work on is how deep you can ask questions of an external service. Giving me my choice of 12 pre-baked table columns or API endpoints isn't going to answer my real goals. 12 pre-designed draggable widgets aren't either. I want to ask big questions first instead of combining little answers.

This is my dream for Torbit (http://torbit.com). We currently show performance data in a typical dashboard, but I wish we had a newsfeed as well. We're playing with some ideas, but it turns out to be a really hard problem. I'd love to show a newsfeed with stories that say "your site is 5% faster because you changed this one JavaScript file" or "your site is 20% slower than usual right now due to a spike in IE6 usage in China". Those stories are simple for anyone to understand, but really tough to generate without a ton of false-positives. You have to account for daily, weekly or other seasonal changes in traffic patterns, even ones you wouldn't have an easy way to predict (like a commercial break during the Eastenders). You have to know what's typical for each site so you can detect anomalies and also be smart enough not to set off alarms when you get 5 visitors from Liechtenstein instead of 1. You have to wrestle with correlation vs. causation. It's a fun challenge & anyone who's interested in tackling it should check out http://torbit.com/jobs. :)

I really liked the idea behind a product called Trendly (unfortunately it got shut down because Twitter bought the team).

They had some clever algorithms that simplified time series data from google analytics so that you could see at what points real changes occurred.

Some screenshots here: http://analytics.blogspot.com.au/2009/08/api-integration-to-...

You can see how it's able to take something noisy like time series data and turn it into a stream of discrete events. It's really quite clever.

There is a tried and true method of getting to the front-page of HN, which is to take whatever implementation choice you've made today and make broad, completely unsupported claims that it is universally applicable as a law of the universe. This applies to a shockingly high number of non-news submissions, regardless of a lack of rigor to the claims.

Except this one http://ts4.mm.bing.net/th?id=H.4680604679800715&pid=1.7

Now, now.. comparing two very different-intent dashboards isn't cool. I know I'm late by 10 hours because people have already vented their feelings about the writeup.

You can partly agree that dashboards should be smart but not every dashboard _needs_ to be so. Creating smart dashboards requires you to actually identify the metrics and the requirements of the end-user.

When you generalize this, you end up having a dashboard that's Google Analytics. It's gigantic and by letting the user decide what labels he wishes to give to the numbers, it's actually way more smarter than the 'feeds.'

That's just my humble opinion but I sure don't feel comfortable about how other dashboards are treated in this piece.

And if someone's interested: http://thisiscsr.com/post/40088895750/dashboards-smartness

What I took away from this, which, reading the other comments, is not at all what everyone got out of it, is that a bunch of graphs can be a great, but a system that automatically makes recommendations based on those graphs (taking some of the decision-making out of the equation) is closer to ideal.

I might be projecting, though. I talk regularly with the people behind Pirate Metrics (http://piratemetrics), which does that sort of thing. (Records, displays, but makes suggestions as well.) S, that sort of idea has been on my mind.

I've been working on my own stats tracker in my spare time (it's gradually getting there, and aside from accidentally DOSing Pusher, it's going quite well), and I've taken a mixed approach to the site dashboard. It has the usual counters and maps around the outside, with a stream of activity in the centre which shows how users are progressing through your site.

This seemed intuitive to me, and it's kind of a combination of ideas I've taken from other stat tracking applications like gaug.es, Mixpanel, and Chartbeat. Each application seemed to be missing one or the other.

Charts, graphs etc are interfaces to data. Boil those down to a language we are more used to using, for example, English, and it becomes easier to interpret the data.

If the best interface is no interface, then here "language", a very common and familiar interface, approximates "the best interface".

All dashboards should be widgets. A widget could contain a feed. But the most important part about a dashboard, is giving the end user the flexibility to add/filter whatever information he desires to see in my opinion.

Show live updates, but allow users to also see older data.

Right. Charts should have meaning and this meaning should be easily readable. We tried to make twitter-like comments in charts (http://explainum.com) but it didn't gain any popularity. Perhaps there is a better approach.

One of the key benefits I get form a dashboard is being able to recognize that something's wrong, even when it's entirely different than anything I've seen on that dashboard before.

I'm not convinced a feed would allow me to do that.