Comment by sardonicbryan | Hacker News Reader

Comment by sardonicbryan | Hacker News Reader

Parent Full thread

So I built and use a realtime analytics dashboard that tracks revenue, projected revenue, revenue by hour for a portfolio of social games. I find it incredibly useful, but I will give a couple tips that address some of the issues in the article:

1) You have to provide context for everything. Current real time revenue is presented right next to the 14 day average revenue up to that point in time, and also how many standard deviations the delta between the two is. Ie: Current revenue is $100 at 10am, vs. 14 day average of $90, which is 0.2 standard deviations of revenue at that time.

2) Hourly revenue is presented the same way, right next to the 14 day average revenue for that hour and the SD delta.

3) Look at it a lot. I've been looking at this sheet regularly for over a year now, and I have a really good feel/instinct for what a normal revenue swing is, and an even better feel for the impact of different features/content/events/promotions on our revenue.

4) This approach also works better when the impact of your releases is high. A big release typically spikes revenue 2-3 SD above baseline, and causes an immediate and highly visible effect. So while I'm not strictly testing for statistical significance, it's one of those things where it's pretty obvious.

5) It also works better if you use it in conjunction with other metrics. We validate insights/intuitions gained from looking at realtime data against weekly cohorted metrics for the last several months of cohorts.

While this sounds very cool, and my inner geek would love this dashboard on the wall of the office - is it actually useful or is it a distraction?

What actions or decisions would you make within minutes of seeing the results? If product changes take days or weeks, daily analytics is just as useful, and stops people wasting time on looking at the data more than once per day.

This is precisely the right point. We look at an analytic display to determine if action should be taken, and what those actions should be.

The vast majority of "big data" is noise, more than effectively summarized with basic statistics.

There are three major classes of decisions that get made from this. In descending order:

1) We generally have pretty tight revenue targets, so one thing I and my teams do now is forecast out expected revenue on a daily basis for the next 30 days, as revenue is highly dependent on a stream of new content, features and what we call live ops.

Live ops are basically a combination of promotions, contests, events and tournaments. We have decent tools, so we can generally spin up a live op in a matter of minutes to a couple hours without doing an engineering push. We also have content we can release with a few hours of work, so if today is not going to go well, it gives us enough time to prep something for tomorrow.

It's highly useful to know if a Live Op or new feature/content release is "working" or not. With the real time dashboard and experience, I can know with high confidence whether it's working within 30 minutes of release, which gives me more than enough time to spin something else up so that we don't lose the day.

2) On many days, we actually do multiple releases or live ops in a single game -- the pace of releases is fast even compared to other web companies. As unscientific as this is, it's often easier for me to evaluate/get a feel for the impact of a release/live op based on how much it moves the projection in the first hour after release, and before the next thing comes along.

Obviously with stuff like that, you validate after the fact against more full data, but I've found after looking at this lot for over a year, I have a strong "feel" for what's working and what's not.

3) Related to 1) and 2), but I've found that using the tool gives me a much tighter feedback loop. When I make a feature release or live op and I'm sitting on the dashboard watching the release move or not move the projection, I have a much tighter feel on the impact. Most of my peers who don't have something like this often are not as rigorous in understanding and evaluating the different levers they have to impact revenue.

Reply on news.ycombinator.com