Hacker News Reader

Comment by ChuckMcM | Hacker News Reader

Nice post. Ops guys though, like to see the bushes rustling right away so that we can reboot that switch before all hell breaks loose :-)

The central theme is a good one though, tactics or strategies have an innate timeline associated with them, and deciding on tactics or strategies with data that doesn't have a similar timeline leads to poor decisions. The coin flip example in the article is a great one.

Ideally one could say "What is the shortest interval of coin flips I can measure to 'accurately' determine a fair coin?" And realize that accuracy asymptotically approaches 100%. One of the things that separate experienced people from inexperienced ones are having lived through a number of these 'collect-analyze-decide' cycles and getting a feel for how much data is enough.

View on HN

> Nice post. Ops guys though, like to see the bushes rustling right away so that we can reboot that switch before all hell breaks loose :-)

I don't think you're disagreeing with the article. He shows a real-time dashboard of ops stats, and then says:

> It's important to divorce the concepts of operational metrics and product analytics.

Real-time ops statistics help solve ops crises, but product analytics can afford a little more consideration.

> It's important to divorce the concepts of operational metrics and product analytics.

I don't think operational and product analytics are different in principle. In both cases, the best speed for analytics matches the speed of taking action on the results.

In ops, there are people whose very job is to respond to a server crash in 2 minutes. In product design, there are no actions taken within minutes of seeing the change in data - so faster analytics just wastes time on refresh-procrastination, and encourages needlessly hasty decisions.

One of the things that separate experienced people from inexperienced ones are having lived through a number of these 'collect-analyze-decide' cycles and getting a feel for how much data is enough.

If you're going off of "feel" instead of statistics then you're doing it wrong. Period.

>If you're going off of "feel" instead of statistics then you're doing it wrong. Period.

I disagree, at least to a point. Take an experienced Operations/monitoring guy who's been around the block more than once, then sit him in front of the monitoring utilities for a new company developing a new service.

Then, take a total newbie and put him in the same place.

Train them both to equal skill on your tools and operations.

Who do you think will make the most proper calls? Why?

At this point, those statistics and that documentation do not exist yet. What constitutes a "false positive" vs a "drop everything and spin up more VMs and get on the load balancer" can be more of an art than a science, especially when you're first starting out.

As hokey as this sounds, certain systems have a "personality" that varies between installations and companies, that nothing short of day to day use will educate one in.

For operations, I agree. A lot of the numbers you have don't have a rigorous statistical interpretation - for instance is a load average of 20 fine or a problem? Depends if you're looking at the Oracle database.

But the original article was talking about A/B testing, and that is the context that I was thinking about. There you both can and should use statistics.

It's possible (though deceptively difficult) to train one's statistical intuitions to be pretty accurate, or at least accurate enough to give them a positive EV. I was a professional poker player for several years, and this is what becoming a strong player is all about.

There's a lot of intuition and experience that is required to be a talented statistician; once the test has begun things are pretty much locked down (ie, if you know the math there is only one "right" answer from the data). The intuition and experience side of things comes with how the test itself is setup. Are things properly controlled, are you collecting the right data for what you're testing, is the data distributed properly for the test statistic you're using...the list goes on and on and on and on and on.

There are plenty of places that require intuition and these are the places that errors are often introduced. Statistics is an art.

You can build a mathematical intuition. This is a valid thing. You just know how to dig down and prove you are right if asked, so it's ok to leave out details.

Of course you are, my comment was that when you are presented with a decision and the data that was collected and analyzed that went into it, if you've seen a few of these you are much better at catching problems.

If you are the one doing the collecting and analyzing and you just think it feels like enough data but you can't actually reason to that point, then yes, you are doing it wrong.

I always thought statistics existed to give you the "feel". Statistics of more statistics is nonsense!

Reply on news.ycombinator.com