Hacker News Reader

Comment by JackFr | Hacker News Reader

Liked it better when it was called "Programmers Need To Learn Statistics Or I Will Kill Them All" http://zedshaw.com/essays/programmer_stats.html, but still kind of wrong.

If you need to reduce a distribution to a single number, the most informative number is going to be the mean.

I understand their point about the 99th percentile, but consider that it's possible to improve the 99th percentile measure, while increasing the mean and degrading the performance of all but 1% of the users.

The real issue is reducing a distribution to one number.

View on HN

No, the most appropriate "average" number is the median - the number that is at exactly 50% of the population. At least, that's what most people think about when they hear "average".

For example, most people intuitively perceive the fact that "the majority of drivers consider themselves above average" as human stupidity, however mathematically it makes perfect sense if average == arithmetic mean.

If you are not convinced, consider Bill Gates walking into a room full of college students - suddenly, almost everybody becomes below-average wealthy (if average == mean).

The vast majority of people have an above average number of legs.

The median has its problems too. If you only look at the median page load time, you won't even be aware of huge problems like 40% of your visitors with slow internet connections not being able to load your page at all. If huge effects don't move users across the median, the median won't reflect those effects at all.

To me, it's not clear that there is a "most appropriate" number. All mappings of a distribution into a number will throw information away - the trick is to pick a number that keeps the information you want. And that seems to depend on the context.

The median can be just as misleading as the mean, if the outliers matter. Properly played, roulette can have a positive median profit for the player, but the mean will always converge on a loss.

I like 90th percentile as the "one number" to choose. Mean is misleading, and median (50th percentile) is too low to capture what most of your users are experiencing.

most people intuitively perceive the fact that "the majority of drivers consider themselves above average" as human stupidity

I intuitively perceive it as good drivers being clustered together near the top, with a few outstanding lousy drivers near the bottom.

If you are not convinced, consider Bill Gates walking into a room full of college students - suddenly, almost everybody becomes below-average wealthy (if average == mean).

That's true in my country even if Bill Gates doesn't walk in. Hell, perhaps it's true even in the US.

I meant "below room-average wealthy", i.e. taking only the people in the room into the account.

Invert it. What we really care about is the percent of users who are experiencing a higher-than-tolerable response time, right? Why not measure that instead of using descriptive statistics whose goal is primarily to describe the entire distribution? Do we even care about the variance in response time amongst everybody below the threshold?

You are exactly right. The goal here isn't to optimally describe the entire distribution but to improve the end-user experience for higher-than-tolerable response times. The difference between 80ms and 100ms isn't noticeable. Between 100ms and 1000ms it certainly is.

> If you need to reduce a distribution to a single number, the most informative number is going to be the mean.

No such rule of thumb is ever going to work, you always need to consider context.

In the case where disutility is non-linear with response time, especially if there is a cliff below which differences are irrelevant, an improvement in the response time to the worst decile may well be worth degraded performance for a majority of users. The most useful single statistic in that case could be, not mean, median, or mode, but the percentage of users who fall above the "unacceptable" delay threshold.

Mean, Median, and Mode are all measures of center. Which is appropriate is completely contextual.

If you need to reduce a distribution to a single number, the most informative number is going to be the mean.

Not necessarily. If you want an general-purpose summary statistic that's easy to interpret, for many distributions, the median is a better statistic than the mean. (The pathological case is the Cauchy distribution, which looks like a normal distribution with heavier tails, but has a median but no mean, and both any individual data point and the sample mean approximate the median of the true distribution equally well.)

But yes, unless you have an a priori reason to think that the response is normal, it is worthwhile to look at quantiles, a histogram, or a kernel smoothed density estimate of the data as opposed to a single statistic.

I'd argue that median would be more useful than mean in this case if you had to pick a single number.

But the key takeaway is to consider distributions and to consider confounding factors.

With the proper encoding, I can reduce any distribution you measure to one (sufficiently large) number!

Reply on news.ycombinator.com