Hacker News Reader

Comment by ajb | Hacker News Reader

In statistics, the mean and median are both averages, but are technically quite different. The mean is the sum of all items divided by the number of them, while the median is the middle item in the sorted list. When they are different, this can have strange effects. For example, when estimating engineering tasks, the mean error is often positive while the median is often negative.

This is why your manager believes that engineers are too conservative and always pad estimates, while projects are always late. Indeed, they can learn both these things from observation without realising that they are contradictory. Because humans learn by the most common observation, which is represented by the median, but outcomes can be dominated by the less common, which affect the mean but hardly affect the median.

View on HN

One of the rules of thumb of process management is to reduce the median, rather than the average time taken, because outliers are more likely to have one-off causes and so be harder to fix systematically. In fact, attempts to fix outliers can be a distraction and even net negative.

Interesting. I wonder if that applies in SW dev though. In standard processes each correctly produced item satisfies a customer, but for sw dev individual tasks are internal and customers only care about the final outcome. Also standard processes normally involve Gaussian distributions, so large deviations are rare and can be called outliers, but software estimation has fat tailed or bimodal distributions, so large deviations are part of the standard order of business.

> In statistics, the mean and median are both averages, but are technically quite different.

But numerically, they are not too wildly different! The difference between the mean and the median is at most one standard deviation. If the distribution is unimodal, then the difference is at most sqrt(3/5) standard deviations.

> In statistics, the mean and median are both averages, but are technically quite different. The mean is the sum of all items divided by the number of them, while the median is the middle item in the sorted list.

There was an article posted here, one or two years ago, that explained how mean and median are very similar, as they are the values that minimize residuals or the squared of residuals.

I'm sorry I can't find it right now, I hope some other user knows which article I'm talking about and can link it down here, because it is, in my opinion, an excelent read.

Interesting. I always learned mean/median/mode were different things. The mean is the average, the median is the middle value, and the mode is the largest value. I wasn’t aware of this issue with conflating mean/median.

That being said, I remember learning that and always wondering why I needed to know what “mode” was haha

The mode is the most common value, if it exists.

Whelp that shows you how much I remember from middle school

i found the concept of https://en.wikipedia.org/wiki/Fr%C3%A9chet_mean helpful. it's a generalization of the different kinds of averages and you can express the arithmetic mean, geometric mean, harmonic mean, median or even mode just by swapping out the distance metric in the formula.

If the project is planned according to median, you accept up front that there is 50% chance the project will be late.

It's worse than that. You need the median time for the whole project, which you can't get just from summing the medians of the individual tasks.

There was a class in uni titled Robust Estimation that covers this.

median could be average, but that depends on the probability distribution of your data

Reply on news.ycombinator.com