Hacker News Reader

Comment by nhaehnle | Hacker News Reader

There is actually a nugget of thought in the article beyond the explanations you always hear (and which make up all the comments in this thread as I'm writing this): Even if you base your project schedules on accurate estimates of the median time required for each step, you will still be behind schedule for most projects.

The reason is actually quite simple: when things go well, they can only go so well, but when they go bad, they can go really bad.

Or, in a more quantitative way: while each step of the project will be equally likely to take longer or shorter than median, the steps that take longer can take much longer, while the steps that go faster only have a limited potential for balancing out the delays.

View on HN

According to this reasoning you should set targets based on expected time, not "median" time. You're saying the tail of "completion time" is long, and so the median is a poor estimator for expected time. But if you actually knew the distribution of times for task completion (which is implied if you know the 'median time'), you could simply use the expected value of this distribution and your projects would then tend to complete on time.

I don't think this is the correct explanation. I think it's far more likely that (1) people don't know the true distribution of task times, so estimates are just crap guesses based on hubris, what managers want to hear, etc., and (2) scope creep.

Exactly. All this article is really saying is "most of the time, adding together median expected times underestimates the sum of mean expected times". Well then, easy fix: add together the mean expected times off the bat.

Bingo. When we think about the steps intuitively, we tend to gravitate toward the median. "How long does this usually take?" is asking about a median in surprisingly precise terms.

Instead we have to understand the distribution, and the mean as well, and add up how long it takes on average, rather than just in the most numerous case.

Further, and perhaps more importantly, by understanding each distribution, you have far greater knowledge about what the expected time will be, and where the delays are most probable; and you can begin to start analyzing where those delays might be coming from in the interactions of all the parts of the system. This is straight-up quality talk right out of W. Edwards Deming, and it's how you start to improve how you produce in general. Good stuff.

There's more to it than your easy fix, the article alludes to it but does a poor job of covering it: if you know the distributions of the intermediate steps, but you only sum the means of the steps, then you are throwing away data that could help you calculate a better estimate.

Oh of course! More data (sanely applied) is almost always going to result in more accuracy. I just found it very strange that the core assumption of the article -- that the default setting is to add median times of subtasks -- was never questioned. Why was that the chosen method to begin with? Most time estimating tools (Pivotal Tracker, etc.) work with averages, not medians, for precisely this reason.

The problem with high estimates, if accepted by the stakeholders, is the risk of over-engineering. When given lots of time, development teams often will find ways to use that time which don't add to the value of the features being delivered: over-testing, making things "re-usable", a chance to try something new. And the project is late anyway.

Maybe optimistic estimates are a motivating factor for teams. The desire to finish faster, to be more efficient than you were before, a commitment with a challenge.

In my experience, it takes a strong product owner and a mature development team to meet deadlines. Shipping on-time is always a game of tradeoffs. Accuracy in estimates is usually the result of doing things in a known, measurable way. And by keeping the stakeholders close, you can make critical decisions together to keep a project on track.

And account for correlations. It would be interesting to see if multi step projects with overruns tend to overrun in more steps then expected. I could imagine a stakeholder management heavy project to be more likely to overrun in every step. In other words, even by summing the distributions you might be underestimating the expected time of completion.

Another reason is a kind of selection bias: an organisation wants something built, gets offers from various software companies, and picks the cheapest.

That is, if you are doing the project, your company's initial estimate was the one that had the highest chance of being too low.

That's not only that, the field is full of bullshit. Advertising feature as available when they have not even been discussed seriously is the usual in the companies I have worked for the last 10 years. Our website and sales all go advertising "We support X. We are ready for regulation Y", while we are actually waiting for a paying customer to buy it to even look into it.

And that's all companies. You can't really be honest as all your competitors are similarly bullshitting. ( I used to be upset about that thinking I was always working for the black sheep, then over the years, you always end up working with your competitor one way or another and you find out it is the same everywhere )

There is also the deadline game. The client will push for earlier and earlier release date. The provider will accept because the provider knows that the client will not be able to test the product. I used to be upset to deliver code that would not even compile. Then over the years, we have had client not ready to test for several years. An extreme case, is a client that took a package I developed 5 years after delivering the working version of it.

That's the biggest problem I have had with Agile. Very often companies are not ready to support the lack of bullshit even internally - no more schrodinger status, no creative budget allocation - developer appear to be slower and cost more.

> An extreme case, is a client that took a package I developed 5 years after delivering the working version of it.

You completely lost me there, what happened exactly?

I think he means (and that is also my experience in running services companies) that even if you deliver the software to your client, the client does not have the time scheduled internally to actually test and report issues. So you delivered all the client wanted and they will not even look at it because the internal manager who issues the project and his team are too busy.

We delivered a project a few weeks ago and heard nothing; I heard only yesterday that the manager went on holiday and will be back 3rd week of january. And when he comes back his inbox will be full so I do not expect any testing till the 2nd week of feb...

A client wanted to run our application in windows at the latest, a week before I joined the 2 men project. 2 months (?) later we delivered a "patch" (the first version didn't even work at all) of the initial solution using the same unix scripts running in interix.

Seems like they were not in any hurry after all. Some 5 years later, they contacted my company again asking how to install it in their test environment. They were apparently not happy that after 5 years, the solution was still a bodge solution using interix rather than a proper port in windows. I don't know what happened from there, I had moved on to another project right after that delivery, could not remember anything and to be honest only painful memories could come back from that shitty codebase. ( the company was making something like 5K a year gross from that application )

if it's a long tailed distribution knowing the expected value isn't going to tell you anything you can count on

If you're working with censored data, it's possible to know the median but not the mean of your data.

https://en.wikipedia.org/wiki/Censoring_(statistics)

I think what he/she's saying is you can't predict how badly something can go (since it can go infinitely badly), you can only predict how well it can go (because it actually touches the finite minimum). Which passes the logical sniff test and is in fact the correct explanation. At least it has been for absolutely every project I've ever been on.

You could feasibly be on a project that could last until the heat death of the universe.

This is very true. The best way I have found to combat it is by splitting up the tasks into many smaller pieces. If you have a task that is scheduled to take a month, then it can easily go out to 10 months in the worst case. If you have a task that is scheduled to take a day, it can easily go out to 10 days in the worst case. This allows you to make corrections to your schedule.

There are 2 other important issues. First, you need to update your schedule based on the amount of time tasks are actually taking. If you only have 10 tasks (each taking a month), the error bars can be quite significant. But if you have 200 tasks (each taking a day), the mean completion time will have quite small error bars. So if you keep a rolling average (say over the last 30 tasks), you can have a fairly good estimate for completion time (agile developers will notice that this is "velocity").

The second very important issue is to be completely anal about your definition of "done" and about making sure that the completed tasks actually meet that definition. Your mean completion time will only be useful if you are measuring the time to completion accurately (obviously).

This effect is so powerful that I recommend "same sizing" tasks and planning everything to have a completion time of somewhere between 1 and 2 days.

There is actually one last thing you need to do. Requirements discovery never happens completely before you start development. As you write the code, you discover new things to do. It is obviously important to modify your plan to accommodate that new information. If you don't, you will end up building something that nobody needs. However, the amount of new work seems to be predictable. I made graphs of new work added to projects over a couple of years and it appears that the growth of new requirements is very similar to some of the defect discovery models (for example Littlewood). Just making a graph of new stories/tasks added over time will give you a decent idea, but I have found that a rule of thumb of adding 30% or so (over the whole project) for new requirements seems to work well.

There's another hidden aspect: Be aware of dependencies.

When you have a series of dependencies (A -> B -> C) stacked against each other, once A is late, B and C are almost guaranteed to be late. If B is late too, C suffers even more and has little chance of being on time.

If you can lay out the tasks so that dependencies have slack between them so that lateness can be absorbed without shifting later tasks, you are more likely chance to hit deadlines and potentially complete the project on time. Or in more formal terms, track the Critical Path.

In practical terms, I would see this all the time commuting home in DC. I walked to the train to the bus to home. If I timed it perfectly, my commute was ~25 minutes. But if I couldn't cross the street in time, I'd catch a later train which made for a later bus. Or a slightly later train would be a much later bus. When those variances stacked up, it could take an hour.

You can also try to eliminate dependencies as well as track them.

This is partly why in software it's so critical to loosely couple software. If you're working on a big old ball of mud there are so many dependencies that even the smallest task ends up taking forever.

Fortunately SCRUM provides you with a block of time which you can use to decouple software independently of working on features or bugs, so this necessary work always gets done. Ha.

There is some stuff that can easily take almost no time or can end up taking months. Two items that are commonly like this on software projects are fixing bugs (that can either turn out to be really, really hard or not bugs at all) or upgrading package dependencies (can require no changes whatsoever or can require a significant rearchitecture).

I think scheduling these issues would benefit more from splitting up risk and estimated duration and estimating both separately. Perhaps by drawing a mini probability distribution for each task.

Breaking down the task only helps if, in some way, analysing the constituent pieces actually helps you better estimate the risk or brings unconsidered requirements to light. Sometimes it doesn't. If you do too much of it it can also end up being a contributing factor to the delays (analysing every last detail).

Projects timelines are like naked shorts in the stock market. You have a capped upside but unlimited downside - things can never be done faster than instantly but they can take literally forever (i.e. just never get finished).

You are being a bit loose with the term 'unlimited' but I guess that's marketing for you. You stand to lose the difference between your strike and zero. Last time I looked that's not 'unlimited'.

I thought 'strike' was an options thing rather than a short thing?

I'm not really sure what you mean. If you naked short without covering with options then the price can pretty much go up forever and you cover the difference. I don't understand where zero comes into play.

Sorry, my mistake. I was thinking the option on the back of it. Yes, you are correct, for simple stocks, if the stock price rises forever you theoretically have the possibility of infinite losses. Practically I'd not use infinite, but that has busted many people.

In addition to that, project's stages ahead-of-schedule deliveries are typically wasted, but behind-of-schedule deliveries are accumulated.

As an example: Arriving to the airport one hour in advance doesn't allow you to actually board a flight an hour earlier. But get to the airport one hour late — and you're flying tomorrow (or whenever the next flight is scheduled).

So it's like shorting a stock... Limited upside, unlimited downside.

Interesting observation, I'd never thought of project planning that way.

Yup. If you look at the factor your estimates are off, they'll follow a lognormal distribution.

And it's natural, if you think about it, that any schedule which doesn't include slack between tasks will almost certainly run long. (Even assuming you've got a mean of 0 -- as many tasks go short as go long -- you're not going to get lucky enough that the tasks that go short do it 'just right' to make up for the tasks that go long.)

So maybe include some slack.

Still, maybe you want to know how often will your schedule run long. So do some measurements of historical data to find out how your task estimates work out in practice.

Plug those in and do a few simulations, and viola, you can get a precise number of how you're likely to do. Plus handy tool to evaluate schedule quality.

I'll take me 2 weeks to use this API to make this application. If I work diligently it'll be done on time. But....oh no, the API can't do everything I need. Now its going to take 6 months.

The article suggests that failures in an n-step process are distributed as a Binomial distribution, which doesn't have the kind of long tail you describe. I hazard that you're right and they're imprecise. It would be great if they measured the shape of the distribution of completion times, scaling for project length, and then suggested a theory behind it.

> they can only go so well, but when they go bad, they can go really bad.

Well, they can, but usually the acceptable explanation for a project going really well is it actually was less work than estimated and the consequence is fewer resources next time. So, there is virtually no motivation left to drive a project into that area.

Ah I like this explanation a lot. It's almost as if project task timing distributions are negatively skewed.

Reply on news.ycombinator.com