Guide to Software Project Estimation
scalablepath.com
scalablepath.com
It's either some existing deadline or reporting/sales cycle, some budget caps, like in grant proposals, or some promises already made by/to 'important people', or some fear of 'small people' to underdeliver etc.
The estimation would all be fine if all involved people shared the same goal and responsibility.
I find it practical to split the desired outcome which needs an estimate into two variants: 1) the most desired/promised/advertized and 2) an at-least viable variant.
If noone can see the second variant, its viability and the effort it needs, then some details or skill force are clearly missing.
If the second variant is estimatable, then the estimate could be used as a basis in dealing with the external constraints.
If the devs are saying that in a given timeframe they can see a prototype done at least and you're fine with that, then no one should be damned if that pans out just to be the case. So it has to be clear from the beginning if it's at all acceptable to put such variant/prototype into production.
As usual with software estimation, unless you've read this paper: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.19.3...
and have a thoughtful response to it (of which I've read some solid ones), I've very little interest in hearing what you have to say about software estimation.
2) Even if you accept that "if", this is like a halting-problem proof. Fine; it is impossible to estimate the complexity of ALL software. That does not mean that it's useless to quantify the complexity of software in limited but well-behaved problem spaces. How much of any commercial project is actually spent working on the cutting edge of computer science, facing complete unknowns? An estimate being wrong occasionally is worth most estimates being mostly right.
3) Why do you consider a 20 year old paper that's only been cited 16 times to be critical reading about estimation, when a vast body of research in forecasting exists, written by people who have measurements of the accuracy of estimates to base their theoretical models on?
Have some links to this vast body?
I've encountered very few over the years that actually qualify as "science" as much of it is fake or close to fake.
http://shape-of-code.coding-guidelines.com/2021/01/17/softwa...
But it's certainly made a big difference for me in practice, and given the gushing about Steve McConnell in another comment thread, I'm not alone.
The old parable about bugs, that it's 10x harder to fix a bug than it is to write the bug, shows us that the time to implement something complex is not related to it's complexity.
For example, if I wanted to build something that randomly selects a function from github, and then runs that code on your laptop - I bet I could estimate and implement that code.. but good luck every defining it's features, complexity, etc in any mathematical way.
But you don't need to define things that way to have reasonable estimates, for the same reason that when you build a shed in your yard, you don't need to understand all the physics of how it's held up.
I am even more skeptical about the promises of software evaluation methods.
But what I am the most skeptical about is teams avoiding software estimation altogether because they share the two skepticisms above.
Well, to put your mind at ease, I don't avoid software estimation because I share your skepticisms (although I do), I avoid it because I've observed that it's a complete waste of time. Nobody, at least never in my 30-year career, has ever asked for an honest estimate of how long it would take to produce a software product. What they have asked for is somebody to agree that the time that they have budgeted will be enough for the (vague, still being defined and still to be defined even beyond the timeline) software project and take the "blame" when it inevitably doesn't.
> somebody to agree that the time that they have budgeted will be enough for the [...] software project
For me, that's still software estimation.
What causes estimates to fail are the unknowns: unavoidable surprises when implementing something new, unexpected change in requirements, etc.
It would be easy to have accurate estimates if there were no unknowns. But every innovative project must always be a march to unknown territory.
I would spend an entire day "pre-programming" everything in my head, estimating the length of each little chunk, adding them up, then multiply by ~2.
It worked for me. But I still would never trust the estimates.
'How long did a project this size take last time?'.
As an aside, years ago I worked at a company that did thorough (and inaccurate) bottom-up schedules. I got dinged for not using quarter hour accuracy in the various task estimates.
That is also its downside. It is a reference. Not a textbook to learn things in a pedagogical structure.
Stick to your guns when Sales tries to get you to change your estimate (and they will). Tell them they can discount the project, or change any other variable they need to satisfy the customer, but don't ever let them touch the time estimate. Not really a tip for making the time estimate, but keep your ass covered once you do.
Personally, I'm often dumbfounded that folks still use planning poker when there are so many more reliable methods as discussed in the book, e.g., wideband Delphi.
If that seems like too much effort, I also run quotes.vistimo.com , which takes a similarly (if slightly more advanced) statistical approach, but does all the math for you.
Start with a known past project that is in some way similar in magnitude and adjust from there. For example, "this is twice as complex as some other project I did, and that took 2 months so this one might take 4 months". Most importantly, resist the temptation to say "although 1 of those 2 months was because of unexpected thing X so I shouldn't include that". Overall, it's highly flawed, but much less highly flawed than anything else. This is called "reference class forecasting".
He gave a really compelling explanation of why estimates are almost always underestimates by a significant amount, and this technique is the best defence against it, but I won't try to resummarise because I'll surely misrepresent it. But I do recall he gave an example where he and some colleagues were trying to make a school syllabus about deductive biases, and underestimated the effort required for their own project.
"Dr. Kahneman, you've been at this for 40 years. Do you think you've changed anyone's ways of thinking?"
He smiled and said "No, not even my own!" and then recounted how in his personal life he'd made a mistake which he'd written about extensively (not the one about planning, though). It's a human failing, not a methodological one.
I'm also vague about his example, but I think it was a new textbook. He asked his committee to reflect on their own past experiences with similar books. "Two years" was the past experience. Then they decided that it really should be six months, and that's the estimate they went with.
No one wants to accept that shit happens and it's going to happen again. That's why estimation is hard.
Me neither. When I was first starting out, that really stressed me out a lot until I realized that I didn't work with anybody else who was "good" at it - that is, I didn't work with or know anybody who could take a list of requirements written out in English and produce a timeline that had any relationship to how long the software would take to be ready to use.
Been doing this professionally since 1992. I still haven't met anybody who was "good" at estimation.
For each story, add an initial estimate (in terms of developer-days). This is your "low" estimate. Now in a second column, add a "high" (potential-but-reasonable "worst case") estimate. If you're looking at more than 10-15 days for either column for a story you should probably break the story up some more.
Now add a 3rd/4th column, which are the low/high estimates multiplied by 1.3 ("fudged low" / "fudged high"). Total up all stories per feature in a row at the bottom of each feature's section. Divide by team size, divide by business days, round up to nearest integer, and you have your calendar weeks for each feature.
When sales/marketing ask you for estimates, you then respond "between X and Y calendar weeks from [completion of previous feature]". Just be aware that they will hear X, so make sure Y is very clearly included in every communication where the dates are being discussed. If "previous feature" slips, make sure to communicate clearly that "next feature" has also pushed back by however many weeks. You'll be tempted to, but don't be optimistic with progress reports or estimates of where you are in the range - undersell and over-deliver, and you'll keep more allies on the business side.
To get the mean (expectation), multiply your estimate by about 1.6. To get the 95% confidence bound, multiply by 5. To get the 99% confidence bound, multiply your estimate by 10.
Understand why a distribution results in different numbers for different audiences, and why that's not the same as being inconsistent.
Use the mean for calculating sprint workload and capacity planning, because the average is what matters for that, not the accuracy of any single job. If your manager understands probability then give them all these numbers, otherwise give them the 95% confident value, which you should also give others internally who depend on that specific job being done. Give marketing the 99% confident number even if they understand probability, because they're looking for a commited deadline they can use externally. They will push hard for an early date because they want the work done quickly, but they actually don't want to hear your optimistic estimate. It's easy to make that mistake.
When requirements are understood, experienced developers are actually very, very good at estimating median completion times even just by gut feeling, but often fail to account for the distribution, especially when communicating with stakeholders, which makes them take heat when they're sometimes wrong by a factor of ten.
(I'm glad to see lognormal making more inroads in software estimation. McConnell is great, but assuming the normal distribution leads to some weird edge cases.)
I like the idea of multipliers but the maths here is just meaningless fluff to justify a particular number. If your initial estimate really was a median then it would be an overestimate (i.e. the project ends up taking less time) in about 50% of cases. In practice I find that initial estimates are overestimates in about 0% of cases!
It is, yes. And it frequently happens that you go to fix something, which seems really difficult, and then you realize that it's actually an easy fix or not a problem at all. But when you fix one thing in half the time you expect, and another in twice the time you expect, this doesn't average out, because the average of 0.5 and 2 is not 1.0.
You might just be discarding the cases where estimates were found to be conservative, either because delays are more impactful and memorable, because the underestimates were close enough to be treated as on-time, or because the slack in the schedule was used to buy time for something else, originally out of scope, that was lumped in.
Anyway, these numbers aren't just pulled out of a hat. It comes from studying vast amounts of high quality (but unfortunately, not publicly available) data collected comparing developer estimates and measured outcomes.
> Whatever number you come up with, treat it as the median ...
Especially the "whatever number you come up with" bit, which seems aimed at everyone quite generally. When people usually come up with a number, it is likely to be a substantial underestimate. This isn't just a cognative bias where I've forgotten the times the task turned out to be simpler - evidence shows this to be the general trend (a la Thinking Fast and Slow, as I mentioned in another comment). So your rule that it should be treated as a median isn't correct in the majority of cases.
> these numbers ... comes from studying vast amounts of high quality ... data
Well that's a different matter. Of course, whether the result of analysing that data is a median or a mean (or something else) depends on how exactly you analysed it.
You're already using an abstracted measure of time, by working with a derivative value of "developer estimated hours". You're already doing timeline projections on the average throughput of your "adjusted developer hours" unit. That's most of the value right there.
You can get even better results, with a little less cognitive load, by applying the research that people are much more consistent in estimating complexity than time (note that your method relies on consistency, not accuracy, to succeed). A quick imagination exercise validates this point for most of us: You bought a new IKEA sofa - how much time will it take to build? Honestly hard to do, and we're never accurate. But consider instead: how hard is it? Way easier to answer. And if you already know how long it takes you on average to finish other tasks of similar apparent difficulty...
Try using your exact same system, but ask people to estimate the task in terms of complexity. Use any scale you like, as long as the units have consistent value in your developers' minds (I like "cups of coffee", personally). Make your Dev team agree on the difficulty score for each Feature, to ensure that consistency.
Side benefit: Devs stop worrying about time and taking shortcuts (aka "technical debt") to meet their time estimate that you don't believe anyway. They're also a lot more likely to consider hidden risks and sources of extra complexity in the estimate.
Then you just track the actual throughput with a confidence interval, and use that to make timeline projections with a confidence interval based on that tracking.
TLDR: try asking Devs to estimate complexity rather than time, and use a moving average with confidence interval rather than the static 1.6 multiplier to make timeline projections. You'll find your projections more accurate and developers less stressed about it. You'll also have reinvented story points.
There's no shortcut to avoid the requirement to present different summary statistics to different stakeholders. It's a consequence of decision theory. Unless they're equipped to understand the whole distribution.
It's also the wrong sort of rounding. I think an ikea sofa might take an hour, but if it took all day I'd be pretty shocked. But with software tasks, it's important to accept that the distribution is long-tailed. Sometimes it really will take 10x as long as you expected, and that's not your fault. Story points would have to abandon all meaning to capture that much variance.
I don't recommend incentivizing estimates, though. A big benefit of recognizing a developer estimate as short-hand for the median of a distribution is that when the time doesn't match the estimate, it doesn't mean the estimate was "wrong" or "bad", and the developer shouldn't feel bad.
Besides what is covered in those books, I've found it extremely useful to document assumptions. Every single estimate has some mental model of the project to be done. Code to be reused, vendors to integrate and, most importantly, things that won't be done. The real project almost always breaks with some of those high-level assumptions, but that tends to be lost in the shuffle.
Attaching assumptions to the estimate makes it much easier to do a post-mortem.
A powerful memory cannot compare with pale ink.
- Have the people that will run the work (tech lead, senior dev, whatever) break the work down in meaningful chunks/modules.
- The number of modules is important: it should be roughly equal to the man-months you're trying to budget for --just a rule of thumb. Basically, not two few and not too many.
- Have the same people give you two numbers for each chunk: the best case scenario based on what their gut tells them and the worst case scenario (where 'worst' here is a bit short of nuclear winter, but not optimistic).
- Your project will take the total of the averages of the min/max estimates.
You're welcome. Source: 20 years of delivering tech projects on time...