Project delays: why good software estimates are impossible (2015)
chrismm.com
chrismm.com
And even though it's "impossible", I and hundreds of my peers at work do it every week with a workable degree of accuracy, relative to company-wide work in progress.
In any case, the usual problems are:
1. We software developers think we're the first to face uncertainty, including massive uncertainty. We are not. We are in fact lightweights in this department. Ask a civil, structural or plant engineer for war stories. Or read Industrial Megaprojects to see how skipping a single water-temperature test cost billions of dollars.
2. We tend to embrace the nirvana fallacy. "Perfect or very accurate estimation is worthless; therefore, all estimation is worthless".
3. "Some projects are pure research". Yes. But very few. And even they contain bog-ordinary implementation details. Writing a fantastical new language? Well guess what: parsers have been written before, and someone with data or relevant experience could break down the task into subtasks and estimate them.
I think the real reason that we so dread estimates is:
1. They aren't treated as estimates. People turn them into deadlines. Or try to negotiate with the estimate. Which means they aren't estimates any more.
2. We get them wildly wrong the first few times and throw our hands in the air. Improving estimates is hard. If it were easy, we wouldn't need to consciously do it.
It would be experimenting with different ideas for the language, which would require trial and error and experimentation. That would be hard to estimate.
As with most things that programmers do over and over again, creating parsers has been automated. That's the issue with estimation in programming, if you do the same things over and over they get automated. What's left is the unique issues which need to be estimated.
Is it your sincere, genuine contention that 100% of software development is pure research?
Say you are integrating some software into PayPal. Lots of people have integrated PayPal.
However no one has integrated your software into PayPal before otherwise you wouldn't be doing it.
Your software may have a weird workflow that makes PayPal difficult to integrate. So your going to have figure that out, and that's unique to your software
Your software may not have an easy way integrate 3rd party payment providers. So your going to have figure out the best place for those extension points, in reusable way for multiple payment providers. Figuring out those extension points is unique to your software.
There will be a lot of unique situations.
If however I was using common e-commerce package as opposed to some unique software, I would just install the PayPal add on. no estimation required.
Strictly speaking, every keystroke is unique. And strictly speaking, I cannot probably show that I exist independently of someone else's perception, or that I'm not in a computer program and so on and so forth.
Meanwhile, back in people-need-to-make-decisions land, as an engineer I don't get to just shrug when someone asks me for my professional opinion.
I may ask for time to investigate. I may give a wide range due to uncertainty. I may recommend a different approach. I may suggest we bring risky unknowns forward.
But ultimately I give my best estimate. If I get it wrong, that's not a sin. Because it's an estimate.
In reality though, business try to use estimates to preasure software teams, and punish people who got the estimate wrong.
I'm only such a stick in the mud about the topic because I'm lucky to work in a place where estimates are respected as estimates.
At places where true software engineering work is required, like mission critical systems, the type of planning that you may be used to will be more common. However even for large companies like Oracle, things like the healthcare site can go wrong [1]. That is just the nature of having to constantly adapt. If things just stayed stagnant, we could get the same level of safety and generations of consistent improvement that things like cars have received, which does happen with maybe Unix or C perhaps. But that does not change the fact that most day-to-day software development is all about hiring someone who knows slightly better than you, but, more importantly, knows how to learn how to get many reasonably new things done which involve parts of what he knows. We don't have the luxury of using safety or danger as a warning, usually.
[1]: http://www.bizjournals.com/sanjose/news/2013/12/03/cover-ore...
We estimate every story. Sometimes hilariously wrong (Diego was a classic example of a research-heavy project blasting past the initial gut-check estimates and improving in accuracy as it matured).
But a lot of the time well enough to allow for useful business decisions to be made by ourselves and other members of the Cloud Foundry Foundation.
The reason that I keep saying that useful estimation is possible is because I do it every single week. Hundreds of my peers in Pivotal and partner companies do it every single week. Hundreds of client developers do it every single week.
But I hear that it's impossible. So I guess I'll change my résumé to read "Wizard".
High school level math and problem solving plus having ever worked with software clearly state that if you have a software team, you could at the very least collect a single statistic of estimation error (actual effort over expected effort) and get a long-tail (it's going to be more more than it'll be less) distribution. Now you can spec out new projects, get values for expected effort and factor in the error distribution to get a distribution of possible actual effort for the project. If some high percentile of that distribution is more time/money than you can tolerate you have to reduce risks by cutting features or increase the budget (or have a significant fraction of projects fail).
Of course, there's lots more sophistication that could (should) go into this but even this is beyond any part of software industry I've been allowed to participate in, because it gives answers that people with authority find inconvenient.
They are looking for estimates to hold you to account, because that's how you run a good business right?
If you gave your answer to someone in management. It would just look like your trying obfuscate the answer.
There's a difference between estimates and their misuse.
Or huge range of possible release dates(because there is huge variation between the estimate and actuals), as with other estimation methods.
It's an industry with a high gross profit margin. If you have a hit, you can get with just about any process or non-process. So you never need to invest heavily in better estimation.
If you don't have a hit, you will probably go out of business. So you never get the chance to invest heavily in better estimation.
If the expected ROI on a project is small, then having really good estimates matters a lot, because it can make the difference between profit and loss. But if the expected ROI is large, precise estimates matter less. Tom DeMarco has famously made this point [0].
This is basically what I argued. Software businesses often have such fantastic margins that you can do any old thing and still rake it in.
Oh yes this annoys me. I've been asked to break it into sub tasks then justify those. So best case estimates based on some predicted sub tasks ... before any design has been done! Then the surprise when it takes longer
Humans routinely underestimate the time and difficulty involved in just about everything. Breaking a task down into subtasks leads to higher estimates which will typically be more accurate.
You are of course being asked to do something very difficult. In 3-point estimation schemes, you'd give back very wide bands for a few subtasks that you identified. Maybe point at the cone of uncertainty a few times.
I know it often doesn't work. To me, most of the problems with estimating aren't with the estimating.
I wish I had more than a single upvote to give, because that is absolute gold.
In my previous life I had the dubious pleasure of occasionally doing project estimates. I followed a fairly simple sequence which worked for that particular company and their projects. Please - do not try to blindly use this method, it will give you misleading estimates.
First I checked how much proper knowledge of the problem domain we had in house. Then I estimated (in man-weeks) how long it would take for a single senior engineer to cover a given subtopic, if everything worked the first time. Then, I multiplied that estimate by 2.5 - this covered pretty well the time it _actually_ took them. Then I added a 15% "fuzz factor" to cover for still unexpected surprises.
Multiply that by the number of engineers really needed for the project, and you get a reasonable ballpark estimate of how long in walltime the project would take, as well as how much effort it would require.
And because every single client was always haggling with the estimates, trying to turn them into hard deadlines... I added another +30% on top. We all knew that was the haggling headroom, and expected the salespople and clients to eventually carve that one out.
Our projects were mostly on time and within budget. Yup - mostly.
TL;DR: please do not try the above at home or at work. Those figures worked for a very specific company, for their highly specific projects.
This is what distinguishes software development from other disciplines. If you want to build a bridge, you create the plans and derive your estimate from the plan. But software development is the creation of a plan. Do engineers know exactly how long that planning process takes? Probably not. It's insignificant compared to actual construction. But for software development, that is everything.
Look at something like a consultancy that develops safety procedures for organizations or a company that handles building logistics operations.
Software doesn't have the same physical constraints that bridge building does. Anytime you have fewer constraints, you have more possible solutions which take more time to sort through.
However, if it's something truly novel, then kind of by definition, it's impossible to predict how long it will take, because it's never been done before.
Of course, there's a million shades of gray in between, but it's not like bridge building. Bridges are pretty unchanged in 1000's of years. Most software you write, should be brand new, otherwise, why do it?
This leads to the next quagmire:
How do you reliably estimate how long it takes to find the already written software that does X?
That said, for some classes of software, Drupal plugins and Minecraft mods come to mind, are all varying degrees of abysmal. Those take a lot longer to sort through.
Things like "evidence based scheduling" and your permutations around "velocity" and burndown charts seem to help, particularly over time, but I've found no magic bullet.
1. If you are doing something that people have done before, then it is possible to come up with a reasonable estimate. How? Just talk with someone who has done something similar before and they will help you understand the complexity involved. Example : Rubik's cube.
2. If you are doing something that no one has done before, then it is more difficult to come up with an estimate. Example: Unified theory of forces. In the 1970, people assumed that we are very close. It's been 45 years since then and we aren't any closer.
Of course, I have often made the mistake of underestimating tasks of type 1 because I didn't talk to anyone with enough experience. Maybe this has to do with self confidence. I am very confident of my abilities so I usually tend to underestimate things rather than overestimate them.
Skills, technical environment (system, programming language, libraries, available tooling) and team efficiency vary A LOT.
-- Regarding your comment of overestimation: I think this is a problem for almost everyone. When you get a task described, you often think "hey, this will be easy", without understanding all the requirements in detail, the complexity it will entail or the broader effects in a system. That's where only experience can help (your own or someone elses input)
And yes, it does take time to build that trust. Often times when I'm at a new company, the business people are used to short estimates and overruns. I have to train them to understand that while my estimates are longer, they are typically closer to correct. It only takes once or twice of having something done when expected to win them over.
I generate realistic estimations by attempting to overestimate and failing to do so sufficiently.
Watching coworkers try to estimate how long it'll take to deliver software really makes you wonder if pessimists actually exist, or if they're rather some kind of myth or mirage.
The best we can do is break tasks that we know about into tiny chunks and estimate those. Then, as we are working and discover the unknown unknowns, break those down and estimate those.
Montgomery Scott: Almost done, sir! You'll be fully automated by the time we dock.
James T. Kirk: Your timing is excellent, Mr. Scott. You've fixed the barn door after the horse has come home. How much refit time before we can take her out again?
Montgomery Scott: Eight weeks, sir -- [Kirk opens his mouth] -- but ye don't have eight weeks, so I'll do it for ye in two.
James T. Kirk: Mr. Scott. Have you always multiplied your repair estimates by a factor of four?
Montgomery Scott: Certainly, sir. How else can I keep my reputation as a miracle worker?
James T. Kirk: [over the intercom] Your reputation is secure, Scotty.
By all means, take risks and explore Software. But do it under the premise of Software Development if Engineering practices will not be applied.
In agile environments, push for more "spikes" if estimates cannot reliably be provided for a story.
We do. We call those "libraries". We have more of them than most disciplines could dream of, and they work better than most disciplines could dream of.
So where most disciplines are taking the closest things to libraries they have and still applying a lot of drudge work to put them into the current context, with "unexpected work" forming small single digit percentages of the work, we get to yank all the easy drudge work off the shelf and the resulting "unexpected work" makes up the majority of what we're doing.
I'm not going to pretend everything is perfect in the land of software engineering, but the answer to "why are we so stupid that we don't do the obvious thing that people have been complaining about for 50 years now?" is that we aren't that stupid. We are very good at shrinking the obvious, easy parts of the problem down, so all that's left is the hard part. Which, since everything isn't hunky dory in the world of software engineering, we also often add to with self-inflicted injuries. Still, the amount of off-the-shelf crap that I can just push a button and have staggers the mind.
If "real engineering" worked like computer science, we'd be down to a guy pointing and five minutes of fiddling with his phone to lay a bridge down across the average river, four minutes of which would be rebuilding the bridge in various different aesthetic styles to see which suited the location best. "Do I like the fake aged wood look, or do I go with modernistic concrete? Let's see the wooden one again...."
Your homework is to write "hello world" in your favorite language and step through the resulting program in gdb or equivalent, starting from the moment the OS turns control over to the program, one assembler instruction at a time.
I apologize if this comes off a touch crabby, though upon re-reading I'm not sure exactly what to tone down about it. I'm getting really tired of the inferiority complex programmers have about "real engineering". Yes, we do real engineering. If other engineers had the advantages we had, they'd be much more "agile", and they'd make the same messes we do. They work the way they do because of the constraints they have on them, for exactly the same reason the space programmers work the way they do and for exactly the same reasons it would be wildly inappropriate for all of us to work the same way NASA does all the time.
If I'm building a back-office website for a few colleagues, I may not invest the same rigor as if I'm building a website that will be accessed by hundreds of millions of people. Many software tasks do not require great rigor: you add rigor until the incremental benefit is less than the incremental cost.
As a result, I would guess that the average software engineering task is less rigorous than the average real-world engineering task, but that's only because real-world engineering frequently risks or protect lives. Even the tire rope swing, the hammock, and the treehouse put lives at risk if built poorly. The back-office website on average does not. Business might be disrupted or individuals might be inconvenienced, but typically no one can die.
Yet, many types of software involve just as much rigor if not more as other engineering disciplines. For an example, read "They Write the Right Stuff", a story of NASA avionics software engineers [0]. I consider software engineering to be an engineering discipline because of the fact that rigor is required, and because a good engineer must know how much of it is appropriate, and be able to apply it effectively. A software engineer produces software to specified reliability and tolerances, according to requirements.
[0] https://www.fastcompany.com/28121/they-write-right-stuff
Engineering is not libraries or cook books: engineering is math.
Civil engineers model buildings and other structures. Electronic engineers model circuit behaviours. Aerospace engineers model aerodynamics and materials. Traffic engineers model traffic flows.
Stuff only gets built after a model appears to work. This is a good thing, because otherwise we'd be build bridges with rule of thumb and trial and error, and they'd fall down a lot, and kill people.
Knowing the math makes it as easier to model a business case, complete with deadlines, as a process.
Why isn't software engineering like this? It's not impossible to model software. It's just conceptually demanding, and very expensive.
Professional software engineering doesn't look much like vanilla coding. The percentage of developers who can hack it - or could ever hack it - is not large.
So we muddle along as a trade, not an engineering profession, hammering and gluing libraries together.
The pay is still good, hardly anyone gets asked if they know what the ACM is in job interviews, and a lot of customers don't mind that much if a site only works in Safari and not in Chrome. (NB: actual experience I had with my online banking account last week.)
It's true that being asked to estimate how long something will take may be met with a sharp intake of breath. But the reality is that very similar projects have usually been completed many times in other companies already, and the amount of truly unique and original R&D needed for most work need is miniscule.
Yet when we call it Engineering, we hold ourselves accountable to a different set of expectations.
Engineers work for insurers. For the moment, software engineers work for people other than insurers. I'm not quite ready to turn my culture over to the actuaries just yet.
The blessing and the curse that makes software unique is that you get to make the rules, all of them. Let that sink in for a minute because if you look at all other engineering practices they are all based on the rigorous and immutable constraints they have to deal with.
Any set of automated practices for producing software will only make sense for a tiny slice of software for which a set of constraints can be reasonably agreed upon.
As a team lead who lives in this world.=, good software estimates are not impossible. If you are doing the kind of work that you have done before, then it's very possible. If you are doing R&D, then sure it's damn near impossible. But if the work and requirements are clear, then there are plenty of ways to make "Good software estimates".
I fight estimates when I don't have a good business requirement, but tell me what your problem is and the solution you desire and I will give you a good enough estimate. Estimate is never just a date, it also includes the unit of work to be done, and the resources.
> If you're doing similar work over and over again as a civil engineer, you're not doing a good job.
Hmm. Not sure I feel comfortable with that.
> If you're doing similar work over and over again as a small business accountant, you're not doing a good job.
Actually, I'd rather prefer you to stick to the basics, please, no need for discussions of complex derivatives of treasury stock.
> If you're doing similar work over and over again as an orthopaedic surgeon, you're not doing a good job.
Thanks, but I think I'll get a second opinion.
I expect those engineers still gave estimates and were chosen because they had the most relevant experience and expertise.
I'm just saying they are probably way out, and probably arn't that useful compared to other methods of building software.
A lot businesses people who get this, then ask why bother if they are going to be really inaccurate?
Instead focus on small intervals, work out what you should try to build in those intervals. Implement. Gather feedback. Repeat.
Or even better do the implement and feedback cycle continuously.
If you are not happy on the progress of a feature, cancel it.
Because you doing things in small intervals, and then adjusting. It won't go wildly of the tracks, before realising you have to adjust or cancel it.
I absolutely agree with you. I think we're closing on the key difference, which is the scope of the estimation.
I am accustomed to estimating the smallest self-contained unit of user-facing value. I'm used to doing it weekly. I'm used to being in a room with my fellow engineers to do a simple points-based estimate of complexity, with projected dates derived from the velocity of the past 3 weeks. I am used to PMs who understand that sometimes we hit iceberg stories, but that most of the time, the projections are good into the next few months.
What I don't have to do is build a magical estimate of the next three years. If I was asked, I would try to based on historical data and research, but the bands would be extremely wide.
Thinking about the analogy that we often make back to construction, the problem is that classical project management is essentially an estimate of an integral. There is a discrete endpoint, and we can estimate (and re-estimate) leading up to that end point. Funds are committed in large chunks and the cheque-signers need high confidence to proceed.
But software is never finished, only abandoned. Estimation has to be differential and constantly updated. If the software is developed to be releasable at all times, commitment can be incremental and early abandonment due to negative feedback is advantageous.
I'm not saying anything new here. But I understand better where you were coming from.
Software development is construction of the plan. And all industries also estimate how long planning will take but it's a vastly more significant for software as that's the entire thing.
Which means that we're doing it more often, so we have more data and experience to call on when estimating.
And it means it's more important to estimate the design task, since it's vastly more significant.
They're estimates. They're not called "certains", "definites" or "numerical solutions" because they are none of these things.
You can't download the building package. You actually have to build the building again. If it's the same building I would expect them reuse the same blueprints. So that holds true.
Where as in software it's quite easy and desirable to reuse software. Which is why you should not be rewriting the same thing over and over.
As a result, the problems you face are unique. Your not just stamping out the same building again and again.
You also have to factor engineering time into the plan. If you read Industrial Megaprojects, the author shows that projects which don't do a bit of upfront estimating and research tend to go off the rails on a scale that software struggles to match.
If you make a blueprint and give it to 2 construction companies to build the building, you'll end up with 2 very similar buildings.
If you give a software spec to 2 separate software companies you'll get 2 completely different programs.
You can't create a blueprint/spec that is sufficiently detailed so that 2 different companies/teams will build essential the same software.
A blueprint/spec that detailed would be the code itself.
The closest analog to software would be designing processes, or maybe designing a factory or assembly line. And the analog to the code in that situation would be the factory or assembly line itself.
It's that professionals perform similar tasks very often. Including us.
Massive amount of very similar work being done. How much shared code?
They have probably spent a lot of time and money paying UX guys trying to optimise for their unique situation.
I would assume they reusing either in house or 3rd parry libraries in their code, but that stuff isn't where estimates go wrong.
It's usually experimentation of trying to get UI right for the user.
I mean could tell you how long it would take to make a html page, if you tell me what tags and elements you want in it.
It's a whole different ball game, if you ask for a estimate of creating a complete feature because it will probably involve a little trial and error and feeback from multiple people. The first few stabs at it won't be quite right etc
I've found that without straight up hopping jobs, it's almost impossible to do truly new things rather than settle into a role where you're too valuable to not exploit for the same old crap you've done before.
I'd still call that doing a good job - businesses need people that can pump out good but similar work. People that always need something new once they master the old thing can be liabilities, depending on the context.
They have issues with, let's integrate x product with our product.
Yet if you spend just like 20 minutes to try and break down the tasks and do some research you can usually reduce the margin of error significantly. Our brains are good at ignoring complexity and our egos are good at telling ourselves we understand things we don't. Break it down and do you research. Stuff like poking through github repos of libraries you might be adopting. Lots of forks? lacking documentation/tests? What kinds of issues? Who is using it and why? You still can certainly still run into unexpected complexity but you get good at recognizing where it's likely to pop up. It's hard, and you won't get it right always, but it doesn't mean that shit is 'impossible'.
For people having trouble with this, Jim Shore does a good job giving tips: http://www.jamesshore.com/Agile-Book/estimating.html
"Estimates or No Estimates?"
Bad estimation techniques - eg. guesses without a basis in historical data (most developers don't collect historical data) - are usually wrong. No one would argue this point.
If you do iterative development with intelligent planning and collect some historical data to use in estimates the game changes. First, most of the examples of "unexpected complexity" you gave can now be identified early and factored into your estimates. That leaves the 1% of truly unexpected stuff that you can then deal with as it comes up and factor into your future thinking in a post-mortem. Second, since you now identify risks earlier and can factor them into estimates, most of your work becomes estimable, and with historical data your estimates will improve.
There's even multiple process products around this idea already, but they're mostly used in the "safety critical" industry, see [1] [2].
[1]: http://resources.sei.cmu.edu/library/asset-view.cfm?assetid=... [2]: http://www.sei.cmu.edu/tsp/
Solution: divide the project into fewer tasks :)
Prioritized (short) iterative development is the only path to success.
Apparently software engineers as a group seem to think "fuck you, it's done when we feel like it" is a valid answer?
I disagree, and have always offered estimates, probabilities and contingencies when talking to product owners when I was a lead. I tracked data, and adjusted accordingly, and ended up giving estimates that were statistically sound.
I don't necessarily expect individual contributors to do the same, but anyone that's a lead should be an adult and account for their team's actual productive capacity. This is absolutely expected of a building company, even if you don't expect a single dude with a hammer to be able to estimate how long the project will take.
If you are giving regularly accurate long-term estimates, I'd suggest that you create a consultancy that performs this service or teaches how to do it. Most companies and teams just aren't capable of doing this.
Please explain exactly what you mean by this.
"SIMONYI: We have great difficulty in predicting the time it will take to write a program. There are valid reasons why this is so. It doesn’t mean we shouldn’t try our best to predict, because there are reasons why a prediction might be useful, just like when a weather prediction has economic as well as other benefits.
Really good programs will live forever and take forever to write, at least as long as the hardware exists, and maybe even longer."
Source: https://programmersatwork.wordpress.com/programmers-at-work-... (great book, btw!).
Now, if you want to estimate properly... ask for a min time estimate, max time estimate, and confidence level.
To say "I am not sure", is actually fine. That can be the start of a conversation where you can have the opportunity to improve that estimate. Whereas a closed answer like "4 hours" won't.
1. Top level commitments should be vague
-the statement "we will build you a website" says nothing about what will be on it
2. Scope the top level commitment, don't over scope, commit only to achievable scope
- If you don't know if something can be done don't commit to it, commit to identifying if it is achievable. Green engineers I have worked with tended to miss deadlines and when we dug into it the problem was often that they took on tasks we didn't even want completed. Commit to delivering something you are certain you can do.
3. Write the scope down, and stick to it.
- A good scoping document is like a good legal contract, it keeps everyone honest while limiting liability. It gives someone a commitment they can hold you to, and it gives you an exit clause if they try to rewrite the rules. If someone tries to change your agreed scope after the fact, adding work means timelines must be adjusted and reducing timelines means work must be removed.
4. If you work somewhere where you can't seem to make 1-3 happen find a new job its just not worth it.
Developing software also involves discovering the real user issues by experimentation.
If you scope it, you may deliver the project, but you probsbly haven't delivered what they actually needed.
The best software is developed with feedback loops.
Mathematical Limits to Software Estimation: Supplementary Material: http://scribblethink.org/Work/Softestim/softestim.html
The way to access a Rubik's cube stays the same whereas accomplishing a coding task is enviromental dependend. Software changes over time making your experience how to solve a task redundant at some point.
Source: i used to work at an organization you've heard of that could reliably predict, down to the day, how long it would take ~4000 engineers to build and release the next version of its product. Three years in advance.