Software Estimation Is a Losing Game (2014)
rclayton.silvrback.com
rclayton.silvrback.com
Other tasks I can accurately say I don't know how long they'll take. This new feature is... a couple of weeks? This new application is... a few months? I can roughly guess (estimate!) but there's a wide certainty margin.
That uncertainty is never built into the mathematical model used by project managers, which is invariably a simple addition of scalar estimates. The only common way of dealing with uncertainty is to pad the estimates (multiply by two, etc), but that's still not a statistical boundary -- it's just a scalar.
Furthermore, when adding uncertain values, they wind up not being independent, for a number of reasons -- the lessons from one feature are carried on to the next, or the next task is easier because of new existing code, or because as you approach the original deadline you invariably throw out feature requirements. So you can't necessarily just add all the uncertainty values.
So a better approach to estimation would be to formulate a (simple) model which encodes each estimate as {guess, uncertainty, independence} and then provides a function for compounding the guesses and uncertainty and independence factors in a realistic way.
It depends on the project manager.
The PERT estimation technique was based on giving a 3-part range estimate for broken-down tasks. These are then rolled up with a simple formula.
In the PERT method independence is handled by creating explicit links between tasks, which leads to the concept of the critical path.
Nobody does these. Because it's very tedious. There are a lot of tasks for which "just try it and see" is less effort than performing a full-blown estimation raindance.
PERT: https://en.wikipedia.org/wiki/Program_evaluation_and_review_...
Barry Boehm's recent work:
http://www.amazon.com/Incremental-Commitment-Spiral-Model-Pr...
It was a lot of fun reading up on the estimation literature in various disciplines. Personally I think that the real magic of PERT 3-point estimation is the "unpacking effect" -- making people think about the subtasks of a headline issue.
When I'm in situations where I point a story, I like to ensure that we engineers say aloud what assumptions we're pointing on ("Bootstrap will make responsive mobile almost free") and then a quick low-detail breakdown of expected tasks ("new page, new RESTful endpoint, new model code").
I find that the latter, in particular, makes a big difference in how pointing goes.
The problem I have always had afterwards is communicating the results. Most people I've worked with cannot intuitively grasp confidence levels (90% vs 95% vs ...). They also invariably want a break down of how long each task or group of tasks will take so that they can negotiate down the estimate, not understanding that the total estimate is not a simple sum of each task. I'd love to hear better ways of explaining/using these sorts of estimates.
The first time was my fault: I left the columns that had intermediate values visible. One of those was the variance in each estimate, which is just the Std Dev squared. The numbers were big. No one could stop talking about how big the numbers were. No one knew what a variance was, nor appreciated that it was just the number in the previous column, multiplied by itself. Learned the very important lesson that you hide all your work, no one gets it.
Where I work, Engineering owns the estimates. No ifs, buts or maybes. We own estimates. Product managers have unlimited ability to reprioritise, but we and we alone get to say how hard it's going to be.
That said, another secret is: use story points. They are dimensionless units that are only meaningful within a single project. As soon as you give dollars or days, everybody has an opinion or an agenda.
But if you use points and project based on velocity, it's hard to negotiate. Because you're not negotiating. Those are the actual numbers. The actual historical data, staring you in the face.
(Edited for clarity)
Just out of curiosity, were any of them interested in learning the answers to these mysteries?
This was actually not a time estimate. We were estimating the footprint for our production deployment for a very complicated distributed system. Most of the pieces were only in early development and the load model itself was uncertain by orders of magnitude.
It's a situation where statistics is easily applied, if you know statistics. If you don't, you'll never even realize it would have helped and keep saying that math is useless outside of school.
When you decompose a larger task into smaller ones, that's all still true for each of the smaller ones, right?
But you must remember to add the errors! And that is precisely why overall confidence may not likely improve with decomposition vs. an aggregate, or coarse, high level estimate of the whole thing.
And this is true for more than software. Why?
New tasks can be broken down into two coarse buckets: done before and novel.
Novel things really are novel! It's important to recognize this case and qualify it. Progress might look like:
week 1 - 10 percent done, week 2 - 10 percent done, week 3 - 15 percent done, week 4 - 50 percent done! Yes!, week 5 - 5 percent done... everybody take a drink, week 6 - 40 percent done,
etc...
Because it's novel, there are new dynamics in play, and even the best planning may not account for, and actually usually does not account for, things that just come up and that must be dealt with.
For the done before case, it's better as a lot is known, but there are still many factors out of the team control. Updates, changes, regressions, unfound bugs, all contribute to variance. The more the organization completely owns, the less this may be true, but it's always somewhat true.
The pragmatic person might classify everything as having some percentage of novel, and break it down into known novel and unknown, "it just hit us!" type novel and go from there and likely be better off when having to manage expectations.
Another way to think of this, depending on it being software or something else, is production vs "R&D" and the research and development part just isn't cog / production like. Time and materials estimates, if they are even given, must be considered accordingly.
If they are not, expectations do not match realities, and now there is a management and potentially a funding problem in addition to the primary problem needing a solution.
Estimating is most definitely useful. But you have to fight every step of the way to keep an estimate from becoming a "promise". Promises that can't be kept are not good for any of the parties involved and lead to things like developer burnout, poor quality, technical debt, surreptitiously reduced scope, lack of trust...it goes on and on.
Building construction is also fraught with unexpected events and project management issues etc., but I'd laugh any building who would propose that I just keep paying them until it's done, or who will start building and go home when whatever budget I give him is used, out of the room. Why would my customers be any different?
(articles like this poison the minds of young impressionable developers who think it's actually reasonable to have ideas like this, btw, which is why we shouldn't just shrug them off)
In practice, though, this is exactly how building construction and civil engineering works. After all, if the bridge is half-built, what's the customer going to do? Walk away? When was the last time you heard about any large civil engineering project coming in on time or on budget? It's to the point where we're relieved when the project ends up within an order of magnitude of its projected cost.
Maybe sue you for liquidated damages whilst pursuing all other possible avenues in the contract for claiming against a breach. Maybe file an insurance claim, and weather the mobilization costs for another contractor to take over your scope.
But before any of that you first have to estimate the potential legal costs and determine what's going to work more in your favor. :)
As I understand it (still pretty new to the industry), we're supposed to give realistic estimates for a product's completion, considering that 1. requirements are always incomplete and 2. unexpected complications will always arise. 'Done' is a moving target, and part of its estimation is ensuring the product is deliverable at every stage (with varying degrees of completeness), regardless of whether how many times everything goes pear shaped.
And even more developments end up hilariously past their budget on both time and money. "Megaprojects" like the Big Dig[2] are common examples of this.
These are the failures big enough to be notable. I'm sure there are far more smaller, less interesting but more indicative failed projects as well.
[1]: https://en.wikipedia.org/wiki/Fontainebleau_Resort_Las_Vegas
[1]: http://www.bloomberg.com/news/features/2015-07-23/how-berlin...
Seattle's current pear-shaped project. Check out the missed estimates!
The complexity is just many many times bigger when it comes to software projects. With bridges it's actually fairly straight forward and the budgets are often of a completely different scale.
Software time/cost estimates are exactly that: estimates not guarantees. We tend to think of such things in terms of project management and contract enforcement. If anything, that is where we go wrong. What we should do is not try to understand the whole project in terms of an unalterable roadmap, but rather as a guideline for managing expectation and for informing decisions that are made during the course of a project. I use my estimates to sell the project, yes, but I also tell the client that as we go along we will be making many decisions that will move us away from our initial plan as we better understand both the issues and opportunities a project affords. Sometimes spending more time than planned will make sense, sometimes it won't and we cut the feature. Sometimes we find that we have failed to identify important features and sometimes we identified a feature early on that isn't as important as we thought. The overall project plan and estimate does set a benchmark to ensure the economics are still correct, after all if we lose that we lose the point of the project entirely, but we don't turn that plan into dogma. We don't lose sight of the reason that caused the project to exist.
So, yes, we need estimates no matter how hard they are to get accurate. What we need to stop doing is equating that estimate as an oath to be upheld regardless of what new knowledge comes along.
On my current project, I am the sole full-time developer. The requirements were gathered from the client without my involvement. The schedule was laid out without my input. Then the whole thing was dropped on me.
If your new-fangled-to-you development ideology starts by assuming a fundamental level of congenial cooperation between different departments of your organization, then you've designed your system for a fairy-tale world. I don't suspect most people are able to change organizations easily. I suspect most people are stuck dealing with the hand they've been dealt. The lies they were told during their interview. The other employees that lied during their interviews. It's a really messy world out there.
It gets even more complicated because, at least in my experience, software processes are really subtle and it is extremely easy to appear to be "doing the right thing", while at the same time failing miserably at it. Throw in the power politics, the bad actors who don't really care if something is successful or not (as long as they can extract money from it), and naive developers who are absolutely sure they know what to do but really don't, and it makes it very difficult.
It is difficult to be the one instigating process change. Such a role is fraught with danger. You will be attacked by the power politics people. You will be attacked by the money extracting people. You will be attacked by the naive developers who think they know better than you. You almost certainly will not be thanked by anyone (except very far down the road if you somehow manage to succeed... and even then probably nobody will remember it was you who started the ball rolling). And finally, because it is incredibly hard and the necessary skills are very difficult to acquire, you will almost certainly fail many, many times before you ever succeed.
Even worse, it's not something you can write down and say "Do X,Y, and Z and everyone will be a happy family working hard to ensure success". It is something that requires dedication to understanding every single individual who is involved in the development. You have to sit and listen to them, understand their goals and help them meet those goals (even if they aren't "Make a successful software product" and even if you personally think the person is unbearably awful). You have to devote yourself to finding common ground and bringing everyone together.
Don't get me wrong. The processes and practices are crucial to a successful organizational change. These practices, if well chosen, will support you as you influence, cajole, flatter, criticize and motivate the people around you. It is next to an impossible job and without these tools, you will almost certainly fail.
It is unfortunate that people dramatically underestimate the difficulty of cultural change in an organization. They write about the easy bits (the processes that support the change) and sweep the hard bits under the carpet. To be fair, I'm not sure what you would say that wouldn't sound like "business BS" to most engineers, though.
Anyway, most organizations do not have many people conscientious (possibly read "stupid") enough to attempt real, lasting cultural change. The vast majority that do try are often incredibly naive and lack all of the basic skills necessary to be successful. The end result is unfortunately all too predictable.
To anyone who is really serious about building an amazing team, I encourage you to suffer the slings and arrows of outrageous management and to work on the people skills that will eventually bring you what you want. It sucks for a very long time, but as you get a better understanding, it can get better.
I am a proponent of the idea software is a new form of literacy. We generally do not project manage the next paragrpH or the next chapter. writers are often faced with deadlines, but rarely have to justify themselves on a daily basis.
Improving estimation will just result in slightly better inputs to a process that at best hinders us and sucks creativity and joy.
I have to give estimates all the time. They're estimates, but they're in the ballpark.
But recently, I had to give an estimate for porting some software to a new environment. I knew it was going to present me with a bunch of obstacles, and I didn't know what the obstacles were, or how long each one would take. I didn't even know how many obstacles there would be.
I didn't want to give an estimate for that. They made me give them one anyway. I did, but I knew that no reliance whatsoever could be placed on the estimate. Whereas other tasks, estimates were close to "bet the company on it" solid.
I am not saying project mgmt does not work, I am saying the plan the task, do the task approach is not appropriate for software development, at a given level of granularity.
If you ever watch Grand Designs, they start with a design, a plan, a schedule, and it all goes out the window ten minutes later. This is not that being on top of the process will not give you control, but that relying on estimates and plans as a means of control is worthless - and that's for something people have been doing for hundreds of years
Also, how did you come to the conclusion that "project management doesn't work" is a historical fact?
Also also, if your teams' estimates are a farce you might consider starting to track time spent on projects and build up a base of real world numbers you can use as guidelines for future estimates. It worked wonders for us.
What IS common is a combination of poorly defined projects and a profound lack of self-awareness. By all means, push back against poorly defined projects ("I can't estimate the work until you tell me what it is."). Lack of self-awareness is squarely in YOUR lap ("I don't understand why it always takes longer than I thought it would..." Well, duh, maybe you should consider that reality for what it is and account for it next time).
Estimating work and committing to those estimates is what a professional does. Deal with it. If estimates are imposed on you that you do not agree with, that has nothing to do with the virtue of estimation. That's a social problem, not a technical one.
Of course it is constrained, but how can you estimate it? Because "software development" is the continuous process of converting a real world idea into instructions that are so tiny and specific that a computer can understand them. Obviously you can't know in advance what those instructions are, or you've already written the program. When do you stop "pushing back against poorly defined projects"? You can literally continue asking for more specifics until the project is complete. This is different from other areas of engineering, because almost NOTHING demands preciseness like a computer does.
"Estimating work and committing to those estimates is what a professional does. Deal with it."
The reality is, we have been trying to deal with, for decades. And failing badly. The author is suggesting (probably rhetorically) that perhaps we need to abandon the idea entirely, because it is not working (for very large projects).
It makes sense when there is a large cost to not meeting a deadline. Or when it's difficult to change scope.
If I'm doing prod support for three different products and build work for two products, then I'm naturally going to be juggling priorities. There are a lot of different and changing variables in the mix and it makes committing to deadlines very difficult.
If an organisation wants to isolate developers and have them work on only build work for a single product estimation becomes much more accurate. Of course most orgs would rather less accurate estimation instead.
I agree with you that estimation has value, especially at the large scale when you are trying to budget for a multi-year project. (and at that level- the detail you think should be there simply isn't) You are also right that at some level it is a social problem, estimates and commitments are two very different things- and you don't just commit to an estimate. Estimates are predictions, statements of probability about the unknown future, hopefully based on solid data from past performance and some understanding of the work to be done. Commitments are promises to make things happen that are only partially within one's control. In many cases, an estimate is needed prior to getting the level of information that one would need to make a truly accurate prediction, so we respond with a range of dates to reflect that uncertainty, but this can be too early to make any kind of commitment.
If you estimate at the point in time where you are 50% likely to get the job done, and commit to that, half of the time you will be late, even though it is "average time" it would take to do that work. You can't take the easy way out and commit at 100% likely time, because you could be sick in the hospital tomorrow (and with a big enough project, someone eventually is). If you commit to an 90% confidence estimate, you get stuck with Parkinson's law type problems. Not to mention that those predictions with percentages are hard to base on real data...but if you make enough of them, you can actually get feedback on your accuracy.
The problem is that most audiences for estimates aren't able to deal with the uncertainty, and many givers of estimates are easily pushed into overcommitting. This is why some of the ideas popularized by the Agile movement seem to work- if you fix the time increment, it is much easier to flex on the scope, especially when it is not possible for it to be well-defined in advance. For example, there are many cases where an high confidence estimate can't be made- as is often the case in more research oriented projects, where the path to a working solution, or whether one will even be found, is unknown, and it is part of the job of the project to discover that. You are stuck with estimating how long it will take you to try approaches x, y, and z.
McConnell's classic "Software Estimation" probably says it better. See ch 1 of McConnell's book here- page one is on the commitment/target/estimation problem: http://www.stevemcconnell.com/estimation-01.pdf
What I get instead is a stack of vaguely worded "user stories" and, if I'm lucky, some wireframes that will invariable wind up looking nothing like the final design.
I would argue that that would in fact be the source code for the program!
I know that sounds flippant, but - civil engineers and architects will correct me I'm sure - a blueprint to me is a set of pretty clear and hopefully unambiguous instructions, just not necessarily in a chronologically ordered list format. To torture the metaphor even further: the act of building the building from the blueprints is the same as running the program.
The key difference is that in the case of building a computer program the hard part is writing the instructions and in the case of building a building the hard part is execution of the instructions. Building designers have the advantage of centuries (millennia, even) of knowledge and some pretty hard limitations set by physics. Money and time aside, computer program designers are more or less limited by hardware and imagination.
I really do hope someday programmers will look back at conversations like this and think not that their counterparts of the past just needed to deal with it and be more professional but that we just didn't know any better way, yet.
In general, unless you've done something very similar before, estimation can't be done accurately. Your manager should know and accept this.
The best you can do is probably this: Repeatedly split the project into tasks, estimate the time it will take for each, until the next task takes several hours, and then do the next task. Keep track of estimates and actual times spent. Your manager should ask you to do this and keep it updated. Joel Spolsky wrote something similar here: http://www.joelonsoftware.com/articles/fog0000000245.html (I'm sure I read similar advice elsewhere, but can no longer find it.)
It should go without saying that you should include unit testing, code review, debugging, etc. in your estimates. And, of course, making and updating the time estimates.
The project, when "complete", will still usually require maintenance, so work on the project only stops when the software is no longer used.
You should never be given deadlines, as you won't know in advance what problems you might encounter, and the quality of your work (and probably your health) will suffer. This will result in a build up of technical debt, increasing maintenance costs.
It's well worth reading The Mythical Man-Month, especially if you're managing a project, or working with others on the same project.
I've never worked on two projects with the same team. There's as much learning about how to work with new, particular individuals as there is to learn about what the client needs.
Insomuch that anything has remained constant, it was pretty much just the underlying programming language, but even that has changed dramatically over the years, as well as opinions on the best way in which one should write it. Certainly my understanding of it has drastically improved.
And I'm not exactly sure what you're suggesting by "raw data". What sort of data? Collected and stored how? Reported on in what ways?
Now, I don't work in software engineering, I work in traditional engineering, but I find the lack of any formal Risk Assessment, Management or Lessons Learned in any Project Management methodology/framework a little scary.
One month? No, six weeks. Two hours? No, four hours.
The reality is typically somewhere between the two numbers. So the estimate remains fairly accurate and most things are finished ahead of schedule.
There are too many variables that cannot be accounted for when giving an estimate. How often am I going to get interrupted? Is there going to be legacy code that causes a bug to surface and has to be refactored? Etc.