Why are software development estimates regularly off by a factor of 2-3?
michaelrwolfe.com
michaelrwolfe.com
Developer estimates are regularly off because they seldom impact the developer directly. Experienced development managers will pad the hell out of the developers who give them the worst estimates. Most developers will explain all the complexity of what threw their estimate off without acknowledging the huge mistake of not anticipating extra complexity in the first place.
My estimates in my early career were no better than anyone else's, ie way off -- especially for more complex projects. I'd explain what happened to my managers and soldier on. The very next task that came up, I'd give my manager some best-case estimate of how long something would take and the cycle would begin again.
That all changed once I started to do consulting for myself using "not-to-exceed" pricing. The first multi-month project I did killed me. My effective hourly rate went down to sub McDonalds levels and took much longer to deliver than I had expected. After that project, I did a post-mortem on the project to figure out where I went wrong. I came up with several spreadsheet templates and checklists to run through before giving any more estimates.
Mostly I just concerned myself with getting a hell of a lot better at estimating project duration and difficulty. Like most things, when you really pay attention to it and practice it, you get better at it.
I think I most disagree with this. Developers, in my experience, are quite often asked to leave the company after such estimates.
It's good to have a template - it helps not to forget things - but it is inherent in the work to have unknowns. Large unknowns. A good consultant would probably refuse to work on a project which would require him to use several major technologies which aren't familiar to him, but developer in a company may not have such a luxury as selecting a kind of a project.
I've seen developers let go because they weren't very good at writing software. I've seen developers let go because they were painfully anti-social to the point that they were negatively impacting the rest of the organization. I have never ever seen a developer let go because their estimates were crappy.
The developer's ability to estimate accurately is not an significant factor in this scenario.
At the VP level, you're expected to understand what the Developers and Development Managers don't know about the SDLC. You're expected to bridge the gap in their lack of understanding by instituting processes and controls while mentoring them to become better at what they do so that the organization can succeed.
Frankly, I would expect such a company that employs those practices sees two things:
1) A very low retention rate among employees, with a lot of churn.
2) A risk-averse organization, heavily skewed to under-promise and just-enough over delivery.
The trouble is now though, my competition seems to be underbidding me, but in reality they're providing those ~33% estimates they will never realistically keep, while I'm at ~100% estimates. Not really sure how to relay that to clients. One of the many reasons people like me need a salesperson in front, I guess.
Right, that was part of my method as well. I learned to keep really good notes of each part of the project. At the end of each project, I compared my "what I thought it would take" with "what it actually took". In subsequent projects, I tried to match up similar complexity items with "what it actually took" notes to remind myself of the pain.
I don't have a solution here though other than noticing that underbidding and then getting skilled at convincing clients to do paid extensions later actually appears to make the most money at the cost of your ethics. I'd avoid that approach, but it does seem to work for a lot of companies.
My personal approach is to quote for very bare projects with only the bare essentials (eg, poor UI design, minimum possible feature for the client to see what they're asking for, etc). This can usually be done a lot cheaper than most people think as 90% of the work is in the last 20% of the features. Then once the client has something, you can give them a quote to touch up the parts they need. Basically you split the project up into many small projects each with their own quote which helps you to estimate tasks as they appear and helps your client to minimize costs by leaving off features that are more expensive than they initially appear.
I would just as soon not keep customers only interested in the lowest price. They're the ones that typically will be the biggest headache.
It's a great position to be in, provided you have the throughput to expand to other clients and aren't dependent upon a single client for income.
On the other hand, giving realistic estimates I often hear clients say, "I asked around, and other folks say this should take half what you're quoting." Usually this happens before the work, but I once had it happen in the middle of a long-term engagement after I delivered a two-week invoice, and they were disappointed I hadn't accomplished more. So if I'm not careful I can lose my clients' trust, which is my highest asset. I've gotten pretty good at explaining why a job isn't 5 minutes (or 1 day, or whatever), but I might save this article as something to share if a client doesn't have much experience doing software projects.
That a project is estimatable has to mean that novelty is low and predictability is high. But low novelty is a sign of duplication, which should be factored out into things like libraries, frameworks, and self-service tools. High predictability is either a sign of duplication or it's a sign that people have basically agreed not to learn anything during a project. If you are shipping early and often and study users and metrics as you go, then you will always learn things that affect the plan. Which means that the schedule isn't predictable even in theory.
Besides, most of the things you mention can be accounted for. If part of your development process involves iterating through the design with the customer every couple of weeks then you build that into your estimates.
Likewise, I always build documentation and code handover support into any serious project rather than acting surprised that customers will want such a thing and expect it to be part of what I'm delivering.
I think we have different intuitions here because it sounds like you're more in a service business than specifically writing software. I agree that a lot of any service business is standardizable, because it's mainly about people and their needs; that has a lot of regularity.
But I don't think the software creation part of a service business is standardizable over the long term. During the first wave of "put smallish businesses on the internet" each web site was custom, hand-rolled software. Early on those schedules were unpredictable, but for a while, it became a known, predictable business.
That business has, in the long term, been basically destroyed. People spotted the regularities and developed common code and tools. What was mainly a problem of software development became a (much smaller) problem of installation and configuration. The competitive advantage for those people now lies not in coding ability, but in customer service and in helping people manage the essential complexity of the domain.
Perhaps you weren't around in the early 2000s when the RAD tools were all the rage.
Perhaps you missed out on the colossal frameworks of the late 2000s when everything was a factory and understanding HTTP was actually a disadvantage as the whole thing would blow up if you actually tried to access the request body.
We went down the road you talk of. It was horrible. Now the pendulum swings to the opposite ends, the light-weight APIs which don't try and abstract away all the details which it turns out mattered a lot as everyone has to do pretty much the same thing, but ever-so-slightly differently. Tiny tools that do one job well.
All you are talking about is writing HTML and doing server config, neither of which are software or programming. When someone in 2000 came to you and said 'I need a website, therefore I need a software programmer to write html and setup a server' they no more needed a programmer back than then they needed one today. Just back then it was developers who knew the markup language and how to configure servers and they weren't about to turn away silly money just because there was not much actual coding involved.
I like crow-flies vs hiking analogy. I'd like to add upon it for so-called "green field" programming as it relates to getting cost predictions correct and understanding programmer time growth in relation to program size. Some parts are linear. Some parts are exponential.
Before I go on talking about "hello-world" space, note that I do architecture and really like architecture, big-picture solutions etc but I try only to solve what is apparent and then iterate on it whenever I notice two pieces tangling. There are excellent wins to be made here, but a working program itself constitutes part of the information necessary to arrive at the final architectural decisions. This is why even on FOSS code, I try not to sweat the architecture on the first pass -- when I do, dead code results. It calls out to me that I have solved something not relevant to the emergent implementation.
Once I've decided what the API might look like, what data structures have to be accessible to what, and what the minimum program states are, I try to stay in what I call "hello world" space. The idea is that the code I'm writing is never doing anything more than one problem at a time, line by line if I don't know exactly what a line does.
This is incredibly efficient because I deliberately break my problem into something testable and well understandable at every step. However, it's impossible to write production code this way. Not only is test feedback, (printlining and frequently more complex testing) not part of the final behavior, but more importantly, the elimination of concurrence of many states, the overlapping of problem workflows in execution sequence, implicitly says that "hello world" space problems are not production code. They have eliminated some functionality or consideration so as to make "hello world" solutions utterly oblivious to concurrent states or processes.
However, two "hello world" problems that need some coordination themselves creates a third "hello world" problem to implement their coordination. The third problem isn't apparent until the first two are explicitly solved. This implies hiking from the article.
Solving "hello world" problems brings more of the problem domain into "hello world" space. The concurrence problem itself will become apparent as a "hello world" problem when it has manifested itself. If there are two states that have been independently implemented but can overlap in execution, there is at least one set of logical statements for dealing with the maximum concurrence of the two states. Growth for N states is exactly an NxN truth table unless some of the states are sparse. In the worst-case, for N "hello world" solutions, one layer of abstraction where all states depend on all other states will result in N^2 logical blocks. While in practice this concurrence is usually much lower, the growth in abstraction is exponential if one is to keep breaking each problem down into an independently writable, testable piece.
So my conclusion is that growth is pretty much linear in proportion to writing "hello world" problems and exponentially proportional how many layers of integration are necessary. Slamming code down ichi-geki style is linear for problems that don't have a potential for overlap and geometric for ones that do. Code that eliminates duplication of routines through logic creates them in implementation, but the payoff is a net gain in many ways, so of course it still makes sense to do it.
As for making predictions, while running my internal monte-carlo, the concurrence of states has a higher influence on the outcome sometimes than the raw code size I expect to come out in the end. Linear and exponential growth occur in different areas. Recognizing that each integration incurs smaller linear component, and that each abstraction layer involves N integrations of solutions, I think it's pretty achievable to have a damned good estimate. The exception is if scope -must- creep and this is only determined in the middle of a project. Scope creep gets paid. Bad estimations do not. Usually scope creep is itself revealed when a new facet of the problem comes up, one which represents a competitive opportunity, which the client is happy to pay for.
Saying that no estimate should ever be good is like saying programmers should make gigantic mistakes. This is only possible when working on W-2, and only when the company won't die. End of discussion. Independents have to get it right. Any thoughts to the contrary will soon land an independent programmer back in W-2. The fact is that clients like predictable cost, and programmers who can deliver it can ask for more because they don't have to start squirming around in client meetings later and making justifications that are inevitably open to self-serving padding and other temptations. Be brave and keep scope, or go W-2.
The big unknown - and the highest risk - was managing integration between external systems, or doing complex integration between internal systems. Anything that might benefit from aspect-oriented programming, like access control lists, or a more elaborate front-controller, or universal exports in json format, or a new paradigm like AJAX, would be difficult to estimate.
That fits your breakdown of projects into linear parts, and exponential parts.
Anyway, my estimates were good, but my business management was not - I was always in learning mode because I never studied business and had no experience. So I went W-2 and went into a different field because an opportunity to learn politics emerged. What's interesting is that successful political campaigns are entirely about estimating and budgeting, in real time more or less - you spend money and see results day by day, or after a couple weeks. It's driven by statistics (and messaging). I just watch from the sidelines, and don't do politics, but I think we could learn something from it. It's more like agile XP than waterfall.
The Obama campaign was XP, by necessity. The Healthcare.gov website thought it was waterfall... again by necessity.
I spent a couple of years traveling around to visit friends and family all over the US, and regularly my estimates were off in when I'd arrive somewhere. My software delivery estimates are similarly not perfect, in the later direction like most people. My conclusion was that my estimation of how long it takes to do anything is off a bit. For me, this is because I'm trying to say I'll do something sooner to meet the perceived needs and desires of others.
Plane, train, or for adventure, Greyhound. A sailboat makes more sense. A bicycle makes more sense. And if you insist on walking, at least follow a fucking road instead of cutting cross country.
If you decide the first step to driving to LA is mining iron ore to smelt for casting an engine block and building a Bessemer furnace for the steel for hand forged pistons, then there's lots of shit that can go wrong, EPA permits and zoning laws, even if you've done done it before.
The reason software production estimates are bad is because they aren't taken seriously. The guild of programmers can say, "we shall get from A to B by cross country hike."
Yes, the analogy is perfect because its absurdity is acceptable.
Because of this I offer two estimates: one that might be off by a factor of 100x, and those that are fairly accurate but take more time to make. My boss usually goes for the former.
Edit: also there's two types of estimates. One measures how much of my time it will take. The other measures the date it will be done and/or live. The two are oftentimes different numbers, and as a developer I can't always control the 2nd.
Then you get an urgent message from your customer: you weren't supposed to hike from SF to LA after all. You actually need to go from New York to London, on a submarine. Specs require that you do this without surfacing or refueling. So you need to design a nuclear reactor, figure out a way to put it on the leaky old U-Boat your customer just happens to have lying around. All within the same timeline as before, and in the same budget.
You've had all this happen to you a few dozen times, so you factor in the time you know you need to build submarines (all just different enough, of course, that you can't reuse parts, even if the customers let you adapt, say, the propeller shafts to a new customer's design: I hear that in software, this is called "DRY" and "using libraries and frameworks" and "refactoring") — and you give an honest estimate: "it'll take eight months of full-time work, and will cost you about X hundred thousand dollars (a bargain for a nuclear submarine, if you ask me)" — and the customer will then go to someone else, leaving you wondering how you'll pay your bills.
There are simple methods to combat gross inaccuracies in the client specification. You document the requirements for what you'll be delivering in a carefully worded Statement of Work that the client signs off on, you ask to be paid in chunks as you go with a portion up front, and you implement a system of change control so that it's totally clear if the customer is sending you off in a completely new direction. At that point, you book what you've worked at your hourly rate and you submit new work and time estimates for the changes.
Developer estimates are regularly off because "experienced development managers" are all too eager and willing to allow organizational politics and external influences to add complexity in ways the developers cannot anticipate. Inexperienced developers' estimates are even further off because of their relative lack of experience.
I assure you, the first time a "development manager" tries to scapegoat his subordinates for his own failures to deliver to these external influences his subordinates realize that their estimates impact them eventually, and directly.
Nice try, but no.
Developers who are DIRECTLY affected from missing estimates (e.g one man shops paying out of their own pockets, people building an app while having quit work, eating ramen and maxing their credit cards etc, developers in companies who will personally pay for any delays by attrocious, unpaid overtime crunch marches, etc) underestimate the time needed to get something done just the same. Just ask Allan Odgaard (of TextMate 2 infamy).
Now if I am to hire a developer I will never think that "his estimates are bad because he's not directly affected by missing them"!
You're like the expert sherpa trying to convince people to use your message delivery service when the internet exists.
As a developer, I have pressure to keep my estimate down. I can give you a guaranteed date with few problems, but the project is going to end up more time-consuming than what another developer is going to estimate.
The good news is that I will usually significantly beat my estimates under this system. However, I will then be accused of padding my estimates and will be given more aggressive targets, some of which I will miss.
The best way to combat this is to give me a team that has worked together on multiple projects within the domain, and account for changes and ambiguities as part of the process.
If we drove cars like we develop software, we would pre-calculate a list of instructions, set a timer, and then execute those instructions without regard to what's actually happening on the road. With practice, you could probably get better at making those lists of instructions, but why bother? The general approach is just ridiculous.
Decades of experience have shown that there is enough inherent uncertainty in software development that trying to plan everything from the start just doesn't have a high success rate. Even if you know everything that has to be done today to get the job done, your environment will change, and your stakeholders will change their minds. What good is software that perfectly solves the problem you had two years ago?
Once you dig through all of the buzzwords, "Agile software development" is simply about applying closed-loop control theory to software development: You structure your project to provide high visibility and to allow frequent changes in direction, then you iterate and make adjustments until the result is satisfactory.
Everything else generates unnecessary risks that somebody ends up paying for.
If you don't want to hear the truth, you won't. Humans have known that since long before software engineering existed... or a subset of them, at least.
I do work in a place where we are a software company but very few people, even some of the devs, understand this culture. I don't say something is complex just for the hell of it. I want to give good estimates but at the end of the day that is all it is, a somewhat educated guess before I've gotten in the weeds. I don't need people that absolutely don't understand this concept to mandate a timeline that is unrealistic. It does absolutely no one any favors. That is why I leave. I don't need to promote unrealistic expectations down the line to customers who think everything is extremely simple. Especially when all we do is create custom software where almost 0% is turn key or off the shelf. Its just insulting to continue that nonsense and the quicker I can facilitate a reality check for all parties, the better and more trust is given to my judgment (and theirs if they actually take the time to learn)
This is actually achievable in a sane way.
I have done it using the XP planning practices. Basically, you make the suit break everything down into relatively small lumps and place them in priority order. Every week, the team completes a few lumps. Before you do them, engineers grade their relative complexity in arbitrary units. (The smallest substantial thing you do is 1 point; something twice as big is 2 points, and so on.) Every week, you count up the points completed. That's your "velocity".
From there, you let managers do all the estimating they want to. If they want the complexity of a unit of work measured, they ask the engineers. If they want to know when X will be done, they look at the team's recent velocity, what's in the queue before X, and do some basic math.
The nice part about this is the mental judo involved. Whenever they want to know when something will be done, it's their problem to trade features against time. Working like this, it's not geeks vs suits; the suits channel their schedule pressure into productive work: grooming the backlog.
You write "I'd rather them take my input and do their job of understanding trends." that's exactly what I'm talking about. They need to take your input, and do their job.
If a worker is told: "Here is what you (personnel) have to do (task), with these tools (environment), and we want you to do it in this amount of time (money)", and the worker fails to complete the work in the amount of time, then it is management's fault. They improperly estimated the amount of time it would take the worker to perform his task.
That management ask the worker how long the task will take means management does not understand the task, and if management does not understand the task, how can they know whether the worker's skill, experience, and knowledge is adequate for the task? Now, they also seem to not understand the worker's skills, experience, and knowledge, so the problem is compounded.
It is a management function to clearly define the tasks that must be performed, attract people whose skills, experience, and knowledge fit the tasks, and fund these people with the appropriate environment, tools, and salaries for them to perform the tasks.
That is the role of management. If they are unable to estimate what tasks must be performed, what people need to be retained, and how much money and time will be spent, they fail at their managerial duties.
That they then blame the workers themselves for improperly estimating the work is doubly wrong, a sign of managerial immaturity.
Unfortunately, it gets even worse if developers have no part in making estimates. If management doesn't know the details of your architecture and code-base (or even the other projects you may be working on, if you go high enough up the management chain), they have no way of coming up with a rational estimate on their own.
Where I work, VPs are constantly making "commitments" to customers to have certain features implemented by a certain date, and only notifying developers after the fact. Then they wonder why the development organization is "so bad at meeting deadlines". (Needless to say, I have no ethical requirement to honor commitments that weren't made by me and are frequently impossible to meet anyway.)
One of my big aha moments about estimation was a bit in McConnell's Rapid Development. He pointed out that most estimates get made with executives pressuring for short numbers. When you iterate a few times with that, you end up with the smallest number that developers can't absolutely prove is impossible.
If you draw out a bell curve of probable completion dates, this is basically the same as picking one far to the left, so you have a 95-99% chance of being late. But somehow, executives are still surprised when their totally biased approach yields the numbers they liked, not the numbers they needed. All the estimation effort was pure waste; our time would have been just as well spent making cotton-candy raincoats.
I now have done a few projects with little to no estimation, and it has gone much better. I have a hard time now seeing why we ever bothered.
Too many project processes treat estimates as commitments with no backup plan when they (inevitably) go awry. I think a better approach would be to design your system around the idea that all estimates are suspect until proven otherwise, which of course leads to iterative approaches etc.
However, you can't run a business without any idea when anything will ship. Estimation is still, in my view, a very necessary step, so long as there is significant incentive to get it right rather than get it fast, AND the business is receptive to a little ambiguity through the project life cycle. This is a very difficult balance to strike.
In my experience you can have certainty in project scope or time, but not both.
I agree you can't run a business without any idea when anything will ship. But to get a good idea when things will ship, you don't need estimates.
For example, in the Lean world there's something I've heard called "Disneyland scheduling". You break the project down into lumps. You maintain a queue of the lumps. You measure the average time from position X in the queue to being released.
Now you have a reasonable early warning system for when things will ship. Which is generally much better than what people have with most projects, which is a fantasy date.
I find that I often don't have a good feel for how long something is going to take until I'm about 1/4 the way through the project. If I absolutely had to produce accurate, high-confidence estimates, I would just do 1/4 of the work first, then estimate the rest.
As to why estimating is so hard, here's how I like to think about it. Tasks form a tree: each project has a number of steps, each of which itself breaks down into sub-steps, and so on. We tend to estimate by thinking about the tasks near the root of the tree; but the actual time required is proportional to the number of leaves -- tasks we can implement without further breakdown. Without planning the entire project out in detail it's hard to know what that number is. But to plan to that level of detail requires making many design decisions; and researching and making those decisions is much of the work of the project.
The bottom line is, good estimates are expensive to produce, and my experience aligns with yours: it's rarely worth the trouble.
What I think people try to get from estimates is a sense of control. But I think there are better ways of giving people not just a sense of control, but actual control.
You can think of it this way. A coast line is a fractal line. As you zoom in its length gets bigger and bigger seemingly. From the top level on the map it looks fairly straight.
Related to estimates. Programmers have and operate with idealized mental models. A FIFO queue works kind of like this ("/closes eyes and sees a line of people lining up at the store"), a tree traversal looks like this ("/closes eyes sees a red cursor move down a tree drawn with blue nodes on an white board"). Same with estimates. How long will this take? ("/closes eyes and imagines an idealized sequence of events that will lead from now till project completion -- use this library, develop that API, test a little bit here, done... 2 weeks").
The problem is, zooming in is hard. "Oh this library we are about to use actually doesn't implement these corner cases". So now we are spending 3 days patching it. Oh, while working on this project, a critical support ticket comes from a customer. And then testing reveals a fundamental flaw in our API design. Re-write a huge chunk and repeat.
Some developers and managers just use heuristics. They do the mental estimate then x2 or x3.
^ All the above is true if the developer is everyone is honest and trusts each other. As other posts wrote, that is not always the case.
Then it is a more complicated game. The main question to ask then is "What is the punishment for overestimating vs the punishment for underestimating?".
Have people been fired for underestimating? No, ok underestimate. Have people been fired for missing deadlines? Ok then overestimate.
You can even play this game with yourself. What kind of deadlines do you set for yourself and how to you handle setting a too short or too long of a deadline? Do you get better with time at setting deadlines? If no, why not?
Haven't thought about it before but I realize I too use a heuristic of x2 or x3 when estimating my time. Nothing would stop me from getting a pen and paper out and stepping through the 'trail' in my head in pseudocode beforehand, but for most of my projects I'd benefit from just writing the real code from the start and apply the time I saved to the project.
Software is a lot like that.
[1] http://www.quora.com/Engineering-Management/Why-are-software...
Tell HN: Add ?share=1 to Quora URLs to display content without login
Why is the problem inherently hard? To see why, let's imagine that the All-Knowing Fairy Godmother of Software Estimation descends from the heavens and lends to us her magic estimating function F that, applied to any software project S, will tell us exactly how much time and money our preferred software team will consume to implement S. Don't worry about how F works, just believe that it does. (Okay, okay. Let's just say that F peers into a parallel universe in which our team has already implemented a software system line-for-line identical to S, and it just observes how much time and money were actually consumed in that universe. Anyway...)
Now that we have the magic F, our problem is easy, right? Nope. The real problem is that we don't know what S is. All we actually know is that our S, whatever it ends up being, must satisfy some set of fuzzy constraints C (commonly called "software requirements").
In truth, there are a countless number of possible values for S. In other words, S is a random variable, a mapping that ascribes a probability to each of the possible software systems that we could build to satisfy C. Therefore, our best estimate, even with the perfect estimating function F, is itself a random variable. In other words, it's not an estimate but a distribution of estimates.
See how fun this is getting?
But, wait, it gets funner. That's because people in the business world don't want a distribution of possibilities. They want a budget, a date on the calendar. So we must squeeze point estimates out of the true distribution. (And, remember, we don't even know what the true distribution is!)
That's where the second reason kicks in. Let's just think about the distribution of possible budgets for our distribution of possible values of S. Because S can range from "the simplest thing that could possibly satisfy C" to "the most insanely complex thing that a frighteningly gifted salesperson for an enterprise consulting firm could get our CIO to throw money at," that distribution is going to be w-i-d-e. From X to 10X wide.
So if we're the folks tasked with coming up with those point estimates, we could plausibly estimate X on the low end, or 10X on the high end, or anything in between. Guess which estimates are going to get us the most push-back from the higher-ups?
Now, we're swell guys and all, and we want to do a good job with our estimates. No question. But, still... We're human. If we think that giving a higher-end estimate is going to make us unpopular with the higher-ups, maybe we'll estimate a little lower. And when we get push-back on that estimate, maybe we'll "take another look at the numbers" to see if we "missed any opportunities."
And that's the old one-two. We start with a hard, fuzzy problem and then add to it the pressure to deliver only feel-good solutions. The result: estimates that are often way low.
Every engineer on my team has a characteristic "fudge factor" that represents his... optimism (say, from 1.5x to 2.5x). Once I apply those fudge factors, I add another 30% slack to the total schedule. And I usually still underestimate.
The team I'm on now has been doing a very good job compared to previous ones about getting good estimates for project duration, but the ability to underestimate is nonlinear in the project size, and most of our projects recently have been pretty small chips.
On our most recent tapeout, we did a new baseline based on a new feature request and underestimated the cost of that feature addition by a factor of 2. The total schedule was pushed out about 20% as a result. I still consider this a success in predicting the project's duration: I've been on projects where we taped out six months or more after our "commit" date.
"It's OK, it doesn't need any code changes, just a modification to the configuration files, so the approval process does not apply."
That company's developers just need to learn the trick of making the configuration file format Turing-complete.
If you don't have time to write your own interpreter, XSLT is an easy one to sneak by PHBs.
I've seen one system where 70-80% of the functionality was implemented in XSLT for this reason. It meant changes could be delivered in 5 days instead of 35. Obviously good news for the customer. The PHBs were pleased that the bug count in the C++ core had dropped (it hadn't really, we just stopped using the buggy modules.) The integration testers were pleased that the end product worked. Basically everyone was happy except the C++ reviewers in QA.
People hiring consultants to build them software are generally non-technical. On most projects the following problems may occur:
Problems With Clients:
1) They don't konw what they really want.
2) They are bad at communicating what they really want.
3) Not enough people from the organization are brought in to weigh in on their needs / actual duties.
4) Too many people are brought in, many with conflicting interests. (bike shedding). [1]
5) Critical omissions are made regarding idiosyncrasies of their infrastructure that are never brought to light until far too late in the project.
6) people trying to change the direction of a project late into development
7) complicated/redundant policies or procedures which are incompatible or inefficient to implement using the tools the client requires you to implement them in.
Problems with Consultants / Contractors:
1) Over confident estimates
2) They are bad at estimations / unfamiliar with the tools required to do the job
3) They are bad at communication / rooting out the the real problem early
4) Unable to detect confusion/bullshitting from a client or are unwilling to investigate it.
5) PMs who make promises they are unable to keep
6) PMs who do not understand the details and make decisions without consulting those who do
7) Developers who avoid asking the client for clarification
8) spineless developers/PMs who screw themselves to avoid confrontation
9) Procrastination
[1] http://en.wikipedia.org/wiki/Parkinson's_law_of_triviality
8 isn't just about confrontation, the person who says 'yes' is more likely to not be fired than the person who says no. You need to frame all of those times you say no as alternatives, not as flat saying no.
Within reason, I completely disagree with this. The best engineers and PMs are the ones who effectively manage expectations.
In short, on both sides of the old one-two punch, the hardware guys have got advantages.
1) When someone buys custom built hardware, they know what they want, and the specifications for it. When someone buys software, they want "something that does xyz, is just like facebook and paypal, but also has our logo." And then it turns out that the problem they're trying to solve is completely unrelated to any of those things. But you can just rewrite it to do those things too, right?
2) If there's a problem with the hardware, the customer that paid the money for custom chips will notice, and will have the technical skill to lay the blame on the vendor accurately. This means hardware manufacturers need to deliver quality, and since point #1 guarantees they actually have a specification, they can therefore engineer to it. In software, often the client takes your application, puts it on machines from the mid 90s that it won't run on, gives it to minimum wage employees to use (so management may not really care if it works), and then ask if its too late for the product to also be an app on their iphone. When they can't do anything with it 3 years later because it was written by people who have never looked at a computer before, the customer will just say "software always has bugs / always going out of date, find a new consulting company to write us a new one", as if its normal to have to build a completely new system to do the same thing over and over again every three years.
We don't do the same with software, though I would argue that we should.
Based on my experience, the more time you spend coming up with application prototypes that closley resemble final products, more accurate estimates you will get from engineers.
Because software is magic for a lot of people anyway that excuse just doesn't work.
Software is held to much lower engineering standards because it can be made much more complicated, by most measures of complexity; and complexity in hardware drives up development and per-unit costs, whereas complexity in software drives up upfront costs only, so most people push complexity into software.
Consider an all-mechanical watch. Even a watch that just accurately displays time and day of week is going to be fairly complicated. If you want the day of month to account for the length of different months, that's more complicated again, and if you want to account for leap years, especially the mod-100 years that's more complicated again. To say nothing of products like the Calibre 89 [1].
A product like Pebble will do all that without batting an eyelid, and a hundred other things that would be impossible to do mechanically. Setting the time automatically. Programmable watch faces. Scheduling meetings and synchronizing with other watches, phones and computers. Correcting across time zones and daylight savings transitions. Accessing real time train schedules and weather.
Hardware engineering is possible as physics is the boundary and since has become categorically managed by human's mind of structure and logic. A jumbo jet can be decomposed to millions parts each confined by own specification therefore once design is done the production timeline is easy to estimated.
Software production to this day is still struggling between modular development and any attempt to standardize on a way to create software is overwhelmed by thousands of new frameworks or libraries even engineering methodologies every day. The reason software has such "joy" is because not like material or aerodynamic physics, software and way of making software are frequently derailed by its foundation, computer chips new advancement. Literally, every few years whoever can exploit the new capabilities of computer hardware has a chance to become next Steven Jobs.
Granted, most of corporate enterprise software projects don't have to aim high and certainly can be managed like any consumer production manufacture, so long as people are willing to settle on certain software framework, engineering approach, and treat pre-existed software component like real "physical" component meaning quit thinking the possibility of customize anything, simply focus on assembling. This is the engineering propaganda from school of IBM, Microsoft and SAP et al have been trying for decades but reality is no one but the labor cost sensitive offshore software firm would be interested to this. Everyone in corporate world pretend they're working in a creative campus and fancy about the software they make is the core competitive edge of their employer. The price to pay is uncertainty, but the question to ask is if their employer is in the business is in software business. The hard part is even this question is full of uncertainty, who'd imagine Walmart will be compete on the software it runs based on its past business model, well that might change.
The art to be able to estimate accurately (heck, even Boeing failed 787 estimate) is to turn craftsmanship to production assembling line. Stop thinking creatively, focus on manage a collection of proven engineering approaches of "manufacture" features meeting requirement. Everyone in software industry should realize there needs a distinct separation between the creative making of software world and the basic business operation needs. The line is not static though, as Google's engineers constantly made IBM's scalable/reliable enterprise software suite a joke. Still, not everyone is google and effectively manage the evolution of this distinction is not easy but necessary, because while google is looking for uncertainty, most of corporate enterprise hate uncertainty.
Once accepted the confined world, it will be possible to maintain a model of software composition by different combination of modular components, pre-existed or further decompose. The process can be fully automated and emulated as long as the time factor is added to each component. The upside is this can be estimate/managed down to minute level yet the downside is that the software created would be boring lack lust, but built with complexity to support operation reliably which is enterprise software all about anyway. Most importantly, once the estimate and cost become predictable, managing software system evolution will become a much easier and lucrative job. Imaging how joyful will it become if you go to your CFO asking for next round of upgrade (or even revamp) due to the "demand" of some fancy new capability the business users have experienced from their personal consumer digital gadget, while your commitment to the money and time has a solid track record.
The key to resolution to the challenge of corporate software engineering, including the majority of creative product builder in start ups, is the lack of an effective and efficient way to manage the software evolution and the chaos it brought. Sourcing to less cost labor or adding more bodies to the equation will not make it better, except short term psychological benefit.
In sum, software industry desperately needs automation. Ironically, software starts as the automation king to everything else, but itself has been largely remained in the hand of artist. The good news is cloud and DevOp are closing fast to bring an end to the nightmare of software engineering, at least an interim until the day software is no longer made by human.
And then those crappy estimates are never revisited once more information becomes available.
You might end up taking twice as long. You might end up taking half the time estimated. Even with a perfectly honest estimate (which is rare) both scenarios are equally likely. So the overall distribution skews towards late delivery - you are as likely to require 2 more days, as you are to save 1.
I occasionally just paste this link with no explaination at the end of emails containing time estimates.
Another issue is when you have an "indirect estimate" - basically I will estimate the work, but someone else is the one that ends up doing the project. If you aren't careful to consider who will be doing the work, you might estimate too low (if you are an expert and the work is done by a bunch of new hires).
And none of this even touches on misinterpreting client demands, scope creep, dev team turn-over, or often neglected timesinks like documentation and meetings.
An Agile approach to this walk would have managed the expected target date much better and improved team morale by not overstretching each day ("sprint").
Lots of people think they know (and implement) Agile. I did, until a course I took about a year ago where I learnt skipping seemingly unimportant bits (measuring velocity, sprint retrospectives, etc.) bring the whole process crashing down.
I'm inclined to agree. Often though a client will want an estimate (to a ridiculous accuracy) before agreeing to the work. Unless you churn out web page templates or something equally repetitive it's hard to have that relevant velocity.
I still come up against client managers that don't "believe" in Agile approaches too.
Is it a missed estimate? That depends on who answers.
Never, ever say this.
Personally, I always give my estimates as a range. If you break down your project into small tasks, and estimate each task as an optimistic/pessimistic range, then you can get a pretty good idea of the expected completion date, along with the deviations from that which you might reasonably expect.
However good your estimate, committing to ONE number is madness - it hides the complexity of the situation, making it impossible for you or your manager to see the real picture.
To many in Management and accounting (and with varying degrees of pointy-haired-ness), profit centers "make money" while cost centers "spend/burn/"waste" money.
The sun shines on the former, while the inquisition perpetually stalks the latter.
Systems and systems software are often seen as a cost center. There is constant pressure to "control" these costs and to do more with less.
People are required to be over-optimistic in their communication while at the same time overly pessimistic ("frugal") in their resource allocation. Note the use of the word "pessimism". It is not so much a matter of "optimisim" thinking that "we can get it done with less"; rather, it is the pessimism of "we can't afford that".
Much of my professional life has been spent in big, "conservative" (cough), risk-averse organizations. I'm sure things are to some degree different elsewhere -- some degree.
Nonetheless, it's worth keeping in mind this (simplistic, and to varying degrees counter-productive) concept of "profit centers" and "cost centers". And if you are working for the latter, expect to be squeezed into ever more unrealistic positions.
It's something the seasoned professional stays attuned to. There will always be some squeezing; when it starts to worsen and become excessive, it's time to go. Your personal solution, as a "simple" employee, is to find another organization -- the sooner, the better.
The range is usually between "this is the likely amount of time with everything going as it should" and "this is the worst case scenario time with all that can go wrong".
It's a pretty broad definition between the two but I find it gives clients a sense of the scope of the project. The minimum time does include some padding to include all the usual likely overruns (testing, requirements gathering, etc.) but not too much.
The worst case time better really be worst case as I provide clients the expectation that it should not take longer than this.
I find that usually I will end up somewhere in the middle of the estimate and the client is still happy since they didn't hit the worst case scenario.
I find that this usually tends to be relatively effective even if as a rule of thumb the worst case time is 3-4x the minimum time.
Of course, there is still a lot of analysis done to come up with this range and there is a lot of discussion with the client to help them understand the details on why and how it could go.
At the end of the day, communication is king. I find that educating clients about the complexities of the process always helps to gain trust.
It's because estimates are really just a test of your imagination and skill. They're a "different game" from delivering something. Here are some examples:
How long does it take you to add 1 + 1 together? The only correct amount of time is "0 - no amount of time." But clearly that's not true, is it?
How long would it take you to multiply 5 by 7? Well, zero. No amount of time - you know it 'by heart'. It doesn't take you any amount of time to multiply 5 by seven. Well, clearly, that's not true is it?
How long would it take you to test whether 31337 is prime or not, by any means? Like, 1 second because Wolfram Alpha can tell me I bet. But clearly it wouldn't take me ONE second, would it?
How long would it take you to write a function to test whether an integer less than 2^32 is prime or not, without having to be super-efficient about it? Well like 10 seconds. But, clearly that's not true, is it?
And so on. How long does it take to put a twitter bootstrap page up? Like, 5 minutes.
How long does it take to...
You should be able to build an app that doesn't crash and does what you're trying to code in like a day.
You should be able to build something that is software you can launch with in like a week, as an MVP.
You should be able to replace someone else's mature product with a reimplementation in like a month.
You should be able to build a scalable business that is ready for a series A in like a year.
These things are not meant to be accurate, so much as show that you have imagination.
The "big" message is that if you have never done something before, you have no idea how long it will take, even if you can describe what you are going to do pretty easily. Returning to the hike metaphor, this is why you need to listen to the cranky old guy saying "walk along the roads, not the beach" instead of the inexperienced folks saying "Hey its just walking, and walking on the beach rocks! Walking along a road is boring and lame!"
The first time or two it made us hate the customer, and feel like it wasn't our fault. But then we started writing it into the bid, and telling the customer they could save money if they delivered conforming data. Hopefully we'll be happier from here on out.
Some of the groups are showing up on time and are waiting for you, some are late, all won't have all the supplies you need.
And to make it more realistic: half way through the hike, your friends call to say the lunch has been moved to San Diego and is now a formal dinner party at which you will need to wear tuxedos. And can you bring the Champagne as well?
It's hard to estimate something that is complex. If I am very confident the job can be completed in under an hour my assessment is very close to 100%. This is because something simple does not have a lot of variables and interacts with very few things. Things like fixing a display bug, a simple calculation, adding a print now button are all easy. Once you go beyond this it gets more difficult to determine how long something will take.
Business wants the estimate done quickly. Many businesses want estimates done fast. An accurate estimate for a complex job cannot be given fast. There has to be several rounds of back and fourth. Use-case scenarios need to be determined, which will open up even more questions. The underlying architecture to accomplish the feature also has to be scoped which will take time and usually we don't get it right the first time we think about it
Some more reasons are, businesses want the job done quicker than is realistic. Businesses don't know everything they want. They think they need stuff that they never end up using. Haven't thoroughly thought out the feature or problem themselves and that uncertainty gets passed into the development phase more often than not. Business is not solely to blame though. Sometimes developers think a particular portion of their code will be simple when its not or difficult when it isn't.
Basically the whole process is riddled with human error on both sides.
The other factor is feature creep, if you find a project so well defined and set in stone then it would be fair to say it will have a better chance of finishing on time.
That is just the software side, then there is the hardware to run it and that opens up a whole new aspect of poetntual delays that will get leveridged at the project and when people say project behind involving software people will just focus on the software aspect and all other factors are buried.
Bugs and issues come from many factors and even suppliers of hardware can have bugs which can have a impact, design could have bugs and many aspects can have issues that impact the end result.
But it is impossible and silly to plan for every situation as whatever you plan for there will be an exception and you get a deminishing return, hence easier to have a area of time labeled contingency as a catch all. Best example of such exceptions would be say an earthquake killing your entire development team, that can and has happened and at the same time is not something you plan for. Partialy why we have insurance. Though you will find it very hard to impossible to get insurace again software development delays beyond life/death of parties involved, though that will not chance the delay into a non-delay, mearly compensate.
With all this Engineers have for many lifetimes gone with the engineering factor or whatever you think it will take, muliply that by 2-3 to counter optimisim. But that does not help feture creep and anything added to the design after code has started is creep.
Anyway, nice article!
a) For one, giving enough time to come up with a project estimate is often rare. Usually people try to ask for those upfront.
b) Usually we try to add a general buffer for any unseen issues. However, the deeper you go and list tasks against time, the closer to realistic is the total.
c) Often it is just a customer giving a requirement needed at date x, or a management guy setting goal for a date y. When all you can do is to cut features but have to deliver by a given date.
d) No matter what we say, but at the back of our minds, we are not enough prepared to defend a timeline when giving it in the first place. Implementing feature x needs a week? gosh boss would say it should take a day. Latter it turns out the assumption was based upon a library which for just internal implementation reason (that you only discover upon learning it enough through actual code) is completely useless and it should actually be planned for 3 weeks at least.
Everyone who has been a part of a software project knows about poor assumptions. From the start, you don't have well-defined requirements. You don't have a complete design. You don't understand the consequences of decisions you have made or will make. Any estimate is doomed to fail in the face of all these poor assumptions.
Staggering a software project into smaller chunks goes a long way to controlling these poor assumptions and the impact to schedule and, more importantly, customer expectations.
The principles of the agile manifesto go a long way for mitigating the too common pie-in-the-sky wishful thinking that dominates software engineering. They work even better when the customer realizes they are a part of the system under development and its success - if they can keep that balanced with giving the development team the space to do what it is they do.
Devs make mistakes, sure, but as often as not in my experience it's absurdly vague end user requirements married to unrealistic expectations.
Most decision makers have to choose some number for ETA however because world does not understand probability distributions or variances. Most likely this has to be lower end of the number to get project sponsored or fend off the competition.
This is not unique to software development. Any projects that requires large connected graph of small task nodes would show same characteristics. For example, fighter jets, Boeing Dreamliner, space station etc etc.
In practice a good strategy for companies is reduce unknown to a minimum, the Steve Jobs strategy, just focusing on very few products, with each having the less complexity possible(simplify and simplify).
Only focusing on very few products you could make perfection possible, estimates will be off, they always are when there is unknown, but at least completion will be finite (you will complete it!, which is number one issue with research work for example).
Sounds simple, but is really hard, leaving creative people on their own is dangerous for this reason, they could start new projects, not finishing anything.
Now, if you think about that metaphor for a while you may find value in seeing: (a) the metaphor doesn't really fit intuitively because software projects are often charting new ground, not zooming in on a predefined complex problem (b) the metaphor is not too bad of an intuitive explanation -- you learn more about what is really needed as you get closer to the details.
Hence the completion estimate has to be continuously refined as the details of implementation are continuously refined.
Just like the slow zoom-in on the hike.
* Take a day to come up with "estimates" for the project,
can't bill that time of course.
* Sell fantasy gant chart to the customer.
* Take 10-20% of the projects time away for "project management".
* Actually develop the software.
* Fix "bugs" until customer goes silent.
What about the agile idea that you deliver something incrementally useful with every sprint?
Obviously, some projects can't be done that way. But if they can, it would mean 1) estimation can be done on a 2-week scale instead of maybe a 6-month one, and 2) they can start deriving benefit/making profit from what you've built while you build the next thing.
The developers estimates generally are best case scenarios without any external or motivational factors included.
For most people it's around 2-5 but for some as high as 10 to 20.
You might feel like an idiot when you tell people "Hm, change the color of the logo to a different shade of green? Six hours." This is probably how long it will take, though, for that so-called "five minute fix".
estimate
This is then followed by extremely loosely defined requirements being passed down to the technical people in hope of getting an estimate back. Technical people, especially experienced ones, know how this goes. First, you know you don't have half the details you should have, even though you've most likely asked for them numerous times. But you know, "all I need is an estimate" says the sales guy (who in this case also happens to share the board, made solely of other sales related people). You already know you can't win here, so you give a ballpark estimate considerably above what you imagine should be the real value, hoping it'll cover your ass, but highlighting the fact that until more details are given, this is still a ballpark figure.
It doesn't matter. The sales guy takes your estimate, shaves a third off of it, throws in a couple of bonus features in there (which are most likely of no use to the client), and makes it a hard estimate. When you point that out, you're told not to worry and that it's not your problem (spoiler alert: it very much is). So you write your specs down, or rather, you slap a few bullet point lists together with whatever little time you're given (you're expensive). This is given to the client as what both sides should abide by, despite you repeatedly saying that it's a bad idea.
From that point, you realize that the time you've been given to achieve the work is, roll drums, not nearly enough. You can already see poorly defined features growing arms and legs: clients, and I can't really blame them, will use any vague requirements to their advantage. You also realize that in order to do all the work and still meet the deadline (you still most likely won't), you have to half-ass a bunch of the work. You can already imagine the support issues pouring in once the project is in production, and you see all those legacy projects constantly getting in the way of your new, poorly estimated work, thus slowing it down even further. You will also get blamed for all these issues that are cropping up from past projects, and this ties to the next point.
The best one of all, that thing that wasn't your problem, suddenly is. You've somehow become responsible for the poor estimates you clearly said weren't to be relied on. So you know, you better make an effort and fix that by, say, working extra hours here and there, like, every single day. You burn out and grow frustrated, which slows the work down even further. Even better is that your crazy work hours become an expectation, if you happen to finish a project on time this way, it'll be used as "See, no problem here, you got it done!", pat on the back and condescending compliments ensue.
So you realize you're never given time to do decent work, that at the end of the day you're better off not meeting deadlines, and that whatever happens, you're always on the losing side.
I'm not saying this is the case in all companies, but I've also seen this far too often. On the very few occasions I had the chance to be the main point of contact with clients, projects were on time, clients were satisfied, and of course, some sales guy still slapped his name somewhere on there and got his bonus, but at least I got to do good work on reasonable deadlines.
I loathe most sales and marketing people, I often find they're a completely unnecessary layer in product and value creation. It's not that their position couldn't be of value, but I don't believe it can be if the money incentive is always more prevalent than the idea of good work done or happy people. As Henry Ford said, "A business that makes nothing but money is a poor business".
Note: I've resigned from this job to work in research.