Software estimation considered harmful? (2011) | Hacker News Reader

Software estimation considered harmful? (2011)

gigamonkeys.wordpress.com

48 pointsluu55 comments

Unless I missed it in there, there is another reason for estimating work - deciding whether it's likely to be worth the effort to build it.

It's rare that a development organisation has run out of things to build. There's almost always going to be several ideas competing to be the next thing to start, and there's got to be some way of deciding which you should go for. One obvious way is weighing up the likely cost v benefit of each idea, both of which require you to make some form of estimate.

Of course those estimates will almost certainly be wrong, but at least starting with a view of whether you think it's going to take a week or 9 months to add a mobile optimised version of your website, or whatever feature you're considering next, is going to be more helpful in making that call than starting with absolutely no idea at all of the effort it's likely to take.

I tend to agree with Tom DeMarco on this issue: "This leads us to the odd conclusion that strict control is something that matters a lot on relatively useless projects and much less on useful projects. It suggests that the more you focus on control, the more likely you’re working on a project that’s striving to deliver something of relatively minor value."

(From http://www.computer.org/cms/Computer.org/ComputingNow/homepa... )

Totally agree. Whether you're creating a new product feature, or doing a specific piece of work for a customer you need an estimate to define the cost, resource and time-line. Without it you can't understand whether the work is worth doing.

For managing cost and resource the estimating process acts as a way of discovering the complexity in the problem. Yes, it will probably be wrong to some degree, but there are ways of handling that (ie probability) on the effort analysis side. The other thing that helps is to identify the edge of what you can see, if something truly is totally unknown then breaking the work into blocks or discovery stages makes sense.

At a practical level time-line is the one that's the least useful to estimate up front, and seems the least amenable to assessment.

Without the ability to make some sort of estimate you cannot assess whether the work is likely to be worth the investment.

> Without the ability to make some sort of estimate you cannot assess whether the work is likely to be worth the investment.

I completely agree. But I'll add that without some notion of risk, the estimate is itself risky. In other words, when building a business plan around software estimates, getting a one number as an estimate (30 days) is a bit crazy. It would be much more useful to attach a range or confidence to the estimate

What happens now is that Project Manager asks Team Lead for an estimate. Team Lead pads the estimate a little and says 40 days. If Project Manager expects a 20% ROI, there's a significant risk that the project will still end up in the red. Or the team ends up crunching like crazy to "catch up", which is basically the same thing since your spending other things (quality, employee health and goodwill, etc.) instead of budget. Whatever the exposure, there is significant risk that is not clear in the business plan, which is a failure of management.

In contrast, maybe Team Lead could give a 50% chance that Project Jabberwocky will be done in 30 days and a 95% chance we'll be done in 120 days.

Wait, Project Manager says, that's a wide range of values to plan around! He's right, but the emotional reaction is due to him being educated about risk, not due to the Team Lead dragging his feet or something.

So what is a Project Manager to do with that estimate? Take the risk into account. Plan around 95% estimates (keeping in mind that 1 in 20 projects will still miss the mark) or make sure the risk is mitigated in other ways: lots of little projects, relatively low cost projects, projects with huge expected ROI, etc.

The book mentioned in this article, 'Software Estimation: Demystifying the Black Art', actually goes into a lot of depth about this. For example, what is statistically likely to happen if you only use single point estimates when they are aggregated together, and how to correctly handle range estimates.

DeMarco and Lister of Peopleware fame wrote a book, Waltzing with Bears, which models risk and value as curves. You look for the overlap in the risk/value curves to help you decide whether, on balance, you're on a good thing.

> Without the ability to make some sort of estimate you cannot assess whether the work is likely to be worth the investment.

In case of a product, estimating the development time will be the least of your problems. How do you estimate the revenue of a new feature? Your estimate can easily be off with a factor or 10 or even 100. So in such a case, even with perfect implementation estimates, it's impossible to calculate the profitability of a feature.

Yes - it's impossible to reliably and accurately estimate (which is why it's an estimate).

But you've got to have some form of figure, both of cost and revenue - even if it's just back of a beermat guesswork based on a vague hunch - to make any form of sensible trade-off judgement.

I do a fair amount of work around business cases for new projects, and simply forcing the proposer to go through an exercise of answering "how do you think this project will increase revenue/reduce costs/whatever, and by how much?" helps weed out ideas that don't really have legs or haven't been properly thought through, as does having a first stab at "What's involved in delivering this? Is this a 2 week project or a 6 month one?"

Surprisingly good read. I will be looking more into the study showing that folks with no estimation had a higher productivity. I can say from experience that the Herculean task of estimating my own work adds to my stress significantly. To the point that I have no doubt that it hinders me.

Oddly, just having a deadline is nowhere near as destructive. So, coordinating with folks is a lot easier if they tell me when they'll need something, rather than asking when I can have it for them.

The worst case scenario is when there are only vague requirements or too many half-baked ideas. And somehow you're supposed to say when those poorly-defined features can be done. As if the completion date is not closely tied to the full definition of the finished feature. This is why I almost prefer the waterfall approach when hard estimates are required.

In my experience this is entirely correct, which is why I have always pulled estimates out of the air and stuck to them, flexing scope to fit.

The need to estimate highly depends on whether you are developing a project or a product. On a project, the customer always wants to know what will be done at which date, and how much that would cost.

A product however, can be released "when it's done", so there is no real need to estimate. The additional benefit is also that it can be released when the developers feel that the product is ready for release, unlike a project, where there is a lot of pressure to deliver things.

My personal view (and experience) is that developing without estimates saves time and creates a higher quality product. But other (smart) developers that I know like to work within estimates because they think it pushes everyone to deliver things.

We probably are both right, because there is yet another factor that comes into play when choosing to estimate or not: motivation of the developers. Highly motivated developers who love their job, rather work in their code instead of goofing off on the internet. But when the project/product is not that special, and not everyone feels passionate about it, estimating and follow-up is probably preferred to put a little pressure on getting things done.

Estimates should never be used to apply pressure(people take shortcuts), they should always try to reflect reality. In products estimates are used to work out which features provide the most value, in the shortest amount of time.

> In products estimates are used to work out which features provide the most value, in the shortest amount of time.

With products, estimating the value of a feature is even harder than estimating the implementation time/cost. And calculating the future revenue of your product is also harder than calculating the future implementation cost.

You might argue that your product must be finished before you run out of money (which you can calculate with estimates), but MVP's don't rely on estimates that much and you can test your product/market fit at the same time.

In theory, product profitability and selecting features seems highly depend on estimates, but in practice I always saw that gut feeling was good enough, for both features and how long a development will take. You can't drop the must-have features, and nice to haves can be implemented after your first release.

> In products estimates are used to work out which features provide the most value, in the shortest amount of time.

And this is why there is actually no difference between a product and a project. But value doesn't depend on estimates at all, as there may be many different ways to implement. Calculating value is therefore more important than estimating net value for any specific implementation - it tells you what your direction of travel should be.

> A product however, can be released "when it's done", so there is no real need to estimate. > The additional benefit is also that it can be released when the developers feel that the product is ready for release, > unlike a project, where there is a lot of pressure to deliver things.

This view probably depends on your definition of "estimate", whether it's being done at an individual developers level or whether it's being done at a team/product level.

For team/product level (as another commentator mentioned) if you don't have a cost/benefit analysis then you don't know if the work being done is worth doing, and you may run out of money before your product is "done".

Additionally, (the article mentions this but it's worth expanding on) other bits of the organisation need to co-ordinate and depend on knowing status and when the product will be ready. To take an extreme example an enterprise software company might need to know with more than six months notice in order to start planning and preparing sales, marketing and PR teams for the coming release.

Sure, but you need to release before you run out of money. You also need to cost/benefit a feature which is hard to do without any sort of estimate.

I'm a fan and student of estimation techniques, but this is more of an argument for always having releasable software.

"a Wesabe developer who consistently gets excited about developing things that Marc refuses to allow into the product should probably make sure their resume is up to date"

If you cannot read the mind of our CEO who changes the direction of the company every two weeks, you are fired?

    If you cannot read the mind of our CEO who changes the 
    direction of the company every two weeks, you are fired?

I've experienced this.

Pretty much. Whats even worse is if it's some guy between you and the CEO, who can then pretty much travel back in time and paint you in whatever picture he sees fit to put his interests above yours as a lower level developer.

I think it rather mean that if you consistently come up with and fixate on ideas that are so misaligned with your company and product that the boss has to step in and say "No dude. Please don't." than it's probably not the right company for you.

I think it's more that estimates don't work well with projects. If you know you're going to be building a, b, and c as part of the project then there's not a huge benefit in estimating them.

I also think projects don't often work well as a way of building software. There's often uncertain deliverable a long way off into the future, and a guess at whether it will actually generate any value.

I don't just think that estimating the cost of features along a project is a bad idea, I think trying to come up with features a long way into the future is a bad idea. Things you learn along the way are likely to require you to scrap them or make significant changes, unless you're ether prescient or learning nothing.

Without projects estimates become more useful again. Focus on the most valuable things you can be building in the next couple of weeks. Estimating the top n that seem the most valuable will tell you a) if it's achievable to complete them in 2 weeks, b) give you an idea of the value:cost ratio to help prioritisation.

> There's often uncertain deliverable a long way off into the future, and a guess at whether it will actually generate any value.

That sounds more like lousy planning than anything else.

So... agile?

Well yes. I suppose the problem is people think they are being agile when they actually have a 6 month backlog of user stories and won't release anything for months.

In that scenario estimating things becomes very tedious and not particularly useful.

But what if you actually have a 6 month backlog of real user stories? As in, you know these requirements pretty well, for instance because you're re-implementing a legacy system?

I would prefer not to do re-implementation of a legacy system as a project. For a start legacy systems often have a lot of dead features that nobody uses any more, so we risk wasting time implementing things nobody needs. A 6 month backlog also sometimes means a build-it-all and release approach which means 6 months before we start using the replacement in production.

I prefer to take a strangler application approach http://martinfowler.com/bliki/StranglerApplication.html Start replacing small parts of the legacy system in production as rapidly as possible. Then move across bits of functionality bit by bit until enough functionality has been moved to turn the old system off. This also allows you to rapidly get benefit from new functionality that the old system did not have, or improved performance for the most critical parts of the system, without getting bogged down in implementing lots of rarely used features.

I would constantly be looking for the highest value features to add or replace from the old system and use estimation as a tool to aid prioritisation of those.

What's the difference then between the new system and the legacy one?

In this hypothetical scenario? Any number of nonfunctional requirements.

So no added business value can be conveyed in stories?

Sure it can. User story: "The software is orders of magnitude too slow. Rewrite it from the ground up using scalable technologies."

Point being that it isn't a rewrite but a new piece of software.

If you're implementing, say, a general-purpose accounting system, from scratch, then I think your requirements and feature set is well known beyond the "a few weeks" horizon, and planning that feature set months in the future is not unreasonable.

Actually wouldn't agree with that example. Accounting systems are very diverse e.g. Sage, Wave, Freeagent, SAP, Excel+Access.

I think part of the explanation for why the software products with estimates performed poorer in terms of productivity can be explained by parkinson's law[1].

Now, this phenomenon isn't always deleterious imo. I've noticed in myself that my work 'expands' in two ways: if I'm working on something dreadfully boring, then it expands through time wasted on hacker news or time spent on side projects. If on the other hand I'm doing meaningful work, then it's quite often that the work expands when I take on refactorings and minor additions that will benefit long-term productivity.

If the work is indeed expanding in a meaningful way, then even though the measured productivity might not seem 'optimal' it doesn't necessarily indicate a big problem either.

[1] http://en.wikipedia.org/wiki/Parkinson's_law

My big point is:

- If you don't know what you are doing or you are doing for the first time, there's no point in estimate as your estimations will be off anyway.

- If you know what you are doing and already did it before, there's no point in estimate as you already know how much time it'll take.

If you know what you are doing and already did it before, [...] you already know how much time it'll take.

So you've estimated it then?

If the argument is that we spend too much time producing detailed estimates that don't actually reflect reality, then I agree.

If you're suggesting that a sensible way to run a business is to say "I have absolutely no idea how long this will take, and how much it will cost, but let's do it anyway" then that's crazy.

Even the team as Wesabe (in the article) are estimating - He does encourage his developers to spend most of their time working on things that they can finish quickly and When they need to, Wesabe developers will tackle bigger projects. You can't distinguish "finish quickly" from "bigger projects" without estimating. It may be the quickest easiest form of estimating "big or small", but it's still an estimate.

The reality of the argument is that a lot of organisations can make do with a lot less formality and rigour in the estimation process. Moving from "vaguely approximate" to "slightly more accurate" is an expensive process with all sorts of cultural side effects that too many people assume is "worth it" without any real justification.

But we all estimate, because it's pretty rare that you can justify starting a piece of work that might take a day, or might take 10 years - I'll let you know when it's done.

I think that part of the problem is that when people say "estimate" in our profession, they're thinking of capital-E-Estimates. With a whole bunch of theatre and disruption and danger and political nonsense.

It's not the informed prediction of the value of some unobservable variable that upsets us. It's the larger picture around it.

My estimate is that you've created a false dichotomy to avoid the very large area between pure research and pressing a button, where estimation can provide value.

In my opinion these two points only hold if you work alone.

As I see it, you don't estimate for yourself, you estimate for transparency, so that other team members can plan accordingly.

In your first point, you might not be able to estimate, but others might be able to do so and give you hints. Your final estimation might still be off, but that's how you learn.

In your second point, you estimate so that others can see how long it will take and plan their work better.

If you've already done it before it should take almost zero time because you just re-use your previous code!

Of course the higher value a thing has, the less you need to quantify that it has high value. Conversely, the lower your margin, the more you need to control it to within an inch of its life. You could even say that if it needs estimating it maybe isn't worth doing.

Where I work we have weekly sessions to plan the week's work. We ask our customer to put stories in order of value to them. It might look like this:

    A
    B
    C
    D
    ---- week marker ----

Then we perform a quick estimate for each of the stories. We can talk them over quickly, ask clarifying questions and so on. Because of research on the unpacking effect, I like to make sure to say aloud the sort of subtasks I think will be involved, but sometimes we don't.

We quickly do a "shoot out" and then simultaneously reveal our point estimates for the story we're estimating. On divergence we explain our reasoning. Sometimes it takes a few rounds to reach a consensus.

So after this process, it might look like this:

    A    4 points
    ---- week marker ----
    B    2 points
    C    1 point
    D    1 point

Because we've got velocity of 4 points.

At this point the customer can change the order of stories. They might decide that having stories B, C and D is actually better than having story A by itself.

Just trading purely on value or on estimated effort/complexity doesn't give you the full picture. It's the ratio of the two per-story and between stories that helps a customer make the best-informed decision.

Agree that you want to get a view on whether a task is easy or not - but this is rarely understood until you have started. And frankly most advocates of estimation aren't talking about T-shirt sizes but some kind of currency they can tot up to measure 'productivity'. The danger with trying to predict it is that you are getting deep into a specific implementation at that point and it is exerting a pull on the project aims. I think that can conflict with business value as the driver for doing anything. What's more important for eliminating waste (and death march ftm) is a set of alternatives (outcomes and implementations) and a framework which makes it cheap to back up and switch routes when you find something isn't turning out to be easy thing you imagined.

We deliberately focus on simple estimates of a unitless measure (story points), rather than imposing a unit that will cause distortions in our thinking. The value of those points is specific per-project and it emerges from actually doing the project.

I agree that allowing change is important. Reduce the cost and risk of changes, and you will make changes as and when required. You'll also become less preoccupied with perfect planning, because it won't be necessary to manage risk.

There's two loops here. One leads to small, light, manoeuvrable projects. The other leads to building bigger plans because the risk is higher, which means that a bigger plan is needed, which raises the risk, so the plan is more detailed so that ...

Impact mapping gives you that big plan without having to detail implementation and keeps it manoeuvrable. Kanban lets you switch the backlog around through the day and doesn't lock the project into sprints (which are mini waterfalls imo). Estimation is entirely optional here. (The main benefit I've seen from estimation is in forcing a dev team to collectively confront what's involved in implementing a feature - as an input not an output.)

I've found that Kanban leads to coordination overhead because you always have too much work in flight in too many stages. I prefer story-by-story development, which is the direction to which most Kanban implementations wind up evolving. One small thing, to completion, at a time, per pair.

WIP limits should iron this out

The author questions the concept of an estimate in Software Development, and proposes four sub estimates. I'm not sure that replacing one estimate with four will do the trick...

One of the things I sometimes say to our clients at my workplace is that estimation can be done two ways. The traditional way is summative. Take all the things you want to do, estimate them (which is where McConnell's book really shines), then produce a range that expresses your likely outcomes.

The other way is emergently. Build a pile of things you want to do, put them in rough order of value, then estimate a few things at a time. Then start working. Only after a few weeks will you be able to begin to predict outcomes.

This is old news. I'm talking about velocity. Which is actually the perfect word, because summative estimations are about a total distance travelled; velocity is about the speed and direction you're currently moving in. Velocity is the differential of the project's outcomes.

Velocity is a more powerful metric because it includes its own mechanism for updating estimates. A classic "estimate smell" is that the first estimate made is the last estimate ever made. The closely related smell is "estimates that fit the plan", aka "estimation by fiat".

Velocity, however, emerges from observation. No additional action is required to update summative estimates. And what comes for free is typically, in software methodology, all that gets done.

Another way in which moving to velocity as your estimating technique is that you change what is fixed and what is fluid in discussions with clients. Traditional project management takes the work breakdown as a fixed outcome and then looks to optimise around that. Time, schedule and quality are then the variables. Velocity allows you to vary scope and work as well. You can then hold quality, time and schedule constant instead.

It's hard to convey how big of a deal this is in practice. It seems like a minor alteration. But taking seriously that the speed of light in a vacuum is fixed (per Maxwell's equations) means that spacetime varies. Which makes ... a big difference to physics.

I have seen this change in thinking bring high-risk projects to a successful conclusion under tight deadlines, with high friction, with high technical demands, in short timeframes. We had the data to show where the project was going, and that allowed us to focus on ensuring we were delivering in order of value. It also allowed us to discuss with the client what was possible and what was not possible.

Under traditional schemes we'd have had a fixed pile of features and the project would've been a "failure". Yet it did everything that was needed on a particular day, with very generous margins of safety.

OFF TOPIC RANT

Just have to register my frustration with the cliched "[x] considered harmful" headline in the IT world. It devalues the proposition instead of lending it authority.

EDIT: "Betteridge's law of headlines" also applies here (Any headline which ends in a question mark can be answered by the word no.) Since it's an opinion piece, state your opinion in the headline, and stick with it. The question mark suggests even the author's not sure he agrees with the premise, which makes me think the article is not worth reading.

If you don't rant how do you learn from your mistakes?

It makes you reconsider what went wrong like The Mythical Man's Month (tools considered harmful aka silver bullets) by F Brooks the project manager that nearly killed IBM and OS360 is such an example.

Coding is about creativity and innovation. Not about respecting the authority. It is a hard mix of conservatism (standing on the giants' shoulders) and like the other giants ... to rant.

At one point the best developers new ideas are rants ex robe pike vs Unix (plan9, Go) (all his rants against unix (see cat) are a very good source of knowledge for unix devs)

And rants are always instructive when done by great person.

* http://harmful.cat-v.org/software/ (I learned a lot with http://harmful.cat-v.org/software/symlinks http://harmful.cat-v.org/software/dynamic-linking/ ) * computer for cynics on youtube (a rant against computer industry in general)

“It’s a curious thing about our industry: not only do we not learn from our mistakes, we also don’t learn from our successes.” — Keith Braithwaite

I meant that my comment was an off-topic rant.

I have no problem with the article itself (I didn't even read it), nor the right of anybody to rant about anything. It's the cliched headline on which I was commenting.

You may enjoy: http://meyerweb.com/eric/comment/chech.html

I enjoyed that greatly! Thank you for the link.

Reply on news.ycombinator.com