Hacker News Reader

And one of the classic problems with estimates is that common processes tend to bias estimates.

For example, what (I think) McConnell calls "picking the first non-impossible date". An exec asks how long project Magic Pony will take. A manager asks the team and the team comes up with a number. If the number is higher than what the exec imagined, or even higher than what the manager imagines the exec imagines, pressure is applied on the number. "Are you sure? That seems like a lot." A negotiation ensues, and often the number lands on the first date that engineers can't absolutely prove is impossible. Or the first date that engineers won't quit their jobs over.

For an estimate, we in theory want one where the team has a good chance of hitting it. Say, a 75% chance of success. But iterative pressure from the powerful will shift the distribution to 25%, 5%, 1%.

The classic cognitive trick--people, unless specifically trained not to, will always estimate the best possible scenario, even if past experience has told them that the best possible scenario never happens.

I'm dealing with this at my company right now. Our clients, and even our internal teams, will always refuse to estimate according to our past behavior.

Us: "We estimate your build will be functional in prod in five months."

Client: "Unacceptable. That's way too long. We need it in three."

Us: "Okay, we technically can do that, but that relies on a big internal release that we think is risky to base estimates on."

Client: "All I'm hearing is that you can do it in three."

Then we get to three months and hit delays over and over again until finally, five months later, the build is functional in prod.

The thing that really frustrates me is that when we deliver the initial estimate, the clients always refuse to accept our medium-case estimate, with the implication that they'll drop us as a vendor if we follow that. So we give them our best-case estimate, fail to hit it, and then they buy more from us anyway, complaining the entire time.

Why don't we all just look at the past 10 builds we delivered for them, admit that the best-case never happens, and stop pretending like it's unacceptable to take longer? The point here is that they keep accepting it. Everyone complains, but we all keep selling and buying to and from each other. Let's just admit it's harder than we're saying to get software built and accept longer estimates, given that we're all going to keep selling and buying anyway.

Totally. In most places what people call "estimates" are just wishful thinking.

And that client behavior you describe is what I think of as "Kirk-style management". An executive thinks that the way to get technical things done is to be demanding and shouty toward the technical person, insisting that the number given isn't good enough. Scotty's pathological response to the pathological situation was to lie his ass off about estimates: https://wiki.c2.com/?ScottyFactor

Thanks for that article--bookmarked.

> Why don't we all just look at the past 10 builds we delivered for them, admit that the best-case never happens, and stop pretending like it's unacceptable to take longer? The point here is that they keep accepting it.

It's theater, mostly. I think some people think they have to do this. How did they get you to keep delivering when you did? It was their threats/complaining! If they stopped doing that... there's no telling how long it might take you to deliver.

Given that every project you've called at 4 months has delivered in ... 4 months, for the past 7 years... your estimating/delivery ability is still tangential to their managing of the delivery by regular barrages of threats, complaints and haranguing.

Wow, great insight.

It reminds me of the dynamics with walking down the sidewalk past a dog. If I walk too close to his yard, he decides I'm a threat. He barks! I walk away. It's easy for him to do post hoc ergo propter hoc and conclude that his barking chased away a dangerous threat. In reality I didn't do a thing different, but his lesson is that barking works!

It's a great point, and one that could be rectified by their teams actually having standards. Threats can't be empty. "Deliver early, we'll give you a bonus. Deliver on-time, we'll pay. Deliver late, we'll hold you accountable and pay less according to this attachment. Deliver really late, and you get a strike. Three strikes and we drop you."

Of course, that would require backbone, fortitude, accurate forecasting, integrity, and management support.

I've had arguments with PM's about this. After a few years of doing similar projects you can sort of squint sideways at something similar and have a gut feel (if it's completely different from anything ever done, that doesn't work of course).

But every PMP (tm) certified PM I've worked with thinks if you can just break the project into granular chunks, estimate those, then sum the estimates, there's your final deadline. The fact that it never works doesn't dissuade them on the next project is the depressing part, it's all just theatre for the execs.

All I'm hearing is that the client signed the contract, which means you won.

Whether your company makes money on the change order is a matter for your boss.

One technique is to ask for 3 point estimates. Get everyone involved to estimate each item in three different ways:

1. How long it will take if everything goes well and according to plan.

2. How long it will take if everything goes wrong and we uncover further problems along the way. Probably getting those risks listed.

3. How long it is likely to take.

I imagine we tend towards doing just one of the first two, but having to do both should lead to more agreement on the third.

That said - its been about 30 years since I was taught this as part of critical path analysis, and I've never seen it used in the wild.

For sure. I know plenty of ways to deliver solid estimates and hit dates. I think the problem is execs mostly prefer having estimates they like over accurate estimates.

The problem even with that method is that if everyone agrees on "most likely", and that is perfectly correct, and you use it, you'll still be wrong more often than right.

The reason is the underlying distribution: because the probability distribution is usually heavily skewed to the left, with a long tail to the right and lots more room for things to extend than to contract, the day which is "most likely" when looked at in isolation is actually well to the left of the median date, the date with a 50% probability of being before or after completion.

If you want an 80% chance of being right, say, you need to do some more maths with the three numbers to figure out what the right date to use is.

I would very much like to see this done. The problem is that estimation isn't seen as work. It's seen (or, more accurately, framed) as something devs should be able to just magic from thin air with perfect accuracy, so why would we bother adding complexity? Very few organisations bother tracking estimation precision, I think because the incentives are aligned such that it's actually to the advantage of the people writing the cheques that developers under-estimate. I think the power over the developers that a missed estimate gives to an adjacent organisation is a real factor in ensuring this never gets improved.

> you need to do some more maths with the three numbers to figure out what the right date to use is.

We did this all the time at a previous job. We had a spreadsheet in which we put in the three estimates. They were combined in some ratio (I think it was 1 part each for the outliers and 2 parts for the most likely). Then some time was added on for testing, test remediation and project management at a set additional percentage. Worked out pretty well, we seldom ended up with issues when using this method.

Tom DeMarco describes a method in Waltzing With Bears that's quite interesting: use a triangle defined by the three points as a rough approximation for the probability distribution, and then cumulative distribution function calculations are just juggling quadratics that you can plug into Excel.

Mind you, these days Excel's got the distribution function formulae in it that you'd need to do it "properly" with a lognormal distribution, but I like the way the three estimation points have a direct and obvious link to the result with the triangle method.

> A negotiation ensues

... and then the estimate is no longer an estimate, it's the outcome of a negotiation.

> Are you sure? That seems like a lot.

"No, I'm not sure, boss; it's an estimate."

Lognormal distribution is always a good one.

It's applicable where the randomness has a multiplicative rather than additive effect. For example, the work may double or be halved. Lots of natural processes work this way.

https://en.m.wikipedia.org/wiki/Log-normal_distribution

I've written about this before more extensively, but my previous company tracked time vs predictions very rigorously and extensively, and they always fit a lognormal distribution shockingly well.

An useful property is that the Law of Large Numbers doesn't apply to it, so all of your estimate history is basically useless. Also, when you add enough estimates, the total time to completion is mostly determined by the error in one or two of them.

I tried to do that! Didn't use Poisson, used normal distribution, but I think that's not so relevant - at the end of the day it's still just more elaborate guesswork.

I think what's useful is thinking about the problem, and communicating in group/ disseminating the information (this is required in order to do the typical "planning poker" or whatever planning technique the team happens to use). Whether you end up with L/ S /M or 7/13/21 points or whatever else you use - it's irrelevant, so no need to sweat it; what is relevant is that you discussed the tasks and everyone has some understanding, and maybe they're a little bit better spec-ed now.

Normal distribution is deeply inapproprite for time estimation because it yields the same probability for 3x the estimated time than for the negative amount(!) of the estimated time.

Your second point rings closer to what we should be doing instead. We should put more effort into analysis and refinement. In my experience, estimation meetings where all tasks got estimated were almost useless in the end; and the most valuable meetings where when we didn't finish to estimate, but instead gained a deeper understanding of the requirements and of the domain.

Product and sales always want exact dates from tech but always seem to be a lot more vague when it comes to customer numbers and revenue numbers.

I've found the following system to work well. Estimates are always given as the following: hours, days, weeks, months, quarters. That's it, not numbers attached. It means "some small number of _______". This allows for discussion without everyone feeling trapped. If a feature takes "weeks" and Product thinks it should take "hours", then that is a discussion that can allow for a change in scope or requirement clarification or just education about how the system works. Of course, PjM systems only work if everyone buys in and that is always the bigger challenge.

For projects with a deadline that matters, I've found that explicitly estimating to "80%" likelihood has helped me hit deadlines much more consistently. I used this to hit a deadline on a 4 month software project only 10 days out, which I think is good for software.

I think implicitly most estimates are the 50/50 case. "I'm 50% sure I can do it in this time". Much of the time I don't think this is very useful, and isn't very well thought through.

You have to assume you're going to hit a brick wall at least a few times in any given project, and something you thought would be a snap turns out to require days of refactoring. Especially if there's some aspect of business logic that wasn't clearly communicated beforehand and/or the customer hadn't considered until it came to light as a code problem. This should be built in to the estimate, but always come as a surprise gift to the customer when you still manage to deliver it on time.

In design and code, customers need to be consulted and asked what they wanted and made to feel important but not about anything actually mission critical. That's what you're hired for. At least a few times in any long project you need to come up with a meaningful sounding but essentially empty aesthetic decision for the customer to make. This allows you to both show progress and let them feel they're in control. It also allows you to extend the deadline if you feel yourself behind.

50% is optimistic. Some (ok, many) estimates are actually 1% projections. The soonest we could possibly be done.

This, too, is not very useful.

The poisson distribution is quite-light tailed. With a poisson distribution, the probability of completing the task per unit time is constant.

Real world tasks are likely to take longer if they have already gone on for longer; maybe a Pareto distribution would be more appropriate.

I think you meant exponential distribution? Poisson is a discrete distribution ( number of task completion events occurring in a given time interval if task completion is exponential)

Exactly. One of the fundamental tenets of science is reporting estimates with appropriate precision, e.g. significant digits. If the probabilistic interquartile range spans multiple months, why would I report the mean of the estimate in "hours of effort"?

What's frustrating is I've articulated these thoughts clearly to managers in the past. They nod their head and agree and then ask for the hourly estimate again and we all collectively share our disappointment when our estimate is inaccurate after the fact.

Nothing like running head first into a wall and expecting a different result.

My management suggestion? Frequently identify and advocate for simplifications, and modulate the degree of simplification based on progress to date. If people enjoy their work, they will work hard regardless of timeline hawking. So the variable factor is really the volume and complexity of work, not worker intensity or motivation.

Yes that would mean managers have to do something, and while some are incidentally useful and productive, there are few external incentives for them to be so. Yet there are many incentives to let you burn out after hitting a wall a hundred times.

So a Poisson distribution is for count data, not continuous like time. A gamma distribution is a continuous distribution that have the shape you want. But, this approach should involve getting data and plotting it to see empirically what sort of distribution it might have.

I’m also unclear how the dependency you mention is going to come in. That said I agree with you in general that some statistical reasoning should be brought to this problem. Maybe just showing people the way variances are additive and the impact that has on the overall variance of a project made of many small projects. You could pick a plausible gamma distribution for individual task duration and play around with seeing what the sum of a number of IID gammas looks like —- the sum is a gamma distribution also; see wikipedia but the formula.

> Useful to anyone much? Unclear.

If you're the leader/manager/person in charge, do you want to see reality, or do you want to keep your head in the sand so you can have a simpler planning process? If you want reality, and that's what the most accurate view of reality, then by definition it's more useful than anything else.

Is there a way do calculate this in Excel without using a Monte Carlo simulation?