Why is estimating so hard?

http://news.ycombinator.com/item?id=3522910

To quote one part:

Let's say that you have 20 tasks. Each involves rolling a 10-sided die. If it's a 1 through 8, wait that number of minutes. If it's a 9, wait 15 minutes. If it's a 10, wait an hour.

How long is this string of tasks going to take? Summing the median time expectancy, we get a sum 110 minutes, because the median time for a task is 5.5 minutes. The actual expected time to completion is 222 minutes, with 5+ hours not being unreasonable if one rolls a lot of 9's and 10's.

1) People are pretty terrible at understanding things exponentially (or overall, just non-linearly). Bumps in the road for programming I've seen can set a time goal back by 10x. Thought it would take a week? Suddenly its more than two months.

2) As highlighted here, we're often bad at translating all of our assumptions and intuitiveness into code. Its just a website with a login, with a, etc.... Yet, there's a lot of details there that you're forgetting about.

Experience is the best thing here you can hope for, but even that can fail. Unless you're doing exactly what you've done before, you're going to hit bumps.

Also, no client wants to hear an overly long and safe estimate. You might even think it will take 6 months, but the client wants to hear 2. Everyone else is telling them 2. You look incompetent if you say 6, so you say 2... maybe 3.

When we lived in caves, there were men who were perfect estimators and there were men who were hopeless optimists.

When the optimists said "I'm going to get a deer!", the perfect estimators would explain with great precision and accuracy how difficult it was. The optimists ignored them, and the perfect estimators just sat around the cave. The optimists did finally catch the deer, even though it was probably more work than it was worth.

That's why most of our ancestors were hopeless optimists. The perfect estimators just didn't reproduce. So how can we help but be hopeless optimists ourselves when it comes to generating schedules?

The solution: multiply your schedule estimate by pi to get a round number.

  def break_13(text):
       words = text.split()
       lines = [[]]
       for word in words:
           last_line = sum(len(word) for word in lines[-1])
           if (last_line + len(word) + len(lines[-1])) > 13:
               lines.append([word])
           else:
               lines[-1].append(word)
       return '\n'.join(' '.join(line) for line in lines)

> If you are good at abstracting, you’ll likely come up with three different scenarios for breaking a line. 1. you break it at the 10th character of a word if that word is longer than 10 characters. 2. You break it at the 11th character if that character is a space. 3. You look backwards from the 10th character looking for a space and if there is one, you break it there.

I think it is less valuable to break down your procedure into smaller pieces before given an estimate, and more valuable to have a good knowledge of what sorts of problems are hard and why they're hard. That way, you don't have to resort to glib phrases like "Write down a procedure to tie your shoes".

a. not enough time is given to come up with good estimates

b. Programmers usually don't enjoy the process of estimating and so don't do it properly

c. Previous estimates are usually not shared amongst programmers

To elaborate.

a. There is usually pressure from managers or clients to come up with quick 'ballpark' estimates but only treat them as such when they don't fit their plans. So say they want something done in under 6 days. If you estimate a day ... there might be no questions asked, until you start blowing deadlines. But if you estimate 10 days, then a lot of people will quibble with you, trying to pressure you to give rationale or otherwise reduce the estimate.

As the poster shows, properly estimating something is actually a pretty involved process that means constructing the thing you're trying to build on paper (or in your head). Clients/bosses don't want to pay for that time, so devs are implicitly pressured to underestimate things, to avoid long discussions about unfavorable estimates.

b. Teeing off of a. clearly building the damned thing is way more fun than constructing it in your brain, estimating it, then going back and forth with clients on each line item as described above. There isn't a particular process, and sometimes you might actually spend almost half of the time of the estimate actually figuring out how to do it. Devs simply don't like estimating things, so they make (bad) guesses that seem safe but eventually turn out way wrong.

On a project I had once, I hated the work so much that I simply took a wild guess at how long it would take to do one particularly hairy bit and doubled it. Seemed totally safe at the time ... figured I'd be under by a lot. Turned out being even double that estimate ... the culprit ... Internet explorer.

c. If you go to a mechanic shop to get an estimate for work on your car, you might find that the mechanics go into a computer, plug in some details about your car and the problem ... and the computer spits out a number. This is because lots of things with cars are pretty routine ... Software isn't like that ... stuff that's routine is usually pulled out into plugins and frameworks that just work ... so we're constantly tackling problems that seem new, throw in the capabilities and weaknesses of particular frameworks, plugins or programming languages and you can see how these sorts of things can get away from you very quickly.

Programmers don't share their estimates with each other (hint hint), so the only really reliable source of good estimates tends to be personal experience, which is another reason why you should code ... a lot.

The only really good process I've found is to completely wireframe what needs to be done, (either that or build a clickable prototype that works exactly the way they want it to work). Then provide an estimate for that. But that doesn't always work for smaller things like bug fixes or smaller features.

A sealed envelope containing a destination is placed in your hand. Without opening it, estimate how long you will need to drive there. Estimate high and you will be seen as slow or lazy, estimate low and you seem incompetent when arrive late. Others plan around your estimate--so you must commit to it. Fun game, that.

Is that thought so unusual? I don't really see, why it needs to be discussed, what makes a beginner guess wrong. The solution will be always the same: exercise or let someone do the task, who has the experience.

Could someone explain the point of this discussion to me?

I have shifted my focus from writing procedures for humans to writing programs for computers. when I'm done, I have a program the computer can run, and serves as a procedure that explains to humans just how hard the problem is- and exactly what compromises were made in the implementation.

the resulting program is much more thorough and thought out than the procedure I would have written.

On the other hand once its coded, it's way faster (because the logic was abstracted) and it is flawless (it always yields the same result).

1) Very few individual contributors responsible for providing estimates ever invest the time necessary to learn how to perform estimation.

2) Many stakeholders (managers, PM, etc.) do not know the definition of the word estimation.

the message is fine - i don't have a problem with that at all - but the numbers / facts / anecdotes seem way off base to me.

  i = 0
  for c in string.split():
	i = i + len(c) + 1
	if i <= 13:
		sys.stdout.write(c)
		sys.stdout.write(" ")
	else:
		sys.stdout.write('\n')
		sys.stdout.write(c)
		sys.stdout.write(" ")
		i = len(c)