Why are software development task estimations regularly off by a factor of 2-3?

Our biases.

We are not evolved to deal with time. As a thematic extension, every time we remember something, we overwrite the memory. The past is not in our control.

The future is not in our control either. We are poor at acknowledging delayed gratification. Few animals had to invest and sacrifice their current rewards for a payoff months or years later.

On a similar note, when humans imagine an average-case scenario for a task, we imagine best-case scenarios.

In an age where investments of time and money play a large role, those humans who can better grasp time (with external systems of aid) will survive.

Becoming a time lord is more important than ever.

A few reasons: 1. Developers do not consider unknown unknowns 2. Developers are generally asked to estimate aggressively. I have literally been asked "can we get more aggressive with these estimates" that were already aggressive of course. 3. Most projects require coordinated efforts across teams and almost no one knows how to estimate the other team's jobs and schedules.

anther log for the fire - sometimes the estimate isn't so off by itself(although it often is)

somehow we fail to account for the other things. background bug fixing activity, writing unrelated future work out on the whiteboard, reworking an adjacent component. not being on your game, expecting something to already work when it clearly doesn't

so yeah, those are developer failures. but at least as often, i think technical management has failed by laying out a plan with a fixed end date. for all the reasons above and more, the end date passes by, and another date is laid out, and so on.

i think we all need to do a better job of adapting to circumstance and continuously measuring the progress vs the passage of time. where we can settle for less than what we planned, and how to express the truth of the matter. the worst trap we all fall into is thinking that perhaps we aren't as far along as we should be, but a couple days of solid effort will put us back on track.

'just two more weeks' is the death knell of many projects

My answer is a metaphor:

When we plan to climb a mountain, we aim for the top and focus on everything that's required to get there. It takes real experience to also plan for the trip back down.

All of the above. The answer is a pie-chart with its most prominent slices: lack of experience, incomplete specs, unforeseen complications.

it's really quite simple - it's like when you ask a fisherman - how long does it take to catch two fish - he'll give you an answer as to not appear incompetent - but he has no clue...