Why people always say this? In my team we have never missed an estimate of what we have planned. I also don't remember it in other projects where team does take time for planning. What is so hard that people find estimating their work?
Unpredictability for any new work. Even knowing my codebase well, I sometimes forget how changing some piece of code will require changes in other places. And without having done exactly the same work before, I cannot know how difficult it will be and how long it will take.
Dependency on other teammates or other teams. If another team is building something that your work depends on, you have no way of knowing when you'll be finished. If the code you write needs to be reviewed by other team members, and then you'll have to incorporate the suggested changes, this makes it harder for you to say how long something will take.
Requirements are often hazy. Pushing back on that is good, but if the response is "do it anyway" then you either quit or handle the uncertainty as best you can. The latter makes estimating hard.
If you are going to have to depend on third parties (whether internal or external) that is another thing which can torpedo an estimate that was made in good faith. There would often be too much uncertainty in that situation to be able to accurately account for it in an estimate.
"estimates are always wrong" is obviously an exaggeration - the whole comment is written in that spirit.
Sometimes the "estimate" was already predetermined to meet some arbitrary marking goal.
The central difficulty and huge part of many programming projects is actually the "unknown stuff" that is insanely hard to estimate, be it
- requirements set up by users are often very vague
- interacting with some very complicated third-party library/software that takes years to only understand the basics of
- many programming tasks are unique, thus you have never implemented anything remotely related, and thus also have hardly any idea where the difficulties might lie
- the new software "has to behave like the old one". In the old software, there exists an insane amount of corner behaviour where nobody has any idea why it was introduced, but the users will at some point complain if the new software behaves only a little bit different in such an obscure corner case
This is my observation as well.
I also think that devs tend to consider the central problem they're solving when estimating, rather than the totality of the work that has to be done in order to complete a task. If I'm tasked with adding a new feature, my mind is focused on the business logic of that feature and if you ask me how long it's going to take, I'll tend to answer the time for just that. I'll tend to forget the time required for all the other ancillary work around that (documentation, UI changes, code review, QA, etc.) Omitting that stuff results in wildly optimistic estimates.
No other engineering discipline has this level of immaturity. Sure, estimates in other industries are also notoriously hard, but you don't see this utter disregard for them you see in software. And, I bet you donuts that every "software engineer" here as demanded estimates from other fields and felt angry when they were missed.
"When is that bridge going to be finished?" "When is the road work going to be completed?" "When is my house renovation going to finish?"
Estimates are important. And yes, they are hard, but there is spectrum of skill levels at doing them well.
However, most devs I've worked with were also very aware that estimates themselves have such a large margin of error that they can be considered a bit bullshit even though they are necessary.
After 35+ years of software development, you are either lying, incompetent, or worked on exceedingly trivial software. Sorry to be harsh, but I can guarantee you that if you were given a significant software project, asked to estimate and then deliver, that the vast majority of the time your estimates would be wrong.
I’ve worked on teams that have exactly your experience. I’ve worked on others where they could just never hit the mark. Skills and experiences seemed equivalent. There didn’t seem to be a magic bullet reason, just a whole series of reasonable explanations.
1) The people estimating the work aren't always the ones doing it.
2) Engineers are optimists, estimates are often best case.
3) Not enough padding added to estimates (see #1 and #2.)Every org I've worked for ends up like this.
There is a lot or intellectual dishonesty out there. It used to bother me. It doesn't now. I just don't partake in it.