All IT departments I've worked with are behind schedule.
All projects, medium to large, that I've worked on were delivered late.
Seems like making detailed specs and being more structured in managing the project using some methodology helps, but still, always late.
Is it just the nature of software development?
1. Requirements change frequently while the code is being developed. (If we had detailed specs, we'd be constantly re-writing them.)
2. People ask for estimates before I've had enough time to completely understand the complexity of the problem.
3. Higher-priority projects intervene, and when I go back to the original project (sometimes months later), additional time is needed to figure out what I was doing when I was interrupted.
Fortunately, my management understands that estimates are always going to slip in this kind of chaotic development environment. If we absolutely need to hit a release deadline, we usually start pushing less-critical features/fixes off to the next release. (I work on enterprise software that's sold to large companies, which has release cycles measured in months.)
Non-engineers probably liken feature and project completion to a simple shot at a bullseye but it's really more like trying to hit a moving target at the end of a labyrinth.
This, so much. People wouldn't change blueprint of a house being built 3 times in two months, but for software they act like it's not even a minor setback.
This is major for me at my current employer. It is not always a project that intervenes. It is more along the lines of "stop everything and fix my problem(s)" requests that are deemed urgent.
Also, and again as long as it's an environment your familiar with, the Donalds (the known unknowns) are usually relatively predictable i.e. you know the sort of things that will bite you and the sort of impact they have.
So I can usually estimate a project of 9-12 months duration to +/- a week or 2 as long as I've done at least a couple beforehand.
BTW that's not to say I always deliver on time. Depends entirely on who guessed the deadline and resourcing. But I can usually be able to be both annoying and smug by letting senior management know upfront how late they'll be and by how much.
Also worth pointing out that it's not because I work in a relatively static industry. Far from it. But if you have systematic errors then it makes sense to account for them. I do find it amusing that, in technical matters, most devs do this without thinking but seem to have a blind spot when it comes to planning or general project management.
Therefore don't automate things and don't make things more efficient. That will help with estimation.
--edit--
I guess this is something good programmers excel at doing, plus every-time get stuck with a time wasting bug, reminds me I have so much more to learn.
You really can't tell until you've done it.
2 weeks becomes 4 months.