Deadlines are so ridiculous, has anyone ever actually hit one?
There is one very important distinction. When the requirements are unclear or non-existent, then the project could only be for the creation of the missing specification and repeat the process once the specification was signed off.
As you may guess I have had many board level meetings where the stakeholders were horrified at how long things would take. That's where the politics start. Typically somebody pulled a deadline and budget out of their a* and demanded that I make it happen. That's where being a professional is very important. They either meet my demands or I decline the project. For example, in one situation I declined a project. They brought in a succession of 5 project managers and then axed the project years late and millions over budget. If it is impossible, then no amount of table banging and shouting is going to repeal the laws of physics, etc.
You only need to read up on failed projects, like the FBI's Virtual Case File System (great write up in IEEE Spectrum), to realise that projects failing to meet deadlines are typically a result of management incompetence and rarely due to failing by the technical team - but that too does happen, especially if the specs are too loose and the team leaders fail to maintain focus.
I do a lot of "R&D" type projects that have no deadlines. But for the projects that have hard deadlines, my company has only ever missed one in our 5 years of operation.
We do this by not having any 'political' influence on schedules. Our engineers make the schedules based on information they have about the tasks. If we don't have enough information, we refuse to declare a schedule for that task.
I think a lot of companies are effectively doing research projects, and are trying to schedule them. You can't schedule learning, or creation of truly unique concepts. If a project requires research, we split the research out as a separate task and refuse to give numbers for it. Instead, we explain the approach we will follow.
The problem is in the way some companies set deadlines. A bunch of people come up with everything they could possibly want. They pick an insane deadline. People point out that it's impossible. That is, a 0% chance of success. Revisions happen, ending when nobody can prove the the date is impossible. Now they have a 1% chance of success.
A saner approach is the one I first learned from Extreme Programming. Basically, you take every feature that anybody could want and put it on an index card. You put the index cards in order of importance. Do them one at a time, and make sure you have something releasable every week. The first version will be terrible, of course, but eventually the pain of releasing gets smaller than the pain of not releasing.
Using this approach I've happily been very early for deadlines. Sometimes we moved the date up; sometimes we kept the original date and just kept doing less important things. I get to keep coding while product people fight over which cards go where, because I just care about what's at the top of the stack.
I'm at the point now where we don't even talk about deadlines. We put things in order, we do 'em, and we release as soon as possible. We don't do formal estimates; we just keep the cards small (average ~0.5 days, maximum 3 days). That saves a lot of wasted time and stress.
The simplest one is just to do those things as features require it. Most non-developers don't care about what the details of you're doing anyhow, so just don't tell 'em.
The next way is to maintain a technical queue that's parallel to the main story queue. We use this mainly for improvements that speed the development process generally but no particular feature requires it. It's just understood that developers will do some of that each week. As long as we keep delivering other stuff, it's fine.
The hardest way is to put technical cards in the product queue. I tend to reserve this for things that are large enough to have schedule impacts because getting a product person to understand the importance of some technical thing is always an uphill battle.
For a lot of things, the real trick is to turn something from a "technical" card into one with direct end-user benefit. E.g., putting in a load balancer. Instead of talking about that, I write cards like "system handles X simultaneous users" and "system survives single-box failure". I'll still advocate for them, but when do do them ends up being a business problem.
If they want a deadline, you can always count the number of cards left in the stack, and say num cards x weeks. If they're unhappy then you say: "pull out the cards you don't want delivered".
The "N cards / X cards per week = Y weeks" is perfectly reasonable. But you can also make this work with a more formal estimation process for shops like that. When I need to, I use planning poker, velocity measurements, and occasional re-estimation as part of release planning.
Usually this indicates that management has stepped into fantasy land and doesn't want to acknowledge something about their business (processes, limitations, staff, industry, etc.) and would like to use deadlines as a way to maintain hope at the top for the future. It's not like this at all employers, but it is staggeringly common. I find that "top heavy" companies tend to suffer more fantasy-thinking management problems, but I'm not an expert.
If you're planning on staying at your employer, you can try to help the process by more strictly defining the deadline scopes or by breaking up the deliverable scopes into more manageable chunks. If your employer has any hope of improving, your more-realistic-but-not-as-optimistic deadlines will be met with opposition at first and then acceptance when you start actually meeting them.
Obviously the larger in scope the project, the more likely the deadline is going to slip because more things can go wrong. A good project manager can help anticipate this and mitigate risk.
"How is progress going?", rather than "Are we gonna meet the deadline?"
-- Douglas Adams