Why are software development task estimations regularly off?

The fundamental reason is that we start with a fallacy, that you can predict with any accuracy how long it will take to build a complex and likely poorly specified system in its entirety.

On top of that, there are two common development environments. In the enlightened one, management asks you or your team to go off and create an estimate. You come back and tell it to them, and you're informed that it's crucial to deliver to an important customer by date N. So you go back to your desks, lie to yourselves, then come back and tell management that you've recast the problem a bit and you can bring it in by the required date. Then you either blow the date, or deliver garbage by the date.

The more common environment, management either baldly states or subtly telegraphs the required date. You go back to your desks, then come back to management and show them the estimate, which will be done by the required date. Then you either blow the date, or deliver garbage by the date.

I've heard that some people don't do it that way.