Speaking from experience at a large multi-year project at a BigCo, I am quite familiar with the failures of estimation. Some information is absent because some things are simply unknown due to inexperience or inability to predict future changes or mistakes (reasonable). However, other things are lacking because there is no motivation or demand to produce them. For example, waterfall processes generally throw the software development lifecycle on a schedule for a system or subsystem: requirements, design, implementation, test. If the lifecycle stages are not divided into tasks upfront (which then should be further divided), then the complexity of each stage is basically not being factored into the estimates for each stage. This is setting the stage for failure unless it is a project that has been done repeatedly over many years in the organization.
Furthermore, continuous process improvement and monitoring is not done enough even in organizations that declare achievement of higher levels of CMMI. Our team had subsystems that overran their initial estimates by over 100%, causing nearly a year in delay. However, there were basically no penalties or major process changes despite the fact that multiple subsystems overran estimates multiple times. Process professionals, engineers, and managers are simply not aggressive in tackling these problems.
In addition, estimates tend to come from individuals. I imagine that a more collaborative team-based estimation approach would be better, factoring different levels of experience and sharing the burden of making estimations realistic. Also, recorded estimates need to be coupled with recorded justification.