It’s not like software engineers to underestimate how long something will take :)
I frequently see one answer fail - we just need to plan it more thoroughly, don’t just estimate, go through exactly what you’ll need, identify dependencies now not during coding. These efforts always fall for the same reason, when it comes time to implement, the plan is hopelessly out of date, incomplete and just plain naïve.
It’s easy to deliver software on time up to around 10kLOC (i.e. single developer short project). Beyond that, the variables cascade and avalanche exponentially. The problem with this is all the interesting problems need >1dev and certainly more than 10kLoc!
On balance I’m more persuaded by XP than by Royce but honestly i don’t think either approach makes or breaks, both can work well.
I think the issue plaguing estimation of projects is that a lot of software development involves running into problems and trying to solve them. Sometimes figuring out solutions to these issues is difficult and solutions appear as if by magic (in this was the art analogy makes sense in terms of inspiration). I’m not sure how we can ever estimate large projects accurately while we often need these moments of inspiration in order to solve problems.
How do you do that though? If we’re only talking about nailing down requirements (so lets ignore the problem of dependencies for now), then a 300 page BRD buys you enough spec to gainfully employ 4 devs for maybe a month-ish? I’m going off fin reg style work, i have no idea how that might translate to civil engineering so my numbers might be well off?
The BRD wasn’t free to produce, it contains 3-9 errors (assuming human rate of error is 1 to 3 in 100), and it’s being updated to v2 as you’re working.
How is this getting us all to the overall goal with predictable timing?
The problem is that in real-world testing, conditions around the changeover points produced behaviour that was not addressed by the spec. It turns out that dealing with radio-based systems in the middle of a busy rail network in the middle of a city is really hard. Many of the problems were caused by radio interference in the area. It's definitely the sort of problem practically nobody on this forum works on.