Example: How long will it take you to lean advanced physics? Even if I gave you a week, your estimate is likely worthless (maybe precise, but not accurate). To me this makes sense because you cannot know the unknown unknowns until you hit them (and throw in some of the journeyman fallacy)
Meanwhile, how long will it take to brush your teeth is easy to estimate. How long will it take to brush your teeth with a new electric toothbrush is also reasonable to estimate.
I like that as an example to a layman of why this might be a hard thing to estimate.
[1] https://www.ametsoc.org/index.cfm/ams/about-ams/ams-statemen...
1) Experienced personnel are estimating the work (experienced technically and in the product being developed); 2) The work being estimated is of similar technology to existing product; 3) The time period being estimated is less that 6 months
If any of these are violated, e.g. new hires are given the estimation task, a years worth of work is being estimated, new technology is being estimated etc, i would not trust the estimate, and apply big 'ol bucket of risk money.
But then take the same company, even many of the same people, and put them on a new project, and the estimates go to crap.
It really takes a lot of domain-specific experience, and I mean "domain" pretty narrowly, like knowing the code base, knowing the developers, knowing the customer, knowing the external surprises and constraints you're likely to come across because you've seen them so many times before. I've never seen it work for greenfield, even in an organization with the discipline and know how to do it correctly for their older applications they've been working on for years or decades.
If I had to summarise what I have read on the topic, it would be that either way works, but the up-front analysis is orders of magnitude more expensive. Some sectors require this: defense, nuclear, medical, and so on. In most areas, you don't need that and can't afford it in a competitive market.
But to hint at the complexity of the question: Not only is every project different, there are also sneaky feedback loops in there.
By the time you have completed the large up-front analysis the world is likely to have moved underneath you, invalidating some of the assumptions that went into the analysis. You can shape your analysis to account for this, for sure. But at that point does it still count as one up-front analysis or several? Is it truly up-front if it leaves options and decision points open for later?
And it's a continuum: the more options and deferred decisions in the analysis, the more it starts to look like it's clearly done in parallel with the implementation. Where do we draw the line?
There is absolutely no problem with intensive up-front analysis. In fact, this is what at least 80% of the effort should go into. But we have pretty much 0% staffing for that role. It's no wonder something doesn't work when no one's there to do it. Compare that to M&S in civil engineering.
Regarding the alleged downsides: Any wrong implementation will incur massive cost down the road (but that cost usually is not accounted for in a SaaS world), so you actually cannot afford to not do it. Assumptions can change, yes, but at least you make assumptions. In agile development, people simply ignore any assumptions and end up with a completely broken product that eats 80% or more of development time with maintenance. And of course, planning and analysis can involve prototypes and iterations.
I came to the conclusion that agile development just shifts the blame for bad planning from management to engineering. And I also think that the correct framework to planning and analysis is domain specific. The more constraints your domain enforces on your software, the better your analysis and estimation will be. Hence greenfield projects have the most problems.
From the perspective of the seller, these estimates don't even need to be "accurate" as long as you're making many of them and the errors are symmetrically distributed. There's even math theory behind this called Fermi estimation. The problem is, from the client perspective, they can't average out the errors over many estimates if they're only buying one thing, and from the seller perspective, I don't think our errors are symmetrically distributed. Instead, we consistently underestimate how long something will take.
It's not even unique to software. There's a huge housing boom going on where I live, with condo developments all over the place, and not a single one is actually ready to open by the season their sign out front claims they'll be open. Defense acquisition. Highway construction. Everything takes longer than budget forecasts claim.
To avoid estimation you need to avoid dates and avoid projects that have high timeline risks. This means you would need to judge on speed to completion, or better customer satisfaction for project execution.