Is there a better way to do this than agile? No duh, but that's another topic. :-)
I have two problems with this.
One, almost all work that is easily estimated falls into the realm of trivial work. Most work is not trivial given the rapidly changing work environments. If it isn't the tech itself, it's the people around you and above you. If it isn't the people, it's some context which in common human fashion, has become overcomplicated to high heaven.
Two, because things change so rapidly and riskmanagement is the way it is, we are in a perpetual state of businesses trying to pressure one another into getting the best short term deals at the cost of almost everything else. How many times has a deadline not been met with it being critical? Not that often, most things can wait another month. So why do we keep making unrealistic deadlines? (Semi-rhetorical)
If there's some set date (eg. demo/launch at big industry expo) it makes sense keep that constraint in mind, plan work in small chunks, iterate, and always try to deliver tangible things needed for the demo ... and it also makes sense to periodically check in (let's say at the end/start of every iteration) how the demo is coming along ... and it again makes total sense to adjust course if it seems that progress is better/worse than expected ... and it makes sense to try to do this tracking and estimation in some consistent/rigorous way, instead of just gut feeling.
"And that's how story points were born". (Which is just a fancy name for reference class based forecasting - https://en.wikipedia.org/wiki/Reference_class_forecasting )
And, again, if there's enough reliable data it's possible to give educated guesses about how the remaining tasks compare to the already finished ones.
...
Now, doing this top-down, from day 1, expecting 100% accuracy, and whatnot is complete business madness, and anyone who does that needs to be thrown out of a fucking (ground floor) window again and again until this gets through their (apparently) very thick skull.
...
I've seen this done well. (Not the defenestration.) And seen this done horribly. The "methodology" is not magic, and it definitely doesn't make sense if the people trying to "do" it are senseless imbeciles forcing things they don't even understand on people whom they also don't understand.
Usually what's not trivial is how much to build and when (which is product design).
> it's some context which in common human fashion, has become overcomplicated to high heaven.
yes indeed :)
If business (and/or management) is too neurotic, that definitely will fuck things up and it doesn't matter what methodology you try to manage projects with.
I used to scoff at the first dictum of the agile manifesto (individuals and interactions over processes and tools), but it fits exactly these problems. Business has to plan and build on what Engineering already does, not try to force things down onto it. Management has to work with the people, not try to force the people into their own crazy world.
And if they don't, then out of self-preservation, leave.
I've literally had clients go "Hey, can you add search to the platform? Can't be that hard, it's just a single input field on google dot com".
Your task is to tell them that this is too big to estimate and needs to be specified better and broken down into managable chunks. Well ok, in this case, just telling them to sod off would've been smarter, but that's a different story.
The common pattern I've seen is that everything is about estimates, which end up becoming deadlines. The management worry about them to the point of bloody-mindedness, at the neglect of gathering actual requirements from the customer. In my company, I work in a software department (I suppose we're an internal agency with just one customer). But, we can't speak directly to the customer and instead speak to another Project Manager (apparently because of "SaFE Enterprise Agile" policies), so we can't ask the customer what they truly need, and so the product isn't accurate, and neither are those estimates!
Entering a contract with a client based on an estimate without a large margin for error is a statistical failing, not a strategy one.
That said, I still think waterfall and other related variants are overwhelmingly the way to go. I've done both styles and the difference is night and day.