95% of the time the reason teams "need" that is bullshit internal politics and everyone is better off if you can structure the organisation to not do that.
When I worked for a larger company with a pretty mature product, this was true - deadlines were internal and arbitrary. It also helped that the product was more of a “nice to have” for our customers than “absolutely crucial” - they didn’t truly NEED anything it did (it was software to make social media marketing more efficient).
But I’m now at a smaller Enterprise SaaS startup, where the product is absolutely crucial to the customer’s work (core operational software for public transportation agencies). The reality is that the product isn’t yet mature enough to close big sales without making sales commits. I dislike sales commits, but we’d legitimately lose every big RFP unless we commit to adding the ~2-5 features they really, truly need that we don’t yet have. And sales commits come with real, external due dates, that you need to be able to meet if you don’t want to piss off your customers. You need to be able to say “yeah, we can build X by date Y”, and then actually deliver on that.
That's just pushing the issue one level higher. Most likely both the customer and you would be happier if you were able to iterate with them rather than building the thing they thought they would want however many months back. (And while it may be harder to set up that kind of contract, it's absolutely possible)
Possible? Sure. Just like it's possible that I could win the national lottery.
In practice, multiple companies bid for contracts, and all of the bids have to be of the form "Deliver $X in $Y months for $Z dollars".
No company is going to accept "Choose us, and we'll work until you're happy with the product or you run out of money, whichever comes first".
Agile doesn't work where there's competition for the project. It can't, because the buyer of that project cannot evaluate "we'll work until your money runs out" against the other bids that state "$X in $Y months for $Z dollars".
Agile works well for internal teams that are sheltered from having to provide competitive bids.
Which still loses to the other bidders who say "$X in $Y months for $Z dollars".
Typically, the intermediate step is to tie some compensation (such as a bonus) to a particular metric, but why not just go whole hog?
Even if X% is projected to be larger than $Z (and in fact, the better the project succeeds, the larger the %X total ends up being, but you can cap it, or make the % progressive up to a point and regressive above it), the advantage for both client and vendor being aligned on the upside is considerable.
“We’ll work collaboratively and evolve the product over time in yet-to-be-determined ways” can work when you have an established relationship and 2-way trust, but there’s not much of that during an RFP. If a competitor commits to their biggest asks, and we don’t, we lose the RFP.
As almost everyone knows, construction delays and cost overruns are the norm, not the outlyer. Construction projects here very often suffer dearly from the "2 more weeks and/or a couple more thousand dollars" overrun issues.
In fact, I'm sure at least several people who have been trapped in a home renovation nightmare LOLed deeply after reading your post.
These stories are SO common (and I've seen it in my own family too) that I truly question the sanity of anyone who takes on a home renovation project.
Houses (not housing tracts) tend to be built by much smaller contractors who are in absolutely no position to absorb any setback without going bankrupt, you can't apply the rules for 'big business' to small time operators and expect the exact same outcome.
Constructing resorts also has all kinds of economies of scale that building individual houses does not.
This is very doable in software, and I’ve seen it done for decades from all levels of the org chart.
Overshooting deadlines and increasing costs due to materials not being available or them suddenly costing twice as much is normal.
Also coordination between all the various contractors is always getting messed up because the person putting up the scaffolding didn't show up, so the plumber cannot work and cannot come back for 3 weeks, which messes up the plan for the drywall, which gets pushed back, and so on.
They could probably do it more accurately but it would lose them bids, so they knowingly low-ball it. You shouldn’t have that problem inside a company doing software.
Engineers low ball partly because they don’t know what’s involved in a new piece of work. Like walking between two points on a map; they don’t know what they’ll run into on the ground.
Managers and stake holders then bully the engineers into lowering the estimates because even that’s not quick enough. Worse, they scale the team without discussion without realising that will make the project harder to deliver.
The only way you are going to get a hard quote is if you buy a prefab or a house that the contractor has already put up multiple times (and even then there will be some room for adjustment). Oh, and foundations are a completely different story.
Or this: imagine seeking an estimate on when a house will be built, trimmed, painted, furnished, decorated, and filled with every item needed practically and for comfort by the homeowner.
It’s a living process that requires continuous reassessment of priorities and scope, and in fact never finishes when you consider the life of the home.
It absolutely is; look at some of the biggest and most successful software companies in the world. No-one has any idea what facebook will cost or when it will be done (Zuckerberg has said that he was off by at least 5 orders of magnitude IIRC). No-one has any idea what google will cost or when it will be done. No-one has any idea what amazon will cost or when it will be done. No-one has any idea what netflix will cost or when it will be done.
By contrast, software development is doing stuff that hasn't been done before. You may be lucky, and have done something similar. If it's really been done before, you would ideally be using something off the shelf.
For me, a project manager who uses this analogy has no place running a software project.
Also, tons of software dev isn't actually about pushing boundaries or doing something that hasn't been done before. In fact, I'd say most development is about implementing known solutions but in a customized way. Just like every other engineering field.
And the project managers who use the analogy don’t make the comparison with significant and novel civil engineering projects. They make the comparison with having their kitchen extended.
That's the reality of software development.