Its almost as if service based economies are not that compatible with property rights based capital markets.I don't see it quite that way. Services business work in our economy, but if you're doing something on a services basis, you have to understand the implications of that and how to manage them. There are all sorts of services in our economy that work (mostly) fine today.
To illustrate what I mean, let me use an example.. if you take your car in to a shop for repairs, you expect an estimate, and you aren't necessarily pissed if the final tally is off the estimate by a bit. But you will be royally pissed if it's off by an order of magnitude or more.
So, do software projects sometimes cost (in time, money, both, etc.) an order of magnitude more than estimated? Yeah, I'd say so. But here's the rub... are car mechanics better at estimating than software developers? Not necessarily, but their customer doesn't typically come along when the project is half done, and yank the rug out from under them and ask for something that negates half the work they've already done.
"Yeah, about replacing that spun bearing...I don't really want that, just replace the fuel injectors instead".
Nope. Doesn't happen. But I'll argue that the equivalent does happen in our world.
Of course the Agile Manifesto is all about the idea of embracing change, and accepting change even late in the process. So we have to accept that and deal with it. But the rub here is that we have do so in a way that doesn't come across to the customer as an order of magnitude failure.
This is one place where I think we, as an industry, kinda fell on our swords. WE understand agile, but we didn't always educate the customer on what agile meant. And we didn't do a very good job of communicating to the customer what the impact of their change in requirements is. We accept changing requirements, yes... but we haven't always made it absolutely clear to the business customer that "making this change is the equivalent of asking the auto mechanic to replace the fuel injectors instead of fixing the bearing. This is throwing out all the work we've already done, and we need to re-estimate everything now."
Is it because IT people lack the nerve to push back when business users make huge demands? Or something else? I don't know, but over the years I've become convinced this is a big part of the problem. We let the users change things (which they have the right to do), but we don't make damn sure they understand the implications of what they're asking for, and update the estimate/plan/etc. to reflect the new reality.