Further, I can work with you to figure out what portion of each project will give you the biggest bang for your buck. So you can say, "I want to spend x00,000 dollars." -- and I come back and give you a demo each week that you can play around with. At any point, you can say, "This is enough to generate value for me, let's move to the next thing for a while."
Further still, I can build it with enough test coverages and good design practices that it will be possible to extend upon the design at a later point without scrapping everything.
On the other hand, you can spend half of that x00,000 developing a specification. If detailed enough, I can give you a very low variance estimate of how long it will take. However, you won't know if it will actually meet your needs until you see it. You won't know about problems in what you really need until it's too late, and you'll end up spending more money in the end.
That's the message that should go out.
Perhaps the biggest reason this keeps causing problems is that companies have no good way of dealing with change. If you expect to finish any project that you start then you need to know more about that project than is realistic at the time you start it. A canceled project is a big failure for most employees and not something they want to happen to their careers.
However, until you establish reality, all the estimates in the world aren't going to help much. And most customers cannot estimate reality until they are actually in the process.
I'm not saying Agile is a silver bullet -- it can go wrong in many ways, and it's not appropriate for every situatoin. However, it's the best we have for its niche.
In order for a business to be investable, it is necessary to demonstrate that it will be profitable to an investor. (And of course all businesses need investment, whether that is VC money or just a lone developer's part-time effort.)
We have avoided this problem because in the early days of software, most software was profitable. But as the software industry has grown, this is proportionally less true. And I seriously question in 2014 whether more LOC are committed each day to "black" or "red" projects.
Now I'm imagining a website like oDesk or ELance, but which requires employers to "defeat the null hypothesis" for any job contract they want to post. What a wonderful world that would be.
The first, as I believe you presume, is that the business is looking to develop a revenue-generating product. I agree that Continuous delivery, quick iterations, late-binding requirements (a la Behavioral Driven Design), and rigorous testing are not a strong candidate for this problem within the context of big business, and I can point to a handful of systems where this approach did not work well.
However, the second interpretation is that a big business has ten internal projects to increase their efficiency. In this case, return on investment is relatively easy to calculate, but requirements are usually nebulous and thus estimates are by necessity high-variance.
I think the subset of agile techniques I described (as well as cross-functional teams, high customer involvement, and exploratory testing) are well suited to this problem because they, as a collection, allow a business to receive value quickly and allow the business to "fail fast" in lean startup parlance.
Are you building embedded software? The agile approaches aren't, collectively, well suited for that domain (though I'd argue pair programming and cross-functional teams aren't a bad idea here.) It's not going to work for games. But for internal software and <10 client software, I argue that this subset reduces risk and increases the potential of high ROI. ___
Now, as for a few things you mention.
* The programmers are not responsible for calculating return on investment. However, does an extensive requirements gathering mission with months of meetings going to be cheaper when the costs of meetings (in lost productivity and wages) is calculated? In my experience, these documents end up being works of fiction, on which aggressive estimates are produced upon pressure of management. Then, the costs overrun (especially considering that nobody accounted for the lost productivity of the customer) because the estimate was based on business need rather than reality. Then, features are dropped and technical debt accrues, leading to a system that is hard to maintain and is scrapped after five years of frustration when the customer starts again. At least, this time they know what they don't want. I don't think that works.
* I think that is the situation that leads to the "red" projects that you mention. However, you speak about early days of software where most software was profitable. However, I've seen and heard the horror stories of multi-million (and Billion!) dollar projects from the 80s and 90s. While http://calleam.com/WTPF/?page_id=1445 mentions recent studies, I remember seeing studies that 2/3rds of projects fail in the 80s and 90s as well.
So, most software wasn't profitable then (at least in terms of internal projects) and they aren't now.
However, I'd argue that the collection of techniques above allow a project to fail faster and cheaper if managed correctly.
If we were to put it into a methodology, here it is:
1. Determine return on investment if X can be automated. How many hours are saved? How many fewer people are needed to do the same job? What percentage better forecast can this project create (estimated)
2. Use some sort of behavior driven development to get a basic sense of the complexity of the system needed. If the complexity and ROI don't match up, stop here. You've lost a minimum amount of money.
3. Start prioritizing which pieces would provide the most business value for the least complexity cost. Build the first of these pieces, elaborating on the BDD done in step 2. Write this with good tests, so that the system can be extended with a reduced fear of later regression.
4. Demo to customer. Does this fit their needs? Is additional complexity exposed? Does this provide value? This is the next cutoff point. If the system appears to be more complex than thought, or if the team is unable to provide the business value anticipated, stop the project and re-evaluate or shelve.
Everyone gets together and does a retrospective with a question to continue, an honest look at what worked and what didn't, and a set of things to try to fix the problems of the first iteration.
5. Repeat steps 3 and 4 until the software either sufficiently meets the business need or the project is ended early due to a fatal flaw. Always be willing to ask if this is the appropriate time to declare the system "done for now." ___
Now, this won't work in most major corporations for a variety of political reasons, but to me this seems to be a better system than the traditional corporate project structure from my experience in both successful and unsuccessful IT projects.
If that's the actual answer, then anything else is just obfuscating the truth.
Why should we encourage lying to try to cover for the fundamental lack of knowledge? If you don't know, trying to make up numbers never makes things better.
A surprising number of people in business think this way: not that they need to make sure of something, but just that they need someone to tell them everything will be OK, and then put the blame on if anyone is unhappy. That someone will naturally be whoever is at the bottom of the totem pole.
Managing ego and heart is harder than managing a project i think.
Gerald Weinberg
http://en.wikiquote.org/wiki/Gerald_Weinberg#The_secrets_of_...
http://www.codinghorror.com/blog/2008/01/no-matter-what-they...
Asking for efficiency and adaptability in the same program is like asking for a beautiful and modest wife. Although beauty and modesty have been known to occur in the same woman, we'll probably have to settle for one or the other. At least that's better than neither.
For example, since a time estimate can trivially be converted into probability estimate for something happening on time, all of this is relevant: http://lesswrong.com/lw/3m6/techniques_for_probability_estim....
The best technique from that set, I usually find, is betting (with actual money) on results: setting up a intra-company prediction market for when your software will ship can tell you more than you ever wanted to know about how much time it'll truly take.
There are three possible reasons for an answer of 'I don't kmow'
1. Something is unknowable 2. The person being question has insufficient expertise to discover the answer 3. The person being questioned hasn't really thought about it.
You appear to be assuming that 1 is always the reason.
What I'm saying is, that at the moment someone asks you "How long will X take to bring through the software development process", "I don't know" has to be a possible answer or your whole method is corrupt.
And that's the problem with software, there are no equals. And there are always "surprises".
A car manufacturer knows how long it takes to build a car, because it does that thousands of times for a same model. Now, designing the car, that's a different issue. (And don't give me the BS enterprise "model" of architects throwing some UML from a high tower and "developers" filling the voids because that's the worse way you can build a software)
Ok, it's one thing to build a simple ACID application, another thing is to build something that "has never been tried before" .
The order of magnitude approximation is good enough, and for ones in which you can't make an order of magnitude approximation you should be able to back it up.
e.g. c.a. 2002 a PMM said "All we need for this idea to work is speaker-independent voice recognition in a noisy environment; how long will that take?"
The correct answer to that question isn't a time estimate, or "I don't know" it's saying that doing so is an R&D problem, not an engineering problem.
The company does not know how much revenue each of those ideas will deliver.
But seriously scrum with concept of velocity and estimating size of tasks is the best approach I've seen. That's if ou want to ask programmere at all. Preferable way would be to predict the duration of the project scientfically from metrics gathered in the past (of not only the programmers and projects but also the environment they operate in).
How can a programmer estimate when project will get built if he doesn't even know if anyone actually cares enough about it being built to actually decide what the software should do?