Waterfall Process
martinfowler.com
martinfowler.com
Whenever I took over a project I would tear up my predecessor's plan and immediately start to build a prototype of the component. The plans would have no wiggle room for discovering a big oops years later. Did we choose the right materials? Will that mechanism work in the cold? I would do the first iteration myself, building the product our of wood or cardboard or plastic. Then I would move to a prototype hand crafted out of steel. Then I would order stamping tools made from Kirksite (zink) and test those versions before finalizing the design and committing to the final tooling. So, yes, iterative design/build works for physical systems too.
In a high trust environment scrum or xp or any other agile process is great. Of course in a high trust environment any process that isn't actively perverse is great. In a low trust environment no process is great, or even good. The best option is less bad.
Not according to the highly paid consultants with strange acronyms following their names...
Micromanagement looks a lot different to me.
> It is possible for a mix of waterfall and iterative where early phases (requirements analysis, high level design) are done in a waterfall style while later phases (detailed design, code, test) are done in an iterative manner.
I prefer this approach where time is explicitly dedicated to planning out the solution, which all stakeholders agree to. It also respects that as you begin implementing the solution you'll learn more about the problem domain and adjust accordingly.
I don’t believe I’ve met anyone who ever said, “Hey, let’s do Waterfall!” More often, processes became waterfall-like when feedback loops would become too long. Sometimes, teams wouldn’t get feedback for months. Interestingly, some of the scrum teams I’ve worked with recently have succumbed to the same thing because product owners aren’t active in the process.
But, you’re right - “agile” is often followed as a way to get more done with fewer people. But, unless you’re starting with a heavy weight process to begin with, you’re not going to see much productivity gain. A process isn’t going to miraculously make your developers write code faster. But, used properly, it can lessen the risk of rework.
Sadly I see a lot of this today too. I know someone who has worked on an "agile" project with two half hour long standups a day, two hour sprint planning, hour long retrospective, and various other "ceremonies". I'm not sure how much of the sprint comprised of actual work.
> Sometimes, teams wouldn’t get feedback for months.
Feedback is so crucial. It frustrates me that business process and politics sometimes gets in the way.
> unless you’re starting with a heavy weight process to begin with, you’re not going to see much productivity gain
Agree 100%, The scenarios I've mentioned have always been short term (~4 weeks) client engagements where there should've been absolutely no room for needless process. I worked on a project once where we had weekly sprints (!) with a playback with key stakeholders each Friday. Each week the stakeholders would change, so each week we needed to spend at least a day planning for the presentation.
1. It's focus on the User and the business case. In the RUP documents you have to describe what business problem you're trying to solve and make a complete list of stakeholders (this can be a wide variety of people, not just the main app user).
2. During the elaboration phase you have to describe the list of requirements and tie them to a business case - so you're forced to think about the problem you're solving.
3. During design phase you have to tie each class/module back to a specific requirement (this forces developers to focus on the real problem as stated)
4. Use cases are a fantastic tools for describing system interactions with the user, and force designers & developers to think about and provide solutions to the not-happy path in a very quick iterative way. In your use cases you should address the main requirements of your stakeholders and tie them back to the requirements doc.
5. RUP has a specific phase for transition to live, and so forces you to think about deployment, architecture, config and maintenance even before code has been written. Yes, your IT dept is a stakeholder.
6. RUP doesn't care what you use during the construction phase - following an agile methodology would be ideal in fact.
7. RUP doesn't care what you use as a project management process - it will have documents that clearly show the scope of your work and the list of prioritized requirements you have to fill before you're done, which is great for tracking progress.
8. RUP is iterative, so if you discovered something important missing during construction - go back to elaboration and work the new requirements through your design.
9. Creating and keeping up to date a fairly small set of well designed documents will lift the quality of your product.
This is just my personal experience, but I found I learned a lot following the methodology and Rational's standard documents. Not only did I produce better solutions, I also found that the documentation could be shared and collaborated on with other people within the organisation, and so Business Analysts, testers and the system managers were all stakeholders and contributors throughout the whole lifetime of the project. They loved that.
Are you kidding, the 90s were filled with same for and against arguments of RUP as we have about Agile and Scrum now. No process is right for everything so you will have people misuse a tool and blame the tool. The same orgs that micromanage Agile teams would micromanage RUP teams and people will be just as angry about it. RUP is a tool and is as good or bad as the people using the tool.
That said, I agree with you that RUP has some benefits.
_BUT_ you cannot develop software without following the steps of the waterfall model. You need analysis. You need testing. And development has to happen in between.
And here comes the problem: When you have a very fast iterative process, you have everyone working at high pace. But your quality will invariably suffer. If your analyst needs to come up with a decent distributed numerical solver for differential algebraic equations, they won't be done after two weeks. So their 'increment' is going to be incomplete. And no, Pareto does not apply to software specifications. The same holds for any other complex domain. And if your testers only have time to test for two weeks, they cannot cover as much as in half a year.
So yes, iteration is a necessity and quick iteration has its merits. But it also comes with a cost and choosing the right pace is as difficult as it is important.
Hardware comes at a very fixed iteration speed, usually around 3 months for something of moderate complexity. That's how long it's going to take to do schematic capture, layout, pcb manufacture, board population and bringup. Even if you can speed up this iteration process, there is still a sizeable financial cost of going around the loop.
This means that at a project level you need to do a certain amount of requirements capture before you start work on electronics. You can then run your software in a fairly agile manner 'inside' the project, hoping that the inevitable changes in requirements can be dealt with in software. Obviously you design your system in such a way as to increase that likelihood.
Developers, the customer, management, they all might give different answers.
And as far as I can tell the only true answer is: good communication.
Agile does this with short waterfalls and stand-ups, but other methods also work well as long as there is communication.
And I believe for most part this is where you can distinguish a good managers.
Good managers feel when the developers need someting and feel when the customer want's to go in another direction.
A good manager is a medium, not someone who dictates what the developers should do.
Sure, it's fixing things that break as things are used, but it's not like it's replacing something that gets worn out with exactly the same thing, or cleaning, or IDK enough about what actually goes on for physical plant maintenance. Things like that do exist - log rotation, restarting processes / servers because of memory leaks that aren't worth tracking down - but, mostly, seems like software "maintenance" is more like post-release iteration cycles.
Some of these are about the pure tech ("here's a consequence to this design when under use that was not anticipated / handled"), others are product design ("turns out it needed to do something a little bit differently") or additional features necessary to make something useful, or just additional features to make it more useful. Or, you know, it was just built wrong the first time.
Seems like all of those situations are actually just more of the same iteration cycles as during development, we just call them "maintenance" for some reason.
(well, except that maybe they needed information and observation that are only available once it's under use...)
For some businesses maintenance time never happens. For some it is an important stage before moving to the next iteration of new features. For some it is a daily process with dedicated teams. Every case is different. Agile is about practical agility :)
"In the agile world, success is all about business value - regardless of what was written in a plan months ago. Plans are made, but updated regularly. They guide decisions on what to do next, but are not used as a success measure."
so, answering your question with that in mind, the focus should be on business value. does maintenance provide business value? do new features?
Yes, and yes. In a mature company, maintenance provides more business value than new features. To list some of the maintenance activities:
- incident response. It is not uncommon for a company to have losses in the tons/hour or millions/hour if the primary process has a malfunction.
- organizational changes. No software exists in isolation; changing requirements in a connected component may require changes in your software component too. Developers may think this kind of work is feature development, but from a business point of view, it's simply maintenance. Because of the dependency by other parts of the company, this kind of work usually has stricter deadlines than "new features".
- Predictive maintenance. Identify weak points in the current setup, and isolate of strengthen them before they cause an incident. This includes evaluating technical debt. The value to the business is in risk minimization, and the rationale for this kind of work is predicated on proper risk accounting.
These requirements and processes are part of any proper engineering discipline; these are not marginal issues.
Why not? Because the customer already paid. Because Dev wants to focus on getting new customers instead of catering to the ones who have already forked over the money.
"What do you mean feature X isn't working for the customer? It's working for us in our lab. Anyway, we're all focusing on getting new feature Y working in our lab now." - This is the attitude I am sensing below the surface. It's frustrating.
Something along the lines of, "I know what the finished product is supposed to be, why does it have to be broken down into incremental deliveries?" I think that non-tech people often see it as a kind of watering down of their vision.
Getting the budgeting into the hands of the fluid planners (product owners) is critical to successful iterative outcomes.
The alternative was to make shorter iterations. Our professor called this "Cinnamon Bun". Instead of one big waterfall, covering the entire project, you'd break the project up into chunks, but apply the waterfall steps to each chunk. Planning, Analysis, Design, Implementation, Maintenance were the steps. Each chunk would vary, but we typically broke it up into month or two chunks. Of course you'd need planning and analysis up front to figure out the best way to divide the chunks.
When Agile/Scrum came out, its proponents compared it to the shortcomings of Waterfall, which was disingenuous because pure waterfall had been largely abandoned by at least the mid 90s, and probably sooner.
As an aside, One important step in waterfall was Maintenance. This seems missing in current methodologies as a formal step. If you write successful software, the maintenance phase is 80-90% of the softwares lifecycle. Understanding this equates to the understanding that writing maintainable code is one of the most important things a development team can do.
I believe Conway had it right, back in about oh, '67?
"organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations."
— M. Conway