The Myth of the 'Waterfall' SDLC (2019)
bawiki.com
bawiki.com
Instead of using the term waterfall as the counterpoint of agile I prefer to call it the human centipede model. In this model all of the vision, creativity, and flexibility stays with the head and the rest of the centipede just eats their shit. Developers can't see further than the next person's ass and have no idea why they are actually building the functional specifications that are fed to them. Implementation becomes completely disconnected from design which leads to compromised quality, missed deadlines, and products that miss the mark.
No task management framework is going to solve these problems.
If I design a machine, and I need a gear or a cam or even a more complex component, I can send the specifications for that part to a machinist who knows nothing about the "big picture" of what I am building. Yet he can make that part of it to perfection. Does this lead to the same sort of problems with the development of physical systems?
I would argue that physical systems aren’t developed in a “waterfall” method.
In mechanical design classes in engineering school, we learned to start with low fidelity sketches to capture customer intent. We come up with several options, build those into increasingly higher fidelity 3D models, use simulation to refine, take a few candidates and get physical prototypes, do physical testing (strength, endurance, integration, etc.), determine the best one, then go into limited production to prove out and establish the production process, then go into full production.
We are taught (and in the automotive industry it’s a requirement) to have cross functional teams involved in the design stages of both the item and the manufacturing process.
To your point about inadequate spec: My opinion is there are so many different backgrounds coming into software that there is no common language or background.
I think everyone thinks their way is better and it’s hard to communicate technical ideas when you need to constantly recreate and translate between terminology and documentation methods. This lack of convergence and common knowledge is what I think results in poor specs.
So software development are all about writing down the specifications precisely enough for the compiler to create the artifact. Mixing up this process with manufacturing is a source of many of the problems in software development.
It depends, the idea of working in short sprints with deliverables at the end of it is that often you do not know what you need and the best way of finding out is by trying out and talking to the customer.
The place I'm currently working at is agile on paper but 90% of the work is just implementing external requirements we have no say in. Needless to say, not analysing the problem up front and not designing the general architecture up front leads to building roadblocks later on that prevent getting work done.
To stay with your metaphor: Sometimes you have to eat shit (because of legal requirements, etc.). Planning ahead is very useful in those situations for not choking on the shit.
I don't know where this "rigid" model comes from, but I never encountered it in the early days. Everyone knew you couldn't just complete a phase in totality and then move on to the next; there had to be overlap and stepping back up to an earlier phase as you discovered new things.
I suspect the "rigid" variant is merely the hyperbole that waterfall has been reduced to since there are no more proponents of it left. Doesn't make the Royce variant useful, though.
Speaking with refugees from megacorps, I do have the impression that bigger projects had different people (subteams) doing the different phases, with little overlap. I have no trouble imaging how that'd quickly become a Kafka nightmare.
But then again nobody on the engineering team took them too seriously anyway, so it kind of worked out.
The main problem in those days was centralization of decision making at the top, where the decision makers had the least exposure to the reality on the ground. The military legacy of computer systems, I suppose...
I helped a friend with some property tax software for governments. Trivially technically, almost intractable organizationally. Not for me. I deeply respect the people who can swim those waters.
PRD -> design doc -> code -> launch -> promotion.
Of course, it's broken down and full of a mess (like a real waterfall) at most stages.
I feel like in a way it was a disservice to my career having a decade and a bit of work in smaller companies (where we often shipped quicker and with a less dysfunctional process) before coming to Google. Because the people around me seem to find the way things are... all quite fine and dandy while I just stare in cynical disaste.
The new team I'm on seems better, though.
"We want the project to be delivered by X, and we have a budget of Y. Our specs are Z, but not really, so factor in A refactorings (A for arbitrary). Oh, and we don't have Q for Quality defined."
Especially prone in government and military contracts. Throw in some arbitrary public procurement rules so that Y is allegedly not disclosed, except to the select contractors that somehow know what the desired Y is and what the competition is bidding before bid close.
You get the F-35, which pragmatists say was never meant to fly, just to make the players rich in the process of failure.
Sauces: https://www.forbes.com/sites/davidaxe/2021/02/23/the-us-air-... https://pando.com/2015/09/24/war-nerd-why-f-35-albanian-mush...
Agile Methodology was a self-defense coping measure for dealing with psychotic customers who cannot or will not do proper project management.
It was never meant as a replacement for PMI.
IMHO, "waterfall" is not having feedback loops, iteration. All of the methodologists (I read) in the 90s cautioned against "throwing it over the wall", as detailed in this OC's description of rigid sequences.
Plenty of today's "Agile" lacks proper feedback loops. Where primary assumptions are not revisited, when new information does not lead to course corrections.
Maybe "waterfall" is just dysfunctional communication, where reasonable people aren't talking to each other.
Any way. This is a great write up.
The thing with a client/customer work is the clock is ticking at a constant rate and the budget is decreasing at a variable rate. I’d like to research done on methods that try to ensure the two run out at the same time. If you finish on time with budget remaining you’ve left money on the table. If you run out of budget before the deadline you’re in the red. The goal is to run out of budget the moment you reach the deadline.
If you have to misrepresent the alternative (false dichotomy anyway) then maybe you don't have anything of value to bring to the table. That's what it felt to me.
https://acqnotes.com/acqnote/acquisitions/milestone-overview
This is literally the way the US DoD has done acquisition for decades. In the early phases there are often (but not always) multiple R&D efforts which are evaluated later (in the aircraft world think about the various X aircraft that compete to become the next fighter). But once it's selected, it's very waterfall-esque from within the context of the program office which encourages (but does not mandate) a waterfall approach from whoever is doing the work as well. They do periodic check-ins, but they're basically waiting for the entire thing to be produced before it can be evaluated properly. The checkins are just smaller milestones, again not necessarily increments which could standalone if the project were to stop there.
And yes, US DoD does big bang testing at the end. They have test organizations which receive software/hardware systems (cyber-physical systems is a common phrasing for them) and evaluate them at the end of the development process. Ideally, of course, the development team is also doing testing. But even then, in software, they tend to do scattershot testing during development with one big bang test run at the end.
TBF, in the defense and defense contracting world that is not an unusual situation. It's a terrible situation, but it's not unusual. I've seen multimillion and multibillion dollar mistakes caused by this repeatedly.
It's improving, but SAFe is not the answer. If it were, the F-35 may not have been such a clusterfuck. SAFe is overly prescriptive and inflexible, it's a way to make managers happy but does little to actually improve the situation.
This talk from one of the authors is an amazing rebuttal of what agile has become and clarification of what it is supposed to be https://m.youtube.com/watch?v=a-BOSpxYJ9M
Despite that, something felt really off. Requirements and scope kept popping out of nowhere. We tried moving onto something new yet requirements kept popping up from things we didn't consider.
The project was somewhat of a legacy refactor so it was easy to say "just redo X in system Y" but for some reason it didn't work out that way. I think this project could have used some the old waterfall paradigm where you did a lot of the requirements analysis up front.
Sure, you're not going to capture everything, but in this case I think it would have helped everyone involved if there was more foresight put into it up front instead of continually bumping things into the night and making stories for it.
A bit like the KickStarter campaigns: bucket features into MVP, stretch goals, and bonus.
This was mitigated by having a tempo for releases, so everyone knew those extra features were coming soon.
No process will save people from themselves.
If you tried to collect the requirements up-front, the only thing you would get would be to either stall the project before it starts (what could be a good thing), or to do the exact same process, but only after a months-log delay that created a bunch of stuff you would simply throw away.
Nobody defined 'Waterfall' as contrasted with Agile any more than 'they' argued constant climate as the Climate Change folks would have it.
Nevertheless, I have seen, especially in a government context, the very bureaucratic rigidity that the Agile practitioners decry. SAFE wasn't begotten in a vacuum or as a sales driver.
Companies like Oracle are cargo-culting by building cloud platforms that resemble AWS but they have no appreciation that Amazon changed their internal communication practices resulting in AWS! Oracle isn't going to do that, perhaps they can't. Has Microsoft adapted their communication structure and is it reflected in their cloud platform?
What I find interesting is that cloud service adoption is allowing silos inside companies to avoid the structures that impede them--to some extent.
Adjusted for inflation, this guy was earning on the order of $40k per year.
As new fashions arose, instead of attacking waterfall as practiced (which, don't get me wrong, is still labor intensive), they criticized a straw man version, because it made the new methods look even better.
It works in consulting because you know the end product and schedule before agreeing to do the work (or at least you’re suppose to).
I use to be anti waterfall just because everyone else was but, at the end of the day, it works pretty well in its niche.
I'd rather see more rigor being put in making a methodology that's measurably better than the clusterf of short-term task tracking spreadsheets and cargo cult dogma of agile in practice today.