“Waterfall” doesn't mean what you think it means

I led an engineering team building validated systems that supported GxP processes in life sciences and found that a Shape Up-like process produced the best balance between delivering software that could be validated[1] and speed. I only came across it after several years of trial and error, but I was amazed at how it almost exactly described what we ended up doing.

A side effect is that work-life balance is amazing once you get this process working. If you know exactly how much work you need to do and when it needs to get done by for the next 4-6 weeks, then you can pace yourself much better. When you plan 6 weeks at a time, someone going out sick or taking some time off has less of an impact than when you plan 1 or 2 weeks at a time.

The customers and end-users know what to expect and when they'll get it; everyone gets more predictability in the process. Yes, it takes 1-2 weeks of planning at the outset, but we used that downtime for the engineering team to catch up on tech, build some experiments, work on side projects, or do research. (No refactoring since the validated process requires us to plan and document all code changes that make it into production)

I recommend it for any team that wants some sanity in the processes.

[0] https://basecamp.com/shapeup

[1] Validated in this context means that every feature, function, and infrastructure component can be traced back to a specification through the full lifecycle from design to install to post-install verification and operations.

I assume the article’s reference to SAGE is referring to the system that routed early warning radar station data to NORAD. https://en.m.wikipedia.org/wiki/SAGE_radar_stations

I worked at a software company that built extremely sensitive survey applications (iOS, Android, and web-based). A significant part of my job entailed breaking down a literal design specification document into Jira tickets, which were then worked on by a rotating staff of programmers. The rotating programmers were assigned to 10-30 apps at one time and often had to make enormous customizations to the base product. The base product was formed out of 15 separate sub-products whose teams did not correspond with each other. The clients were allowed to make broad changes to the entire system by (a) changing the design document (b) emailing furious demands to the sales team/my manager (c) adding requirements to the master contract and not telling us. There was no traceability established between the contract/design doc/tickets, and I was not allowed to standardize any of those documents. There was no standardized format for writing the Jira tickets, so every person on my team made up their own method. The design document was missing about 40% of the context necessary to deliver a working app, so I added that context into the Jira tickets, but the clients weren't allowed to review the Jira tickets. I spent 20% of my time in "Scrum" meetings including sprint planning, sprint reflection, and hour-long standups. My company was worth hundreds of millions of dollars. I made a fat chunk of change in a year and a half, put a down payment on a house, and quit.

I will never forget sitting in a 4-8 hour meeting for a day reviewing our entire dependency tree, and whether or not SJ-BCOMKA-4.11 had a dependency on SJ-TARGET-4.32. Management moved our "productive percentage time" from 80% to 85% to make the schedule look good to upper management. The scars of "waterfall" run very very deep.

Second, it's clear you need to reapply anything written 35+ years ago to modern sw development. For example, "operations" as a stage of sw development, whereas today we have devops vs SRE vs production engineering vs whatever.

Things like cleanroom development, zero defect software, etc are completely under utilized by software shops today, because we price the cost of maintenance & bugs far far too low. Which is was also unfortunately missing from the waterfall paper compared to Capers Jones or others analysis that showed it's not just that you should have preceding steps to "coding" - the preceding steps (aka "analysis") remove defects from your final product far far faster.

There were about 50 pages of specs for it: descriptions, wireframes for the UI, how everything should fit together and what the typical workflows with it would be. All of this was available ahead of time, before writing code even started, which actually let me get a better idea about the expectations and needs, than focusing more on immediate needs in a more iterative approach. I could also discuss any uncertain items and get some other questions answered, to avoid confusion later down the line.

It helped me write the code better and anticipate how to structure things, so less refactoring needed to be done. I could also focus on the "happy path" first and foremost, getting the critical functionality in place and then adding the nice to haves around it. Admittedly, there were problems with the specs, which I brought up and promptly solved when it became relevant, so the final solution differed slightly, but by then the specs were also updated.

With the critical stuff done early, it was also possible to release the thing early on and then switch to a more iterative method of development, with improvements and all. Not quite the traditional waterfall approach, not exactly scrum either, but something that worked nicely regardless. Complete specs (even an overview of the big picture) can definitely be a nice to have in some cases, if doing that additional planning at the start isn't too much of a burden. Take whatever works from all of the different methodologies out there.

This is mostly fine for people like UX, product or up-front architect types who don't have to deliver the software or deal with crunch time at end, but everything depends on what they spec out.

So, sorry, not sorry, I greatly prefer agile/incremental work to that.

Similarly, arguing over what "waterfall" really means is just so much flogging of a dead horse.

Personally, the idea of floating down a waterfall sounds way more appealing.

The customer wants a specific custom feature and once they have it it should be done and over with the project. They want to know the cost and they want to know the delivery date (the actual person I am talking with is a sales rep who is talking to the real customer, so they usually want to know those things in reverse order, but that’s manageable).

The well-known problem: The customer doesn’t know what they really want until they see a first iteration in practice. The solution: train the sales rep (by courses and experience) to join you in becoming an awesome author team for req and spec documents. Keep the scope small enough to be able to oversee all cross-effects of the new features. Keep the discussion alive with the customer to talk about these effects.

Will this always work? No, I had absolutely insane projects out of everyone’s control. But it will work most of the time and if it did not the cake was just too big to swallow in one piece. Bottom line: If waterfall fits or fits not depends in my experience on the scale of the project.

Ms. Pahlka crawls through some large-system horror stories, and some success stories. The horror stories share something in common: design documentation that outlives its usefulness, takes on a life of its own, and drives implementation decisions in harmful directions.

Brandow's article (the subject of this thread) emphasizes the usefulness of documentation. And it's certain that long-lived software systems (like his php example) need excellent documentation. That documentation is targeted at people like many of us. It answers questions like, "how do I use that blankety-blank IMAP module?".

But long-lived documentation is often misunderstood and misused. For an overstated example, some "non-technical" administrator might read the php docs and somehow infer that "systems developed in php MUST use the IMAP module, or it won't pass acceptance testing." Then, the developers -- employees of some contract development team -- will comply. They'll have to figure out some way to incorporate email store-and-forward systems into the product, even if it makes no sense to do that.

There's a danger inherent in extensive documentation: it WILL be misunderstood in the future, and those misunderstandings may turn out to be costly. More systematic documentation -- for example boilerplate "scope" sections at the beginning of docs -- are not a great way to mitigate that danger.

Seriously, go read Pahlka's book.

Is there anything beyond: "Don't call waterfall bad because what we call waterfall is actually iterative and iterative is good" ?

https://uidrafter.com is a free tool that domain-experts can use for prototyping without graphic design concerns.

But the truth is the people who advocate waterfall in this day and age, either never have never actually done it or have some rose-tinted nostalgia.

Crunching for weeks on end to hit a deadline sucks. Especially when some manager beats you up for not following a specification that makes no sense.

So we're ignoring this, or what?

But otherwise, yes, I've long maintained that waterfall is actually iterative done badly. Nice to see it recognised.

Seems to be "Sana Sans" (but probably more accurately "Sana Sans Alt"?)

Bookmarking this to study it more.

For one reason, because that's how language works. But on a more impactful way, because what people think about it is exactly what was widely practiced at the time the paper was posted.

Corporations did hire some requirements gathering consultant. That consultant would make a lot of documents, sign them down and go to another client. Then the corporation would do an auction and get a "supplier" based on the requirements docs, make a contract and sign down that the docs completely described what they expected and was exactly what should be delivered. Then an administrator would test the received software against the contract, and shove it in a shelf where the software wouldn't get in anybody's way, because nobody would want that thing.

This is exactly how large corporations developed software.

Of course, the researcher that coined the name was so out of touch with the industry that he thought he was creating a strawmen. That makes his description not very useful, even though it doesn't detract anything from the paper's quality.