We always end up with waterfall
amazingcto.com
amazingcto.com
At some level everything is waterfall. If anything is to get done at all the programmer has to make a plan, then put his head down, arse up and implement it.
At another level nothing is waterfall. If the plan is successful it was shipped to the consumer (which may be the programmer himself), and it's very presence changes things for the consumer in ways that they didn't foresee. They then realise they need a new set of changes.
What we call waterfall model is really referring to the scale. If the plan is grand, the specifications for such a grand plan need to be detailed, the implementation long, and the time between putting the head down and evaluating results is large, and we call it waterfall.
But if the strategy is to explore the solution organically, the re-evaluations are frequent, the waterfall periods are short and we call the strategy something else.
So in the end everything uses waterfall. The programmer would not get the long periods of intense focus he need to be productive without it. But also nothing is waterfall, because no plan can foresee everything, it must be continually re-evaluated in the light of unexpected changes it brings as it is implemented.
From https://news.ycombinator.com/item?id=23970156 : by rstuart4133 on July 27, 2020
If you project from 1D to 2D, that linearity will always be there (for completed projects, at least). That extra dimension will introduce additional flavour, and waterfall seems to be a very frequent one.
If the organization requires that everyone is happy, then everyone gets to review the work at every stage (in order to express unhappiness and get it fixed before proceeding). The inevitable result is a Waterfall process to gatekeep how work moves to each stage. The inevitable result is a lack of agility.
But if the organization can lean into the Truth that Nothing Is Perfect, that your product is not perfect, your design is not perfect, your code is not perfect, your scaling, observability, security, and compliance are not perfect, then suddenly, nobody needs anybody's approval. You ship, and you ship quickly. Then you focus on learning and iterative improvement. Make things better over time, but not necessarily right now.
Getting rid of Waterfall is learning to let go and to live with imperfection.
The (very) tricky part, in my experience, is to have these requirements captured in implementation-agnostic way. Even strict accounting reqs require huge amount of work for someone to deconstruct them down to actual compliance requirement, instead of something solution-specific like "we need to store X in component Y".
Not everyone wants to invest into this just for agility sake.
It would be a good architectural decision, actually.
That might work in some fields, but consider life sciences and pharma where some unmet functional/compliance issue could be injurious to the company, a patient, or both.
Waterfall-ish workflows are hard to avoid in such cases. We learned to simply do iterations of waterfall and speed up the validation via automation as much as possible.
Any time an engineering methodology is discussed, people come out of the woodwork with their specific example of one time where that methodology won't work.
After a while, one starts to understand the need for the rigor. If some cluster of patients suffers some injurious adverse effect pre- or post-market, the FDA and a bunch of lawyers wants to trace the history of development to understand how to assign responsibility. Software systems in this space are validated quite rigorously against their functional and technical design specifications to verify that the records produced are traceable.
Software validation in this space works kind of like GPL licenses: the sponsor (like a Merck or a Pfizer) is responsible for validating software from vendors when it is supporting a GxP process. Those vendors are then responsible for validating not only their systems, but all dependent systems as well. (A lot of it is risk-based and obviously a lot of the methodology, guidance, and interpretation has evolved since the cloud era. But we used to have to take our customers on tours of our data center and their auditors would match the specifications of the hardware!)
I think it's one of the reasons why -- despite this process being extremely high in "schlep" -- there are few startups in this space because the barrier to entry is quite high. Even having 15 years of domain experience and running a team delivering production systems used by pharmas to run their trials, I don't think I'd want to do a startup in this space.
There's a really great HN thread on this that resonated with my experience that I bookmarked a while back: https://news.ycombinator.com/item?id=33545209
2. When waterfall delivers the wrong thing because the requirements from two years ago weren't updated to reflect two years of new reality, someone has to pick up the pieces from that, too. It may not be devops - it might be sales instead. But it's still a problem.
Recurring operational issues get fixed mighty quickly when J. Dev has to wake up a few times at 02:00 on Saturday.
Heck, design and architectural issues suddenly get a lot of scrutiny at the whiteboard phase and people decide they don't really need Kafka or Kubernetes or Mesos or GenAI anymore.
It doesn't seem to have changed much as far as the architecture process goes though. With a few exceptions, the things that pop up for support to handle are usually either 'something broke on the content side, and the editors can't fix it', or a third party broke (glares at Google and Tag Manager, Musk and Twitter/X, etc)
Compliance is a set of stories/tickets/requirements like any other, with a priority to be assigned and eventually worked and reworked at some point in the process. There’s nothing wrong with not addressing it as the first thing - it just blocks release. With that, it will eventually get worked on, hopefully at the time where the pieces that need it are understood and the work to reach it is understood.
So those things really do need a proactive gate, or maybe in some cases, at least an extremely fast reactive response capability.
But this just means it is more important to identify the smallest set of considerations like this, in order to give them special treatment, rather than requiring higher scrutiny for everything - if everything is special nothing is special - and to invest in ways for work to be done without implicating those special considerations.
To a degree. But I'm pretty sure one of the reasons why software quality is so much lower these days is because many teams take the idea too far.
I work with a small-a agile team, and the devs who "get it" just make it happen without a waterfall. The ones who don't get it get a super detailed spec to minimize the chances they'll mess it up, and still mess it up. The ones in the middle know when to reach out to ask for an out-of-cycle product clarification before building a useless pile of code.
The main difference is, do they understand why they're building what they're building? Or do they show up, punch through a bunch of tickets, go home? Really hard to fix that with process when that's not working. It starts with good hiring, building a vision together for what we're building, and communicating it repeatedly.
If you have a skilled team with deep domain knowledge, you can afford to not be as detailed in the planning process as the team is most likely already aligned.
If you have a skilled team without deep domain knowledge, you need that sense of unity and alignment, that usually comes through more detailed planning.
It's not one's correct or one's wrong, it just depends on the team composition.
Software development requires some planing and also some flexibility. One decade, companies think software architecture is the most important technical role in the universe, the next decade, the role barely even exists... What we need more of is common sense.
I have. It didn't reach the end.
Our industry is well on the path to ignore the obvious truths for 50 years...
As an improvement, sometimes the project management *forced* the customer to make some checkpoints along the way, but they were a far cry from real iterations.
I'd just add that scrum and waterfall are both more useful in explaining to management and junior engineers approximately how the team/project works, than in actually running the team/project.
Some people do, but they don't get the views or the support.
People are gregarious, they want to be part of a team, middle ground is not a team, unless you create a new name for it. People tend to give support to whoever hates what they hate, not to those who love what they love. So the middle team shouldn't be sold as "the good parts of A and B" but as "we hate A and B".
For example, in tabs vs spaces there's the "tab for indentation, spaces for alignment" bunch, that (we) usually try to sell it like "we use tabs, so you can set the viewing space as you need, and we use spaces so it's neatly aligned" and people don't listen to us.
What if we called it "tabspacing" and defended it like "spaces for indentation are wrong because there are people with visual impairments that need extra space to see things right, and just tabs are wrong, code looks ugly, also if you use space and tabs interchangeably it's wrong, the people with problems will see the code differently than the people without, there will be errors."? Then people might listen to us if they hate one of these things.
So yeah, that's why there are so many ways to do projects: waterfall, agile, scrum, kanban, scrumban, XP, ... .
Bit of a weird mixing of levels there: Everything you mention after "agile" is (more or less) "agile".
The most common defence from Certified Scrum Masters TM © is that you haven’t implemented it correctly/fully.
Does YMMV?
I'm feeling grumpy and decaffinated this morning, so I'll just note that you can't sell books and courses and conferences on "balance and nuance".
The key thing is you can set the parameters so that whatever you're selling happens to fall squarely in the middle. And of course anything can be common sense so long as you can successfully ridicule the alternatives and describe your solution in a way that sounds plausible(ish).
That's why anything that provides a recipe, like scrum, OKRs and so forth, will get sold as a solution for all of our problems. And inevitably lazy people will follow it.
It ALWAYS depends on the context in which these processes are in, but most don't want to spend real time figuring this out. Just adopt what Google, Spotify (squads anyone?), <insert hyped company here> uses.
Surely scrum is too little? Most lowercase-a agile people consider it more of a stepping stone into agile.
And no, AIUI most original-ideals-agilists consider (much or most of) Scrum a stepping stone away from "agile".
We’re biased to overestimate our understanding of problems, so we should lean towards discovery mode and small steps
Dysfunctional agile has detailed specs and little discussion (no discovery, myopic design)
The product is developed using waterfall and then it gets put in front of a stakeholder, who has changed their mind in the time between the initial spec and the delivery. So another, smaller, waterfall is created. This is iterative development. And it always happens, because no-one gets product development right first time.
We'd actually benefit from realising this. Which was the point of Agile - that you can change course during development and so develop the right thing through iteration. But Agile is (by definition) unpredictable. So you end up with this half-arsed solution that pretends to be predictable (except what it predicts is only the first iteration) and pretends to be Agile (except changing course is impossible) as the article says.
What happened then was that after the delivery, the stakeholder was wildly unhappy with the result and demanded re-work, but that would be unplanned and un-budgeted. Many ugly arguments and finger pointing would ensue, which could repeat in several "iterations".
The other thing which doesn't align with agile was that the stakeholder usually didn't participate in the development. They weren't reviewing the progress, they were typically waiting for the finished product to be delivered.
1) You can end at the first iteration and call it a day. The only reason nobody does that is because they don't benefit from an unhappy customer
2) If you have experience, nothing stops you from putting some iterations into the waterfall. As you said the need for feedback rounds should be no surprise.
Traditionally the downside to waterfall was that the distance between those iterations was too big. But smaller iterations and partly prototypes is nothing that wouldn't be allowed in waterfall if you'd like to do it. And if you find a PM that doesn't do the reasonable thing "because the methodology won't allow it" they are probably a bad PM.
Who cares if 6 months ago some important managing director signed off on a useless feature nobody wants anymore: kill it now and move on, we're agile. This is literally the only way I've seen used agile in big orgs: in words only, and it's liberating: we stop spinning endlessly between useless features as the clients move on faster than us, now we can actually skip a beat and go faster, and screw the paper pushers bemoaning that "everything changes". Because yes, everything changes.
Analysis - Requirements - Specification - Design - Implementation - Qualification - Testing - Release.
You literally cannot have good technology without enforcing this workflow and properly completing each step. Its a function of the nature of the way the mind works to solve a problem and encode that solution into a workable technology.
This natural law is resisted by those who do not have experience with it, and who also attempt to make their mark on the universe by either re-inventing this natural law, and slapping their own brand on it, or attempting to ignore key parts they don't like, and in which they are not competent, refactoring it over time only to discover .. as it is a natural law - that they end up re-inventing it, as well. And then slap their brand on it.
But I can say with great confidence after 40 years of developing technology, that the more you resist this natural law, the worse your product will be. All failed technology can be evaluated on the basis of how well this workflow was manifested by those involved.
Scrum is just New-Age Waterfall. Agile is just 'officious scrum'. 10x developers are merely individuals who have stopped resisting this natural law, and in so doing have worked out how to apply it productively in almost everything they do ..
Another factor is that Waterfall is iterative. You can do it in grandiose fashion, and you can do it on a granular, microcosm-level. The more you iterate, the more water flows, and the better things get - but people tend to resist revising themselves. The inclination to check oneself before one wrecks oneself is either bred out, culturally, or reinforced through the gauntlet of fire that is the modern industry. The better you get at iterating over the waterfall, the higher quality your product.
I do agree that the elements of analysis/reqs/spec etc are all vital. What isn't obviously necessary is that they happen in a rigid linear sequence. The point of MVP is to get from implementation back to analysis & requirements.
By iterating through the steps in a non-linear way, you get better results for each step. I'd argue that requirements gathering is the hardest to get right and the part that most frequently fails, probably because workers are not consulted and, even if they are, they don't speak the same language as the analysts.
So I believe that waterfall is still a good methodology, but (in most cases) only if it is done as a directed cyclic graph rather than a strict linear sequence, such that blocks of steps can be repeated.
This breaks estimation, but estimation is broken anyway. It offers that each step is, finally, done right, but recognising that the first attempt at each step is bound to be wrong and must be iterated on.
Does this match your view?
However, there is a subtlety to this:
>So I believe that waterfall is still a good methodology, but (in most cases) only if it is done as a directed cyclic graph rather than a strict linear sequence, such that blocks of steps can be repeated.
Its both! You go through linearly, and if you're lucky, you ship! But if you don't ship, you go back to the prior weak step until it is strong enough to allow you to continue the linear path - and yes, it is always best done linearly from that point, in my opinion. Don't skip forward steps if you've done review and found a weakness, usually in Qualifications, but also very often in weak Specifications or Requirements - which always indicate a failure to properly Analyse. Instead, go back to the prior incomplete/inaccurate step, and redress the issue - and continue forward with the next step from there. No skipping! Skipping makes the whole thing weak again, polluting the stream. Rigorous linearity holds the order.
For this reason, I have added 'gatekeeper' steps to my workflow, which I didn't want to include in my original comment, because I knew someone would then say "but thats not waterfall!" .. Waterfall needs review steps - navel-gazing - imho, which is why I always include:
Analysis - Requirements - Specification - Design - [INTAKE] - Implementation - Qualification - Testing - [REVIEW] - Release.
[INTAKE] is there to protect Developer sanity. If they intake the Design and find it wanting, that is the point where they can kick it back to the team (or internally, personally) to the prior, weak missing step. Same with [REVIEW] which is there more for managers so that they have an opportunity to indicate the weak/missing steps, prior to Release ...
Without these two review steps, Developers and Managers will rip each other to pieces trying to assign blame/responsibility - but with these review steps, there is an opportunity for everyone to get along. Devs can kick things back if they feel things are weak, and nobody gets offended, because at the end of the line, prior to Release, managers can do it too ..
Note that [INTAKE] allows Developers to protect themselves from issues that will negatively impact their ability to Implement, and [REVIEW] gives Managers a means to allow other factors (market conditions, business cases, etc.) to impact Development, and respond appropriately. These tow actions, done iteratively by well-coordinated Developer/Manager roles, produce great technology.
Estimation: the more rigorous you are about the workflow, the better you get at Estimation, as a Developer and as a Manager. Also, the less rigorous you are about the linearity of the workflow, the worse your estimates - because there will be weak/missing steps which hide costs, time and energy.
"Waterfall" in the broader sense is inevitable. What we moved away from was "let's spend 6 months writing up a 300 page specifications doc for this new Operating System that will be handed off to 100 engs" to "a PM and designer can mockup this feature in a few days with some input from the person who will implement it, then the person who will implement it will build it with input from them along the way".
Beyond the IT department everyone else has very to low incentives to adopt "those crazy ideas from computer guys", they care about deadlines, roadmaps, when marketing campaigns can be organised, with what content, what should be on the contracts with business partners,...
Ever since we got those ideas around 2000, the best outcome if agile is adopted at all, is scrumfall as the article describes.
However I have also seen many companies go back to classical requirements documents, and all related processes, as if agile had never been in the building.
God intended water to run from the TOP to the BOTTOM. Don't fight gravity.
The worst results I have ever seen have been projects where the people on the bottom stopped caring because the people on the top tried to make them ignore the laws of nature.
Also: God is dead
But if you need to organize 50+ engineers and actually produce a ... product? That people/agencies/customers expect things of you sure as shit is going to need a definable goal, verifiable output and a somewhat chronological way of achieveing that. Waterfall it is (with reasonable iterations).
Waterfall is a pacifier for people who would rather have lies than uncertainty.
But the often used CER -> PDR -> CDR -> TRR etc. is damn useful.
This is the truth. It's painful to see, but some people would prefer to light millions of dollars on fire than admit they don't know.
No one says Agile can't have full blown requirements or the respective documents.
To the contrary, you can't expect your dev to understand what he has to write without giving him a goal he is working on - Agile won't fix that.
Agile just means you build in small steps from what you have and review your steps ASAP, instead of working multiple months based on some detailed documents, derived from other documents, written by some other department months ago, building stuff no one wanted in the first place.
If you have flexibility about when features can be released then you can do agile.
I wish the agile and scrum evangelists were honest about this.
The most agile company I ever worked for never talked about scrum or agile. We just asked customers what they wanted and integrated it into the product. When features were ready we released them. There was no roadmap or timetable. Sometimes customers would ask us when things will be ready, we always pushed back on giving dates.
I suppose it makes sense given how young SWE is, but I think the fact that cybernetics died and computer science was treated more like mathematics than engineering, so software people rarely if ever practice system thinking.
It’s also striking that a software engineering undergraduate will never take a course on ethics, or social theory, as part of a core engineering curriculum. Whereas within the mechanical and specifically civil engineering, you will absolutely take courses that show you your social impact on your engineering.
That is just never happening in software pedagogy
Only the business world runs on quarterly results and shareholder value and being able to say "yes, we are shipping 10 great new features next quarter, we have a $20 million marketing campaign, new user acquisition campaign, outreach to lapsed users, and PR campaign all ready to go" gets you bonuses and increased stock prices and new investment rounds.
Saying a bunch of Agile gobbledygook gets you...a job as a product manager or lead but you still gotta hit those milestones
Granted, the business world does run on quarterly results and shareholder value and being able to say ..., but you can say any damn thing that you like, only promising it doesn't guarantee that it will happen. Software development is always uncertain, the question is how best to tame that - ignore it or lean into it.
All things being equal, of course the the business world would prefer the most detailed predictable plan. But, they just aren't equal. Software development benefits greatly from short feedback loops. In all phases - from debugging to finding out what the users really need. That is antithetical to big upfront plans.
Moving all software development into some kind of skunkworks part of the business with 0 expected returns and only ever discussing roll outs may be one way to do it. (Shrug)
Its why i think innovation is maybe a bit easier at a privately held or government owned org. They can look at the value more directly rather than through share value impact.
This response is just about how big a deal the shareholder perspectove is, if you can fly under that radar then you stillneed to convince everyone else that you can be trusted to deliver eventually.
I don't disagree with that. In fact, knowing the end goal is even more important when actions are "skunkworks".
> Moving all software development into part of the business with 0 expected returns
Agile's focus on "Working software is the primary measure of progress. ... early and continuous delivery of valuable software." is IMHO the opposite of that characterisation; as it demonstrates value and solicits feedback as soon as feasible. Note that this has very little to do with "is it following a pre-agreed plan or not?".
The issue is more that too many businesses believe that software development is best done as top-down detailed micromanagement, and the tools at hand (i.e. Jira) lean into this delusion.
I just think that anything not top down doesnt play nice with shareholders. Also doesnt play nice with management that want to say they are the ones that got something delivired.
Seeing projects get rejected as they delivier too much of an improvement (e.g. we only predicted 5% savings this is likely to give 40%) really opened my eyes to how important it is to appear like you are more in control than you are.
On the skunkworks suggestion, i was meaning dont put anything from that team into any forecasts until you have a better measure of the impact.that way you controlthe narative better.
Even if you can really do agile (instead of "Agile"), that mismatch can ruin you. In the long run, it probably will, unless you have someone on the agile side able to talk in terms that the other side can understand. The agile side has to actively manage that connection; the business side isn't going to do it for you.
I have seen the best agile environment I have ever seen destroyed by this mismatch. Upper management killed it because it couldn't understand how to measure and predict the agile development process.
I just don’t see it.
But I see bunch of middle managers who think it will affect their bonus if they push little guys to deliver sooner.
The middle managers definitely win more if they ship more.
Waterfall doesn't mean not to iterate. It also doesn't mean to not build prototypes or MVPs. It just pushes the risky decisions back to people that are actually paid to make these decisions. Scrum is not just a framework for micromanaging, it also conveniently removes any domain responsibility from management. It's always "the team decides". (Of course, just on paper. Try to decide to do tech debt only for a couple of sprints with your team and see how management reacts.)
Besides, waterfall is just the straw man of scrum zealots. I once proposed a new team structure where frontend and backend wouldn't be mixed because I observed that most stories didn't have an overlap but where rather mostly one or the other. The scrum master was shocked that I wanted to get rid of "cross functional" teams: "What do you want to do instead, waterfall?". For them it was inconceivable that cross functionality was not equivalent to scrum.
I don't want to be pedantic, but waterfall does mean exactly that by definition (or at least it used to). There are plenty of variations, that include prototypes or MVPs or iterations (e.g. V-Model, Spiral model).
> waterfall is just the straw man of scrum zealots
You can't just change the definition of a word, and then call people using the original definition "zealots", see https://en.wikipedia.org/wiki/Motte-and-bailey_fallacy
When people hear waterfall they think of a pure methodology, when in practise the methodology nearly always will be (and was) sacrificed if the need arises. So as we are discussing a thing we do in practise, we shouldn't discuss it as a theoretical framework in the vacuum of space, but as something that will be (a)bused in reality.
As a film guy I am a big fan of clear up-front planing (you probably know some of the things that go into films, treatments, scripts, story-boards, concept art, ...) A film is very waterfall-y in how it is produced. But good directors maintain their flexibility on set by maintaining a balance in how serious they take the script on set and where other solutions present themselves that are a thousand times better, e.g. because the location/the actors or whatever is more striking than any writer could have imagined etc.
> A film is very waterfall-y in how it is produced. But good directors maintain their flexibility on set by
Right. So they don't stick strictly with waterfall, even though the industry tends to be "waterfall-y".
My understanding is that in some (non-software) industries, waterfall is practiced because you really can't go back and change things. Or maybe you could, but it would mean increased cost and delays to the extent that you might as well have started all over.
There was a time when people thought "software engineering" was really engineering, so you had to follow traditional engineering best practices. eg. Do it right the first time, make plans and stick with them. It just turns out that this doesn't work for software, and as you said, nobody is really stupid enough to do strictly that now.
Hilariously: https://en.wikipedia.org/wiki/Waterfall_model#Royce's_final_...
No.
This article The Myth of the 'Waterfall' SDLC gives you the details (archive version since the site seems to be down) - https://archive.is/jbBta
Technically, that’s exactly what it means. The entire design is signed off and completed before the implementation starts, and there is no feedback loop.
But waterfall itself was a straw man in a paper intended to recommend a much more complex process. It’s more than ironic that it became seen to be a valid process in and of itself.
(Sadly I’ve lost the link to the original paper, but if I find it I’ll update my comment)
> Besides, waterfall is just the straw man of scrum zealots.
The irony is that scrum itself is often just a wrapper for a waterfall process with all the problems it comes with. So many corporate implementations of scrum are AINO - agile in name only. Management that doesn’t understand the complexity of software is doomed to oversimplify it.
That said, if you are working iteratively - one feature at a time, with feedback - then whatever you call it, that’s not waterfall.
I disagree. I've been in many projects which looked exactly like the stereotypical waterfall, with the expected results (delays, death marches).
Unfortunately the strawman was what started getting traction.
The horror is that it got picked up as a legitimate SDLC. Hence your exposure to it.
It’s almost like they saw the pictures and didn’t read the words.
At the same time, that process he described was the standard procedure for government and any large corporation contracts.
I worked as an external employee in an environment that had a very functional scrum implementation. You had groomings, plannings, dailys, retrospectives. Every task was defined, estimated, time on them tracked to get better in estimating. You had scrum masters, capacity plannings. On the surface, it was all very functional and productive.
But the reality is, business analysts and product managers who had little knowledge of how the system works were in charge of talking to clients, writing specifications, getting them approved and ready for development, and very little possibility to iterate if there was a problem in the definition.
It was very efficient at building a bad product. :D
I saw a quote recently from Erik Dietrich that said, "A lot of people mistake activity for productivity” and that describes every scrum implementation I’ve ever seen.
It’s a cargo cult, no doubt in my mind.
https://upload.wikimedia.org/wikipedia/commons/d/de/1970_Roy...
Neither process was explicitly called “waterwall” - and the former is what I was taught at software engineering school.
Now, it IS very heavy and bureaucratic, and 99% of corporations applied it in the worst way possible, with heavy tooling and in strict waterfall fashion (even IBM/Rational consultants).
And I say that as someone who works in Agile, supports it 100%, and has no time for the "it's a scam" whining.
Everything tends to waterfall because the natural human "thing" is to create a process which can be repeated to ensure success / quality / <thing deemed good>. If Agile thing worked, let's package it up into something repeatable and follow that to the letter... Oh, waterfall.
Agile requires constantly working against that, which is hard, because Managers want and love safe and predictable and measurable against something concrete.
Make a framework rather than a process, but that's a fine line.
Agile ultimately needs trust in the team to be able to think on their feet for every scenario coming into their board. I'm my experience this trust either didn't exist, or is treated as a second class citizen to said safe, predicable, measurable, repeatable process.
And I don't believe necessarily that Agile is the be all and end all.
Interesting question. What are the companies doing that have been working remotely already years before the pandemic? How is e.g. Gitlab working? Would love to hear people's experience.
It's not like Agile cannot work with remote collaboration, just that waterfall is more suitable if people have a tendency to isolate and form silos.
The processes you use, or try to use, are limited by the culture of management that work above and around you in other departments. Changing that culture to be accepting of the nature of software development is difficult, particularly in my case where we are interacting with non technically-minded people who are prone to make decisions based on their emotions and/or their own motivations regarding office politics.
The problem with it is it's too complicated. Your management with the mental capacity of a five year old just wouldn't understand it.
https://web.archive.org/web/20230322153353/https://sceweb.uh...
Waterfall is probably the only way of managing large quantities of differently skilled people on a software project and actually getting somewhere though.
True agile is for very small teams of highly competent people only, which are rare.
There are some corrupt organisations out there selling process models which tell you you're getting the latter but actually sell you waterfall again with bits renamed. This drives people away who are sold the agile model at an interview because the people who are invested in it in the organisation don't know either process well enough to rationalise it. So people arrive, realise it's a shit show and leave again quickly. That's reality and it sucks.
I think people need to talk more around the social and political structures of organisations that drive the process model rather than two camps because there's infinite varieties of both ways of doing things.
It's a very recent invention, from the late 20th century, and mostly only "used" in theoretical works until a ridiculous small number of decades ago. It actually being used on practice is very likely the main reason why large projects cost so much nowadays.
It is really not "the only way of managing large number of skilled people".
There is an optimal amount of planning, but knowing it exactly in advance is suboptimal in terms of planning planning cost.
Can't have product owners without actual ownership - without real equity in the product being developed do they really own anything?
If you, on the contrary, invest ridiculous amounts of time in user research and only build things that are validated then waterfall is fine.
Linux project: releases regularly every eight weeks or so. Chrome & Firefox: same thing. In fact most well run larger OSS projects seem to end up using date driven milestones rather than content driven milestones. Inside these projects you of course find people or groups of people that may have their own roadmaps and plans but they keep these separate from the overall plan, which is simply to ship whatever on date x. Either their stuff is ready and it gets merged and released or it stays on a branch until it is ready for a next release.
Branches changed everything for open source developers. Every change is now a mini project. The last step of that project is to merge it, when the project is over. Some branches only last minutes, some last months/years. And those can break down into sub branches/projects. That's actually how the Linux project moves forward. Mostly Linus Torvalds is continuously integrating what is deemed ready.
The mistake many companies make over and over again is managing whole products as one project. It doesn't work. Stop doing that and life gets better. Look at how big companies release big software packages these days. They use calendar driven milestones as well. They have lots of internal projects about things that may or may not end up in there. But the releases mostly go out as planned, i.e. when they are supposed to go out with whatever is in a stable state. The show must go on. That wouldn't work with waterfall. They'd constantly be missing those deadlines. The only reasons for moving deadlines are usually quality related.
I always like this notion of inversion of control. You don't plan what ships, just when you ship and at what level of quality. This is much easier to plan for. Basically it becomes a game of quality control. If it's not ready, don't merge it and it's not on the critical path to anything. The rest just involves looking at the calendar.
Product owners hate this inversion of control because they want to put content on the calendar. A good PO actually understands this and can adapt to that. An inexperienced one will indeed revert to planning lots of silly content milestones in a calendar. And then you are doing waterfall and falling behind the unrealistic schedule. Simple solution: don't do that. A POs job is to prioritize and keep amount of work in progress low to ensure that the right things get done well and as soon as possible. A high amount of work in progress signals slipping deadlines, poorly prioritized work, and lots of people stressing out about everything being super important.
Agile processes tried to fix this. But you still end up with POs in control and sweet talking to senior management. So you end up doing waterfall if they don't know what they are doing (which happens a lot).
Writing this as a CTO who is doubling as the PO (until we can hire someone that is better at that than me, I'm not bored). My approach to this is very simple. I have a hundred things I want to get done. I pick the top two and do those and than we do some more. The hard part is picking the right two things. Not coming up with a hundred more things to do. That requires discipline and a grasp of business and tech. That's my job. A key insight here is that you shouldn't plan out those hundred things in any amount of detail until you are ready to do them. That's waterfall thinking. You are just building an inventory of crap that will never get done and is probably the wrong thing to do by the time you get around to doing those things months/years later. Instead, I set high level mid term goals and plan short term according to those goals. The short term is about what we do right now and making sure that is the best thing to do and then doing that and only that well. We have continuous deployment so I don't care about calendars. Sprints are our short term planning horizon. Day to day work is Kanban driven. Do the most important thing, do nothing else.
“Hey, customer, looks like we burnt through half your budget and it doesn’t look like we’ll be able to complete the full spec on what remains…
I could ask you to increase the budget, but I think I have a better suggestion. You tell me which feature is most important to get in and well focus on that exclusively, and iterate on that until you're either happy with what you’ve got, or the budget runs out. Then lets take a decision on how to proceed from there if still needed”
You can choose to accept this or leave the customer to yell at and sue somebody else.
By engaging with an answer the engineers kept being part of the discussion.
Instead, they should have said: you don’t get to ask that question. That question has no place here. Engage with us in building the product and let us create and enjoy!
Of course, the real world needs deadlines, timelines, roadmaps. And thus, agile is pointless. As it is just waterfall. In a different colour.
That's why we can't have nice things. You depend on having nice project managers, which is rarely the case.
Unfortunately in my career I’ve only ever seen execs recede further and further away, hiring more and more PMs to replace them in that task. The self aggrandisement is getting worse and worse. They go away for retreat camps, and strategy meet-ups etc and pat themselves on the back about doing Very Important Work for weeks after each event
Here it’s a class thing. They see themselves aligned with the capitalist owners more than the “factory workers” (the devs)