Acephalic Agile: Worse than Waterfall?
tech.labs.oliverwyman.com
tech.labs.oliverwyman.com
The same thing goes on with OOP. It started out as a good addition to other programming styles, got turned into a religion ("you can't do this. It's not OOP.") and now into a pariah (FP is next to go through this cycle).
In the end stupid people will mess up anything they touch and good people will adapt things until they work.
I mean... there's no correct application of waterfall. The whole waterfall process comes from a paper as an example of how not to run an engineering project.
It's only for not well understood problems that it fails so completely. In software, in my opinion, fewer problems are well understood at the moment, and so we tend to avoid waterfall.
However, if someone wanted me to set up a webserver to serve static files that don't change? I'm 99% sure I could set up nginx on a virtual machine "fully waterfall" and have no problems.
The same is true for many other real life "waterfall" processes, like buildings, or bridges.
Source: was a commercial interiors contractor before I was a software engineer.
Or if the problem is that well-understood, there's an off-the-shelf solution so good nobody seriously thinks about re-implementing it.
> However, if someone wanted me to set up a webserver to serve static files that don't change? I'm 99% sure I could set up nginx on a virtual machine "fully waterfall" and have no problems.
Exactly. That's barely software development. That's using fully-commoditized components in well-understood fashions to achieve a well-understood goal.
You could script that. I'm sure some places have.
The most well defined piece of software I know of, sel4, had a full machine checkable formal proof up front, and they still ran into issues during implementation that required them to go back and modify the spec.
It's not my fault that it appears you've been using the term wrong. Waterfall with feedback ceases to be waterfall.
Our industry being rather foolish, people read the paper and thought, 'hey, this sounds like a good idea!'
I don't think it really has anything to do with gravity.
That is a very strange thing to say considering that the method comes from "real" branches of engineering (mech, civil, electrical, etc) and you can simply look around you at your built environment to know that it has delivered an awful lot of real, actual stuff. They didn't agile up the building you're sitting in or the roads that lead to it or power station that makes it work or any of the things in it.
Do you know where I'm sitting?
Waterfall as far as software is concerned has its origins in a paper from Winston W. Royce and not an unguided adoption of mech/civil/electrical engineering principles: http://www.cs.umd.edu/class/spring2003/cmsc838p/Process/wate...
one former employer purportedly did agile but it was a top-down autocracy, which is completely incompatible with agile. (i was ostensibly a product owner, but the CEO set our development priorities with only superficial consideration for customer feedback.)
you cannot both dictate and empower at the same time, and empowerment (via agile) is typically the loser in that fight.
that's how i see agile "failing" most often, before it even has a chance.
There is no perfect, but one that is fragile to external business realities is probably worse than one that is resilient to it.
(In my company, the problem with agile was that the sales people would insist on always adding some extra tool for a fixed price. We had no choice but to abandon agile.)
agile seeks to benefit from the intrinsic motivation of each team member to create something beyond any single member’s capabilities in exchange for empowerment.
when a manager doesn’t understand this principle, that leads to failure (as in your case as well). i don’t know of any systems that can self-correct for this kind of external mis-judgement.
What does that have to do with agile? That seems to be a problem that would occur, and be problematic, independent of development methodology.
> We had no choice but to abandon agile.
Seems to me you should have abandoned the offending sales people.
Agile is a culture about project management methodologies, which unfortunately is extremely resistant to actually adopting methodology that reflects it's overt cultural values. Largely, IMO, this is because the agile community has been very bad at operationalizing the “over” statements in the Manifesto into mechanisms by which the right side items are subordinated to the left side items. This results in (for the first statement in the Manifesto) either dysfunctional interaction and demotivated people from constant conflict resulting from the absence of processes and tools, or the adoption processes and tools in a manner which does not respect people and interactions, merely because someone has labelled the tools “Agile”.
I blame scrum for commodotising and cargo-culting all the good stuff out of agile. I'm writing a blog series on the problems with Scrum here: https://www.lambdacambridge.com/blog/2018-05-how-scrum-destr...
Interestingly, DSDM, which the author uses, doesn't get much attention, but was there from the beginning of the Agile movement.
Its never the process at fault, its the gaping disconnect between the specification of that process and the implementation.
Agile involves getting the customer deeply involved from the start, so that after one or two spins of cycle, they have a more sober view of what the magicians are doing, and how little they, the customer, get to ask for in the real world, if they want the project to be done this century (and they do.) Done well, it's the equivalent of sticking your dog's nose into the mess they made on your rug; where dog=client.
He made it quite clear, it's his conclusion:
Now, each iteration offers not only the chance for project management to be exercised, but more specifically, to ensure that it is exercised by capturing the appropriate, timely information from all the appropriate stakeholders.
Emphasis mine.
That’s difficult to codify, especially in contracts for delivery. “Agile methods” more often than not translate into hiding a complete lack of planning in a bunch of mumbo jumbo.
The point is that there are always stakeholders and product owners - it just makes that explicit. If the PO is acting as a proxy for the stakeholder (customer in this case) that's ok.
Of course the PO might be wrong... but with small iterations that should be found out quickly and adjusted for.
https://www.agilealliance.org/agile101/12-principles-behind-...
It would be odd if "business people" just meant the developers' upper management within their firm; supervising costs and parking spaces; but not those without a stake in using the software that results, say.
In fact the only time the customer is mentioned is in principle one: “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.”
Business people absolutely should be involved - product owners & stakeholders (or a proxy for them).
That’s fundamental but again: business people != customers.
You seem to think stakeholders/product owners are devs direct managers too. That’s rarely the case, if ever, on any functioning agile team I’ve seen. They’re orthogonal to each other.
I don't believe I can be of further assistance to you.
You were never any “assistance”. You had nothing of substance to add.
Some smart, well-meaning people write down a couple of brief, broad suggestions on how to do better, and a few years later it's been twisted into an industry of people who want me to pay them for telling me that I'm living my life wrong.
Also, there are larger truths that live at its heart which make discussions about the topic difficult because if you're seen as disagreeing with some of it, you're seen as disagreeing with all of it, even if you don't.
What a good analogy!
I've been meaning to read the original Waterfall article as well, I've heard it's good.
But I believe that if you really want to know how to hit home runs, you want to talk to the king's trainer, not the king. The king has no perspective. He has mostly only worked with himself. To put it bluntly: he's too wound up in his own bullshit to be objective.
Software Developers don't have trainers. Ask a successful person how they got that way and they have a sample size of one, or a handful. All you'll get is speculation.
I liken this to diets. Paleo suggests not eating bread (among other things). However, not eating bread doesn't mean you are doing Paleo. The benefits of Paleo are intended to be realized via a systemic application.
So whether it is agile or diets, make rules that work for you (and yours). Then stick to them. Periodically review if they are working. If they aren't change the part that isn't into a new rule and then repeat. There is no one size fits all in these things.
The reason why Agile techniques were created was due to a serious crisis in software development. Projects were failing, lots of them, in ways and with a severity that few software projects today fail. At its core Agile is very simple, it's a system of hedges against failure:
Always have software that is in some sense in a usable state (builds, runs, passes smoke/BVT tests). This makes it very difficult to complete a project without anything to show for it.
Integrate early and often, don't wait until the last minute. Integrate skeleton frameworks together and then iteratively add features. This helps avoid having a huge amount of unknown risk due to integration failure at the end of a project, mitigating that risk translates to more schedule reliability and higher chances of project success.
Also, working iteratively helps ensure that progress is constantly being made rather than potentially wasting lots of dev-cycles on work that may not be ultimately viable.
Keep the "customers" in the loop. With a version of your software that actually runs they can keep up to speed on the latest features and capabilities as they are actually implemented. This makes it much more difficult to ship something that is wholly different from what the customer actually wants. This isn't a sure-fire recipe to avoid shipping the wrong thing, but it is one of the most reliable, achievable, and cheapest ways of doing so. It's certainly preferable to getting a room of lawyers and engineers to hammer out a contract that embeds a precise specification for the software to be built. Besides which, a lot of software today is not built on contract, it's built for the market.
The cult of Agile is a serious problem, and people who think that any management technique can actually protect them from not doing any management work is an even bigger problem. Agile's core problem is that it can serve as a conduit for micro-management and it can make it hard to sell doing research or expensive (but valuable), long-term investments. But the solution isn't to abandon the concept, the solution is to work better.
You can't make these estimates with any accuracy without the customer and at least some developers talking. Furthermore, customers aren't likely to pick something that the developers have estimated will take a long time, unless it really is that important to spend that much time on it.
Also, large tasks need to be broken down into smaller tasks to get decent estimates, which means knowing roughly what needs to be done. If the job isn't obvious, something quite like traditional planning needs to happen there.
The planning difference is really for smaller changes that don't have far-reaching consequences.
(Given that the Agile Manifesto doesn't say anything directly about tools but entirely lays out a philosophy addressing governance, I'd actually argue that “Agile” tools without Agile governance aren't anything like Agile, and aren't even really Agile tools; Agile tools are tools chosen for a particular tasks through Agile governance, at a level higher than the governance level this price addresses, and applied within an Agile governance process at the level this piece addresses, which is not to endorse the particular governance approaches in this article or it'd horrendous strawmanning of Waterfall, which inasmuch as it is a real model, is an iterative model of exactly the type this piece is advocating.)
I think it's an interesting article, with good points, but fundamentally I have to disagree with this statement. Empirically, it just isn't true, in my experience. It just means that all those changes by the customer to what they wanted, that the devs object to and management (which has to meet the customer face to face) always in the end agrees to, happen at the end, when they are more difficult, because up until that point the project wasn't "real" to the client, and they really couldn't tell you what they wanted until they saw that what you built wasn't it.
Is it some conceit that software development is somehow truly unique compared to the problems and concerns faced by other industries? That's dumbfounding to me if so.
Our industry is barely 70 years old and hardly anybody seems interested in stealing the best ideas and practices from other industries that in some cases have been around for thousands of years.
The requirements are textual. They should be visual like a blueprint.
A real custom home is typically visited and inspected regularly by the future home owner during construction. An improvement for waterfall would be a periodic delivery to the user of intermediate builds of the product – no matter how incomplete.
May I suggest you look into this agile thing?
But it would qualify if you actually delivered partially built product in iterations and incorporate the changes that the client requests as they inspect it, even if you preceded it with extensive blueprinting. Only in that case, it has been found that developing such detailed blueprint is very time consuming and at the same time not very useful. So we don't do it for most kinds of software projects, but not because it's against any agile principles.
The original agile principles (I was just getting into software development when they were articulated) really prohibit very few things, if anything at all. But they do emphasize valuing working, useful software over following strict plan. Use planning to the exact extent that it helps you do create such software.
First, I believe software blueprints should be put together by people with 15 to 30 years of expertise. Secondly, blueprints for a custom home are completely indispensable and are referred to regularly during construction. I believe one of the reasons why is because they are visual, not textual. I believe software requirements should be the same and then they would be easier to adjust and would retain their value.
Custom homes can be adjusted in mid-build. But it is a big deal and you have to get the architect involved and potentially have your plans revetted by inspectors and the city. It's a process that has a high cost. And it should because good blueprints like good software designs are highly dependent internally. I believe every change risks degrading their quality.
If you are in a project where there are large numbers of unknowns about the functionality of the project, I highly recommend shrinking the scope to what you know and actually understand. Build that, and then learn from it. Then come back around and start a new "dave's custom home waterfall" (for lack of a better name) process to implement the next set of functionality.
Constantly course correcting is a recipe for never finishing and weakening quality. There should be natural impediments to course correction and scope reduction should be used until the unknowns have been reduced to a reasonable risk level.
Well, that is for one simply completely impractical, those people are few compared to the amount of software the world needs and also very expensive. But it's only one of many reasons why there are better approaches so let's move on.
> Secondly, blueprints for a custom home are completely indispensable and are referred to regularly during construction. I believe one of the reasons why is because they are visual, not textual. I believe software requirements should be the same and then they would be easier to adjust and would retain their value.
Many detailed requirements have, as matter of fact, been very visual, with detailed user interface mockups, process diagrams etc. When they are large in scope, they have been found not to be useful compared to the effort required to create them. When they are small in scope, they are actually used in real world agile software projects.
And, I have to ask something about this process where extremely experienced people create detailed visual blueprints for the entire scope of the project and then deviate very little from it. Is it something you've actually seen produce good results (repeatedly) for your run-of-the-mill software projects?
> If you are in a project where there are large numbers of unknowns about the functionality of the project
... that would be vast majority of them.
> I highly recommend shrinking the scope to what you know and actually understand. Build that, and then learn from it. Then come back around and start a new "dave's custom home waterfall" (for lack of a better name) process to implement the next set of functionality.
Well, every time you give concrete proposal about what to do in the face of uncertainty it sounds exactly like agile.
> scope reduction should be used until the unknowns have been reduced to a reasonable risk level.
That would, again, be agile. And I say that as someone who is not even particularly fond of of most standardized agile processes, such as scrum.
How many years of expertise do you think architects of large buildings probably have? Engineers of critical bridges and other public infrastructure? One thing we need to do is stop aging out our best talent.
> Many detailed requirements have, as matter of fact, been very visual, with detailed user interface mockups, process diagrams etc. When they are large in scope, they have been found not to be useful compared to the effort required to create them. When they are small in scope, they are actually used in real world agile software projects.
> And, I have to ask something about this process where extremely experienced people create detailed visual blueprints for the entire scope of the project and then deviate very little from it. Is it something you've actually seen produce good results (repeatedly) for your run-of-the-mill software projects?
Absolutely. I have a 100% success rate using these techniques with over 20 under my belt. Granted, I'm mainly a solo operator but I've used it for larger projects too on occasion.
> Well, every time you give concrete proposal about what to do in the face of uncertainty it sounds exactly like agile.
The difference is time frame and phasing. Consider deciding to enlarge a kitchen in the middle of building a custom house. Now compare that to an addition to an existing home. They are very different situations.
> That would, again, be agile. And I say that as someone who is not even particularly fond of of most standardized agile processes, such as scrum.
The difference again is the time frame and the phasing. Messing with the scope/design every week I think is an anti-pattern and a contributor to project failure.
Look at a building shell, even if you have zero clue about houses you can see if they used rotten wood. It looks like it will become a house. The more it progresses.. well, you can easily tell "that door shouldn't be where it is" and so on.
Now look at the code generated by $framework via "$tool new project $name". To a non-coder (or even just not using that language) it's not really discernible from a 90% finished product. Your house doesn't turn invisible because looked at it from the back.
Also house construction follows certain "easy" patterns a 5 year old can grasp - you need to start at the bottom, you can't put the paint on before the walls are in, etc.pp - give a coder of 10 years a project in another language without documentation and you won't hear anything in the direction of "it's more 10% or 80%" unless spending days. I'm not saying people building houses have an easier job - but it's palpable.
To the second concern, first, let’s get on the same page. When I use the home analogy it’s in regards to a custom home - not a from-plan/spec home. To your point, I believe well developed software should also have strong and obvious patterns that are reused in many if not all other projects.
Some are: http://alistair.cockburn.us/Characterizing+people+as+non-lin...
From the abstract: We methodologists and process designers have been designing complex systems without characterizing the active components of our systems, known to be highly non-linear and variable (people). This paper outlines theories and projects I reviewed on the way to making this stupendously obvious but notable discovery and four characteristics of people that most affect methodology design and project outcome. I find these characteristics of people to be better predictors of project behavior and methodology success than other methodological factors.
They are and have been. For one example, much of the Lean Software Development literature leans very heavily on that.
Why the things that are less systematic and empirical are more popular is a valid question, though.
Congratulations, your lead developer is now a middle manager. I've never been a fan of "committing to a certain amount of stuff done by the end of the week" as in scrum sprints. It's a zero sum game. Sometimes you do 3 days' worth of work in an afternoon and sometimes finding an extra semicolon takes 2 days...
What a beautifully simple document, written in clear and plain language.
Now consider a paragraph like this one:
"How are matters improved by Agile? The benefit of Agile is that, by splitting the project into sprints, we have more opportunities to tweak the project vector, to exercise governance, altering the project’s course. Each sprint offers new control points (during both planning and acceptance), at which we can amend direction. Splitting the project as a whole into a series of such sprints offers us a corresponding number of opportunities to reorientate. Each iteration offers more chances to retune the project. We have much greater granularity of control, and more opportunities to exercise that control."
This blog post is, frankly, full of this sort of language. "tweak the project vector". "exercise governance". "control points to amend direction". opportunities to "reorientate" (reorientate? yes, it is a word. it means, according to the OED, reorient).
I'm kind of curious, this is HN, there are dev here. Does this sort of language make you eager to sign onto a development team?
I understand that lots of things have gone wrong with "agile", but the language we use to talk about it seems to have degraded badly, and in many ways, this tells the story better than almost anything else.
I gather from the article's initial paragraphs that these "Governance" point is really the team getting input from all the normal stakeholders and creating the backlog of work themselves? This makes all the sense in the world - getting technical folks involved in discussions is key.
What I would like to see added are concrete steps / workflows that these technical teams have found that help make these governance points effective without eating up a lot of time, and what practices they used to ensure that the project goal / direction / outcomes were decided upon up front.
I personally think Agile Roadmapping / Storymapping with the whole group can go a long way toward ensuring whatever you build is going in the right direction. From there, you'll want the team to work with the product owner folks and keep a tighter grip on the backlog toward that direction and good things can happen.
This doesn't make any sense and is contradictory.
It seems you admit that Agile works for "Rolling out bug fixes and features" and to "deliver code incrementally" without "coming to great harm".
Then you say that this is somehow different from "an entire solution [that has to be] designed and delivered".
What precludes "an entire solution" from being developed in an Agile fashion?
Presumably, agile would be to build out the skeleton, and incrementally add small working features over time in increments such that at the end, you had working software that was what you wanted.
You'd put your skeleton it into beta use as soon as possible once it provides any value, and get frequent feedback to iterate.
This article scrapes the surface: https://michaelochurch.wordpress.com/2015/06/06/why-agile-an...
> "The worst thing about estimates is that they push a company in the direction of doing work that’s estimable. This causes programmers to favor the low-yield, easy stuff that the business doesn’t actually need but that is 'safe'".
> "Good engineers want to work in engineer-driven firms where they will be calling shots regarding what gets worked on, without having to justify themselves to 'scrum masters' and 'product owners' and layers of non-technical management"
> "Scrum [is] tailored toward the body shops where client relationships are so mismanaged"
> "Open-plan offices are the most egregious example. They aren’t productive"
> "[Scrum is] Like a failed communist state that equalizes by spreading poverty"
There is more. In "pure scrum" it's not obvious how to account for refactoring, developer experimentation, improving infrastructure (or creating one from scratch!), making tests better. Product driven things - sure, can be driven by scrum. But what to do when work is driven by engineering?
The kind of work that good engineers thrive on - a hard problem starting with en empty Emacs buffer - is something that the methodology merchants have never experienced themselves. The only product they ever shipped is the methodology itself!
I say this as a one-time “certified scrum master”.
The article makes a very reasonable argument, however I wish he wouldn't try to introduce some new term, Agile "governance". He's just describing what we already know -- that Agile isn't Agile and care must be taken to institute it correctly. via "governance" as he calls it but this isn't a new revelation.