AgileFall – When Waterfall Sneaks Back into Agile
steveblank.com
steveblank.com
However, I still think waterfall is the only approach when trying to design a hardware product. Electronics require pretty specific requirements up front before you design and order boards, obviously mechanical tooling can be even more expensive.
But... I think there are lots of ways to do quick experiments and tighten up the feedback look even with robotics or electronics. Breadboards and development kits can be used as initial electronics prototypes. 3d prints help test a mechanical concept. Within each phase of the design more experiments can be done. But ultimately the project is probably more waterfall than agile!
Modern software projects seem to just start with people typing and little upfront documentation. A long time ago there was documentation (flow charts, Booch diagrams). Now whether these are useful seems debatable, but I do not see any upfront analysis done with software.
> Engineering systems requires an engineering process and discipline, and agile is just not up to the rigor.
Agile just means closing as many feedback feedback loops as you can as early as you can, and to preserve opportunities to course-correct for as long as you can. All this to keep the destructive effects of surprises (which are sure to catch you) as low as possible.
This is good engineering practice, plain and simple.
Maybe a bad example, but it's basically mandatory to include a manpage for a cli utility, but do you always include drag & drop for a file uploader?
1 - Is it a domain where "user story" means something? 2 - Is it a domain where you can iterate fast?
Usually for UI these two conditions are met. From my experience, when talking about big infrastructure projects, they aren't
All else equal, smaller or shorter feedback loops are better. And often shortening a feedback loop is worth considerable tradeoffs elsewhere.
It's certainly the case that being able to iterate fast may not be currently possible, but being able to do so, or even just being able to iterate faster should probably always be a priority (even if not the top one).
This all applies even to physical real-world infrastructure. I remember a story about a city somewhere (South America?) where the mayor (?) was able to shorten the feedback loop whereby roads were built or repaired. That seems like a good, even great, thing, even at somewhat considerable cost.
What kind of big infrastructure projects were you thinking of specifically?
We have an API that is used by external systems, those systems (and developers) are our "users".
It's vital to think of API consumers as users, because it avoids the "REST is CRUD" attitude by focussing on the business/system functions that are being expressed, which leads to focussing on the Nouns, not the verbs of the business.
The Agile Manifesto was largely a reaction against the blind application of process developed for civil engineering to software development, and the features you point to are real reasons that civil engineering process is a bad fit for most software development.
But the Manifesto doesn't specify process, it proposed priorities and objectives that should be considered for process that (changing only the words “working software” to whatever it is you want to produce) are fairly broadly applicable.
Of course, most “Agile” shops aren't Agile at all, they are improved over what the Manifesto reacted against only in the small way that the process that they blindly impose without applying the considerations of the Manifesto is a Frankenstein’s monster combination of things other people designed for some other team (probably not all the same team) working on some other problems (probably not the same problems) in some other part (probably not all the same part) of the software industry.
I disagree; I think those priorities are appropriate because of the unique situation of software development. Prioritising individuals and interactions over processes and tools makes sense because it is easy to adapt and modify software tools, and because software allows abstracting out any form of repetition without needing to make it a process. Prioritising working software over documentation makes sense because source code is understandable in itself to a much greater extent than physical artifacts. Etc.
No, it makes sense because the processes and tools don't work if they don't work with the individuals and their interactions, and that's true in much wider domain than software development (it's true of any highly skilled, or even more generally labor-supply-constrained, work, where you can't easily replace the workers with ones that fit the preferred process; it may be applicable even more broadly, since sense of ownership of process can be a morale and productivity booster), so the process and tools must be subordinated to the team rather than vice versa.
> Prioritising working software over documentation makes sense because source code is understandable in itself to a much greater extent than physical artifacts.
No, prioritizing the production objective over supporting artifacts makes sense because of the fundamental nature of an objective. That's just as true when the production objective is not software (doing so will result in different levels of documentation and other supporting artifacts for different production objectives, sure, and software may often have lower documentation needs than many other tasks, but the prioritization isn't about do less documentation, it's about “do only, but still.all of, the documentation that has utility outweighing cost in support of the production objective of working software”.)
This is backwards. Software enshrines the repetitive business process and (hopefully) automates it as much as possible. This is happening regardless of whether that business process was defined before the software was implemented.
You can still use some agile ideas when working in electronics, thought it might look quite differently than in software.
For example, if you are a startup and your plan is to create a super duper wireless speaker with bluetooth, wifi and AI, you could first build a wireless speaker with only bluetooth, and sell that. That way you build experience with producing speakers, sales channels, manufacturing, sourcing etc., and build a customer base.
Then you could ask your customer base if they'd prefer wifi or AI in the next model, and iterate accordingly.
This is not the same as doing one-week sprints in software, but it does iterate that you can start small, build and sell small, viable products, and work your way up to the intended product.
These quick loops are really the only difference, and are precisely where the advantage lies.
And having as short feedback loops as you can get away with has been sound engineering practice since before software was a thing.
The difference now is more bureaucracy under agile.
Building the actual bridge on which people will expect to drive or walk or whatever is probably better thought of as deploying a bridge (design), even if it inevitably also includes some ad-hoc on-the-spot development too.
But when developing a new bridge design, it seems extremely valuable to be able to iterate as quickly as possible.
They did do it. The MAX part of the 737 MAX was basically an agile iteration on top of the existing design.
>But when developing a new bridge design, it seems extremely valuable to be able to iterate as quickly as possible.
Seems like a new and innovative way to design things. Bridge 2.0. Bridge MAX.
You'd obviously want a new, experimental, or prototypical design to fail ASAP so you can fix whatever the problem(s) are sooner than later.
I'm pretty sure most bridge or airplane failures are either due to mistakes made during building or, probably even more often, failing to maintain (and monitor) its condition.
But the whole point of agile is to make, and release, smaller sets of changes at a time precisely so those changes can then be tested or validated more quickly than otherwise. That may – probably is – more difficult with airplanes and bridges versus software so the relevant tradeoffs are different, but the basic point about minimizing the duration of making changes so those changes can be tested sooner seems equally applicable.
Let's think about bridges. I'm sure most bridges use a standard design with standard materials – that part then has already been tested and validated. But even at the stage of evaluating a standard bridge design and whether and how it could be built in a specific location, I'd imagine an 'agile' methodology would be better than a 'waterfall' one in terms of testing or validating a tentative plan, i.e. you'd want to address the highest priority risks first, and 'fail fast'. You wouldn't want to create a final proposal if you knew up-front that there was no way to anchor a bridge of a certain design in the designated location.
Every phase of the project has iterations, but iterations stress different things (requirements in the beginning, construction and handover in the latter phases).
Now, most people who say they do Waterfall are quite mistaken. They don't, they do some sort of iterative method as you say. But they report as if they have distinct, linear phases with distinct linear steps, as if Waterfall were the reality and not an imagining. Which is itself problematic. This means that upper managemnet comes to believe that Waterfall works because they see it working, not realizing that the reports themselves are misleading.
Everytime one of the project managers tells me they're doing Waterfall I press until they admit it's "Modified Waterfall", by which they mean the V-model or Spiral or other models with explicit feedback loops/mechanisms. And it frustrates me to no end that they keep using the wrong terms rather than the precise ones, because now upper management has begun enshrining Waterfall again after two decades of improvements. Now we have to fight the same battles, to impress upon them that iterative methods are what we actually do, and that a strict linear flow is impossible.
It feels like the Bruce Lee idea of "take what is useful and leave the rest" but either the opposite or a bastardized version. Like, take what is easy and leave the nuance
I see, whatever this idea/principle would be called, all over.
Then like evolution, it's survival of the fittest with the entrenched old idea having the heavyweight advantage.
BTW, this is a great point you make! I hope my comment doesn't seem to discount it as your thought made my mind go to the experiences I have seen that led to this comment.
> Cargo cult programming can also refer to the practice of applying a design pattern or coding style blindly without understanding the reasons behind that design principle.
> Cargo cult programming is typically symptomatic of a programmer not understanding either a bug they were attempting to solve or the apparent solution
> The term cargo cult programmer may apply when an unskilled or novice computer programmer (or one inexperienced with the problem at hand) copies some program code from one place to another with little or no understanding of how it works or whether it is required in its new position.
Why does Kanban help in some offices and do nothing or hurt in others?
Two reasons for success: Luck or understanding. Two reasons for failure: lack of understanding or lack of commitment.
What does Kanban assist with, what principles?
It assists in elaborating and clarifying the workflow or processes in the organization or unit. This can be done many ways, Kanban is just one. But without understanding the, often unwritten, processes you cannot effect meaningful change (at best you're stabbing in the dark).
It also emphasizes focus by setting limits on "work in progress" which improves overall flow/throughput. It turns out that this is, generally, a good idea.
The ones who succeed don't succeed because of Kanban. They succeed because of those two basic principles (and other related principles and concepts that I'm not elaborating on here). Kanban was a mechanism which led to success by drawing on these ideas.
The ones who fail don't fail because of Kanban. They fail because they don't control WIP, or failed to describe their process accurately, so it provided no, or perhaps even negative, value. Why don't they control WIP? Maybe they aren't aware of that element of Kanban, or they don't believe that it can bring value and ignore it. They're too busy fighting fires so they don't have time to focus. Why don't they have an accurate process description? Often, because the process is dictated rather than discovered (You are doing X, even though you actually do A, B, and C).
That's just one practice and two principles that it is related to (there are others). This is the problem with Big-M Methodologies like Big-S Scrum and Big-W Waterfall and Big-A Agile, the easy part is the visible practices. So they're imitated. The hard part is the invisible principles and hidden practices (all the work that went into identifying the workflow, for instance). Repeat this across all the practices in these methodologies and you'll end up with a mess that succeeds by luck, or at least doesn't harm you so badly that you fail, but certainly doesn't offer significant assistance.
There, fixed it for you.
News for you: Not all software has the tendency to become a giant mess.
Without discipline, I'd agree with beat - any process will produce code that becomes a giant mess.
With quick-turn fab houses, you can do two-week sprints in hardware. You can fix up the circuit with X-acto/bluewire changes, and roll the fixes into the next sprint. Or you can make individual subsystems per sprint.
And with firmware, most software methodologies work with minor changes.
Usually you can get something working in the first sprint, and iterate from there.
For a simple project all of that can be managed iteratively.
Not everything has to be changed, that's just a strawman.
But not every detail has to be nailed down at the beginning either.
The idea behind Agile is to reduce risk. If there is no risk of the bus system changing on you, there's no harm in fixing it right at the beginning.
But if there is a chance it might, you'd do well to have systems in place which enable you to react to changes with a minimum (though non-zero, of course) amount of pain.
It's impossible to foresee those all in advance—better to adopt a system that is structured to anticipate them. "Unknown unknowns" are a fact of life, no matter how disciplined we believe we are with our preparation.
"Ah, you have hardware version 2.3.1.3 - yes, it does sometimes emit sparks - we fixed that in the version that came out the month after."
Physical safety risks are an exceptional class of defects that in most jurisdictions has a legal requirement for the vendor to fix or replace the defective part(s) at their own cost (not to be borne by consumer).
Point: SPECTRE/MELTDOWN were not "physical safety risks". Yes you can argue they are security risks, but they didn't physically manifest as a physical safety risk which can lead to severe injury or loss of life.
Your commentary seemed to suggest you were not attaching an appropriate level of mental priority with regards to the severity and risk.
I was once in a meeting with some hardware people who were looking for help with the UX and complementary app for their product. They had already locked the hardware design, but had yet to decided what the buttons would actually do. Essentially, they were looking for help to come up with features for the buttons.
Thankfully I didn’t get the job.
This is unusual to my taste, but thinking about it, I half expect cheap appliance and toy vendors to work like this - decide on a microchip, and only then figure out what to do with it and how.
I disagree, having taken part in agile embedded development.
Sure, it looked a bit different from pure-software, but the same underlying principle of short iterations applied. Our definition of "short" was a bit different, but still...
And yes, you may have some "very specific" requirements, but you may also have some "pretty loose" ones -- just like software.
> But... I think there are lots of ways to do quick experiments and tighten up the feedback look even with robotics or electronics. Breadboards and development kits can be used as initial electronics prototypes. 3d prints help test a mechanical concept
See, there you go.
The core idea behind Agile isn't to have sacrosanct two week sprints (or even to have sprints in the first place), it is to close as many feedback loops as you can as quickly as you can. Whatever that means in practice.
This has been good engineering practice for longer than software exists.
I read that the Mercury space project created their software in half-day "sprints". Back in the '60s, on punch-card machines (I guess).
No, not at all. Just maybe a little... narrow?
> observations from testing or using is fed back quite quickly so that fixes can be included in the next sprint.
Quite quickly, yes.
The next sprint...maybe? Not necessarily.
Sure, you'll stick it in the backlog. But whether you'll start working on this improvement immediately is a separate matter. If that's the best course of action, sure, do it. But maybe you have more pressing matters, and the present implementation will do for now. Or... it's not possible to attack it in the next sprint, because a new board needs to be spun, and that takes preparation. Then it'll be slated for the appropriate time, by necessity. That doesn't mean you forego an agile approach -- it just looks different, because the landscape is different.
> Especially true when the project gets closer to start of mass production.
Sure. Which is why being able to make, and find, and correct your mistakes early, when it's not so dramatic yet, is the goal of that whole Agile song and dance.
As you're intimately familiar of course, surprises in engineering are usually the bad kind, and the later they come, the worse they catch you. So, have as much feedback as you can, as early as you can (and with as much fidelity as you can).
I don't think a single compile cycle per half day meets most people's definition of a sprint...
Hm... I read this quite differently than you do. Not compile cycles, but actual test-driven product iterations.
Do you have more solid sources than I do?
https://intenseminimalism.com/2012/a-brief-history-of-agile-...
1) "A brief history of agile methods" asserts:
1958 — Project Mercury (NASA) software development, ran with half-day iterations.
The only thing that looks like it could be intended as evidence for that assertion is the Larman and Basili article.
2) "Rockets before Rovers" asserts:
Project Mercury did half-day iterations with test first development!
The only thing that looks like it could be intended as evidence for that assertion is the Larman and Basili article.
3) So there only seems to be one source — Larman and Basili "Iterative and Incremental Development: A Brief History" — which asserts:
"Project Mercury ran with very short (half-day) iterations that were time boxed." page 2
[pdf] http://www.craiglarman.com/wiki/downloads/misc/history-of-it...
Where's the evidence for that assertion?
afaict the Larman and Basili article does not provide a source as evidence for that assertion?
So we're left with nothing.
https://hbr.org/2016/04/the-secret-history-of-agile-innovati...
On a different note, it is also interesting that stand-up meetings seem to be popular in many companies again, after going through a period where snippets tried to fill role of stand up meetings.
http://teche.ltc.mq.edu.au/human-science/getting-s-done-goog...
Teams only checked in – wait for it – every three months in a formal schedule review. I listened as Henrich mentioned that the teams complained about the volume of paperwork he makes them fill out for these quarterly reviews. And he was unhappy with the quality of the reports because he felt most teams wrote the reports the night before the review.
That's not a problem with AgileFall or ScrumFall, that's ineffective management irrespective of how the sausage is made. But nobody has to wear the Scarlet Letter of Project Failure if we assign blame to the process!
I've also never found that pure (like the wind-driven snow capital-A) Agile matches reality. After spending more than a decade wrestling with it - I still haven't figured out how quality assurance and test fits into the same iteration in Agile. Most teams I inherit have non-technical QA resources. Most projects I inherit have complex business logic and integration needs that aren't easily testable until some state of development "done" has been achieved. And time and time again I find myself staggering their effort slightly behind development so that test and development resources aren't taking turns sitting on their asses waiting for the other side.
I've written about this before as was even been publicly admonished by an Agile Manifesto signatory (hallowed be their names) that I was doing it WRONG WRONG WRONG! But I was never able to figure out how to contort reality to satisfy their criticism. Believing in iterations and backlogs and prioritization and shortened feedback loops wasn't enough. Thinking QA - any QA! - occurred AFTER any development was done was altogether impure. And despite asking repeatedly for concrete examples of how its done the "right" way, the only responses I received were vague platitudes and consultant sales blurbs that I guess I wasn't smart enough to understand.
In the end, I don't think there's any escaping that work has to be done in a sequence: identify a need, identify a solution, implement the solution, validate the solution. And so every software project has at least a little bit of waterfall mixed in.
Oh, and it has more-frequent check-ins/iterations than Waterfall (which is the defining aspect of Waterfall: fewer check-ins/iterations than Agile). The actual frequency is undefined, it exists only in comparison to Waterfall.
---
The problem is that that kind of stance does nothing but fluff up agile consultant's resumes. It doesn't actually lead to improvements that last, or even discussion that might lead to improvements. Agile is perfect because it's Agile.
Thankfully the agile trend seems to be left out of interview questions these days.
"Do you do Agile?"
- "What do you define as Agile?"
-"Does anyone NOT do agile?"
-"Do you think it actually makes a difference?"
would be the responses that cam to mind. Apparently I didn't get a job based on a variation of the last one. They thought I would be difficult to work with.
If it's not working, but you keep trying to do it the same way (whether because management tells you to or because a book or consultant does), you are doing it wrong. Because Agile is very little except the team continuously adapting process based on observation of what does and doesn't work for them.
(Waterfall can be Agile, but only if it is waterfall adopted because the team views it as the best way to deliver working software in their particular circumstance and is continuously open to reevaluating that conclusion based on evidence. Agile isn't opposed to waterfall but to the way Waterfall was imposed. Which is also the way Scrum and variants are now frequently imposed, producing the same kinds of problems of process/problem fit.)
Sterman's book Business Dynamics has a great diagram that I think is relevant (I saw it there, but it's probably in other books on system dynamics and related fields). I'll attempt to ascii art it here:
Mental model -> Strategy/Structure/Decision Rules
|
v
Decisions
^ \
/ \
/ v
Information <----- Real World
Feedback
A learning organization, continuous improvement, agile, whatever you want adds an additional loop. The information feedback gets pushed back into the mental model:
Mental model -> Strategy/Structure/Decision Rules
^ |
| v
| Decisions
| ^ \
| / \
| / v
Information <------ Real World
Feedback
At regular intervals, the team reflects on how
to become more effective, then tunes and adjusts
its behavior accordingly.
I observed one of the Manifesto signatories for the first time last night. Superb presentation skills, I'll give them that.
I'm more of a Taoist than a Confucian (and a sysadmin to boot) in this sense: The system which can be defined is not the true system.
> The system which can be defined is not the true system.
What do you mean by this?
As far as I am concerned, everything you need to know about agile is on this page: https://agilemanifesto.org/
So looking at this:
> After spending more than a decade wrestling with it - I still haven't figured out how quality assurance and test fits into the same iteration in Agile. Most teams I inherit have non-technical QA resources. Most projects I inherit have complex business logic and integration needs that aren't easily testable until some state of development "done" has been achieved. And time and time again I find myself staggering their effort slightly behind development so that test and development resources aren't taking turns sitting on their asses waiting for the other side.
That sounds like you're following "Individuals and interactions over processes and tools" and "Responding to change over following a plan" quite well. The "QA while you develop" process wasn't working for the skillset of the individuals on your team, and so you prioritized the individuals over the process, and responded to the change of understanding instead of following the plan you now knew was bad.
So, public admonishment of Agile Manifesto signatories be damned, you were doing agile, and anyone (including Agile Manifesto signatories) who tells you otherwise can be told to go read the Agile Manifesto as many times as they need to in order to reach enlightenment.
The problem is, money ruins everything.
Once you sign your name to the Agile Manifesto and the Agile Manifesto becomes famous, the tendency is to try to capitalize on that. And the ways you capitalize on it is by pretending you hold some special knowledge that you can teach in a workshop or give a certificate in it.
However, the Agile Manifesto isn't some special knowledge. It's admirable because it articulated an important set of ideals succinctly and clearly, but it's really not anything that a moderately intelligent developer doesn't intuit with enough experience, even if they can't so clearly articulate it. And you can't really teach it in a workshop or give a certificate in it, because the application is different for every team. The way you apply agile is by getting intimately familiar with the individuals involved in the team and slowly adapting the process to fit their needs: you can't choose individuals and interactions over processes and tools if you don't know what the individuals and interactions are. The agile manifesto may be about individuals and interactions over processes and tools, but processes and tools are a lot easier to sell.
This means that even if you came across the Agile Manifesto and really grokked it, and maybe even signed your name to it, you've got a powerful incentive to then go forget everything you've learned.
In a broader sense, this is why ideas have to be evaluated on their own, irrespective of who those ideas came from. Ideas can be perfect--humans can't.
Pure Agile (the Manifesto and Principles) are all about adapting to reality.
Canned process sold as “Agile” probably won't work without adaptation to your concrete circumstances, but that problem that invariably comes with using canned process rather than team-owned process adapted to the team and their concrete circumstances is exactly the problem that motivated Agile.
That some of the signatories of the Manifesto are now on the dark side peddling canned, one-size-fits-all (but none very well!) process is annoying, sure.
> I still haven't figured out how quality assurance and test fits into the same iteration in Agile
Sometimes it won't. From your description, the concrete composition of your team with regard to those functions sounds problematic for that. In the long term you might be able to evolve your team to alleviate that, and maybe that is desirable, but in the near term, well, “people and interactions over processes and tools”. Forget what any guru tells you about the right way to do Agile if it doesn't work for your team dealing with the problems you face, and especially if they are just telling you your way is wrong without even proposing a testable alternative.
Agile isn't canned process, it's the rejection of canned process in favor of empowering the people actually doing the work.
Process is often vital. I used to write avionics software for the military. As you might guess "move fast and break things" was not our mode of operation. Retrofitting a helicopter requires massive amounts of process and waterfall. You can't not follow a plan. You have to decide where you are putting the electronics, and the specs on rack sizes, power, etc. Then you design the electronics to that. Then it needs to be manufactured, and so on. You can't decide at the flight readiness review that you want a screen on your radio that displays the weather.
This is really old stuff. I was taught this (agile way of thinking) in college in 1983 by an electronics professor. He stratified it differently, talking about goals, strategies, and tactics. People get hung up on tactics - "all code must use lower snake_case variables", or "stand up can't be a status report", or, .... But those are all tactics, in service of a goal. You need to remain flexible he lectured (and I agree), and recognize when the goals have changed, and/or when tactics and strategies conflict or make things non-optimal. That's all pretty much what the agile manifesto says, except they impose a weighting on values that I don't think fits all cases. As in my example above, sometimes process is more important, despite the many costs that kind of thing incurs. Cause, you know, we don't want people to die when they fly the helicopter.
And in that helicopter retrofit, the SW lead (Hi Paul!) had a saying "design a little, code a little, test a little". It was never waterfall, even back then, though you were in a waterfall process necessitated by the reality of complex projects. It was all about responding flexibly where you can, and being rigid when you can't. You don't push a sw patch 2 days before the flight readiness review. You don't decide to add components to a radio if it exceeds the designed power specs of the power network in the airframe. And so on. You may be able to change a screen element on a flight computer if the pilot suggests something is confusing or sub-optimal, but even that has high costs (revise all the training materials and documentation, simulators, etc, run all pilots through the training, it can run into millions). Always think flexibly about why you are making these decisions, why you are doing this, why you, say, empower people vs canned process, as you stated.
Without that, you are just back to unthinking process, claiming to be agile while you are anything but.
Back to how I started - I have yet to come across anyone running around extolling a specific methodology, including agile, that was anything but unthinking, inflexible, and unable to answer real world questions. It's just particularly ironic given the name 'agile'.
Edit: I left something unsaid, and it's important. The Agile manifesto is at the level of 'strategy', not goal. It is not ever our goal to "respond to change", as it states. That's a strategy. Our goal may or mat not make that strategy possible or desirable. I don't want to kill people, so if I'm making a helicopter change has to be planned out very carefully. Agile is mostly a set of strategies aimed to get working software into the hands of clients as soon as possible because things like specifying what you want, pricing out development costs, and so on are so hard. That often just doesn't hold. While I can't disagree with what is written given the states objectives, I think it is just another somewhat inflexible way of thinking about systems. It works for a lot of bay area type development, but goes off the rails in many other domains if you don't recognize the values listed, and the relative weightings, are based on a number of assumptions about what your real goals are.
> Agile doesn't reject anything, including process.
I said Agile rejects canned process on favor of empowering the people doing work, not that it rejects process. By canned process, I mean externally-imposed process not owned by and adapted to the needs of the team.
> Process is often vital.
Process is always vital; Agile (the Manifesto/Principles) is very much about how you get the right process for the team and moment.
> Back to how I started - I have yet to come across anyone running around extolling a specific methodology, including agile, that was anything but unthinking, inflexible, and unable to answer real world questions
Agile is not a specific methodology; it's a loose philosophy about parameters for a meta-methodology that enables you to continuously optimize methodology.
People selling a concrete methodology and calling it Agile are either missing the point of Agile or hoping their customers miss it.
Anyone who tells you it's anything specific is trying to sell you something.
We were replacing an existing, legacy system, and even though we switched over "piece by piece", each module of the old system was at least 10-12 sprints. And the end-users couldn't do a meaningful test of their normal tasks until most of each module was implemented.
How to teams who manage to work more "agile" do this sort of thing?
It's fine for prying microservices loose from monoliths for relatively modern REST api systems, tho. Big problems I see there are that the problem often isn't the code, but rather the api itself that's a mess, and evolving the api breaks client code which you may or may not control.
Interesting, what are the problems you encounter if you try that?
In this particular case, it was a banking system that worked largely by nightly batch jobs. Data queued up during the day was processed at night. Job A triggers Job B, which Triggers Job C; etc. What we wanted was to change those batched transactions into individual transactions that could be processed in synchronous quasi-realtime.
With reconciliation happening near the end of the batch processing, how do you do that?
And mind you, this system moved more money every day than the GDP of many countries. Failure was NOT an option. Not if you don't want to roast the entire US economy. So the consequences of error were truly terrifying.
There’s no silver bullet. Small a agile is really a way of thinking as a team. Unfortunately, the big guys often sell it as Big A Agile with a list of processes you need to follow so you can avoid that thinking.
Of course, at the same time we tried to improve certain workflows and clean up various parts. But mostly, this was a straight replacement.
Having said that, I still think there are projects for which it is best to do it the way you say. I've certainly had a lot of success with that approach as well. As others have said, if it ain't broke, don't fix it.
Except that when the versions are coming out too quickly, you don't know to which version you should roll back, because it took longer to encounter the problem (let alone pinpoint its origin) than the time between releases and several or even many occurred meanwhile.
>Teams only checked in – wait for it – every three months in a formal schedule review. I listened as Henrich mentioned that the teams complained about the volume of paperwork he makes them fill out for these quarterly reviews.
In my experience, embracing agile is an organizational change, not just an IT team change. Product owners and subject matter experts need to be more engaged with their development teams. Not all business folks want to do this: they have their own day-jobs to get to. So when the IT team starts iterating faster, but the appropriate business / project stakeholders do not change their workflow, you get this "AgileFall" combo that just does not really work.
One persons anecdotal experience about agilefall doesn't mean it doesn't work. Don't fall for the trap of using analogies and anecdotal experiences as evidence. To top it off this guy didn't even measure the outcome of agilefall, all he did was declare that it violates some made up philosophical principles and demand change.
I've seen a project manager literally catch himself from doing something waterfall like it was the plague. Do people not realize that waterfall is the only technique used to build airplanes and bridges? I would not get on a plane built/being built using agile.
Note that I'm not criticizing agile. Agile works, waterfall works and no logic or evidence provided by that article says that combining what works for both is not good.
That's the problem, though - they should. (And, actually, they do, but they've been twisted around and used incorrectly so often that they've come to mean nothing).
I'm pretty sure airplanes and bridges aren't designed using 'waterfall'. I'm sure they're built, i.e. 'deployed', using a well-defined process. But, ideally, the well-defined process by which they're built should itself probably be designed or developed with 'short' iterations.
You're right. It is evidence. This is an obvious fact that I mis-represented on accident. Let me rephrase for clarity:
"This persons anecdotal evidence is not good enough or unbiased enough to qualify as evidence to be used to arrive at a conclusion"
>I'm pretty sure airplanes and bridges aren't designed using 'waterfall'. I'm sure they're built, i.e. 'deployed', using a well-defined process.
Ok let me rephrase for more clarity in case people don't have the ability to know that by "designing" a bridge I can mean "build" and "deploy" as well.
"Airplanes and bridges are designed and built using waterfall."
>But, ideally, the well-defined process by which they're built should itself probably be designed or developed with 'short' iterations.
This was how the MAX part of the 737 MAX was designed. Using a short iteration on top of an existing design. It leads to shorter turn around and cheaper costs. With the small expense of 1 or 2 plane crashes.
If you can afford a plane crash go agile, if you want to build something correctly go waterfall.
A lot of software (that's often written about) can be tested 'in production' with fairly minor consequences in the event of 'failures'. That's an obvious difference between developing or maintaining software versus something like airplanes.
But testing or validating the changes made in short iterations is important regardless. Agile stresses the utility of short iterations precisely so the changes in each iteration can be tested and validated ASAP. Whereas in a waterfall methodology, a larger set of changes, i.e. changes that would in agile be split among a number of iterations, is tested or validated together. And the more changes that end up being tested together, the larger the risk is that those changes, and more of them, will need to be scrapped or redone.
The problem with the "MAX" changes of the 737 MAX seems more due to those changes not being tested or validated with the relevant users, i.e. pilots, at all before being 'released'.
Yes, the flawed (or incorrect) reasoning was that the changes were relatively minor and thus didn't warrant the regulatory requirements for 'significant' changes. But that doesn't seem to have anything to do with how the changes were developed, whether their methodology was agile or waterfall (or anywhere in between).
Obviously you're more likely 'to have a bad time' if you don't respond to, let alone seek, feedback from users about the changes you're making.
I'd bet that these days it's a lot more common for a process imposed in the way Agile is a reaction against to be some variant of Scrum with which management is enamored, rather than waterfall.
For this kind of project it doesn't make sense to operate in a pure agile fashion, but we also have broad license to operate how we please in some aspects of the project (i.e. the UI), and those more closely resemble an agile method project.
If it's creating more overhead, it's not agile. It may wear the label, but it's not the real thing.
If the existing process was already the optimal one, the agile thing to do is to not change it. If the "agile" changes are adding overhead, the actually agile thing to do is to go back to the previous process.
"The perfect way to do project management"
With this definition in place the future is solid, no criticism can bring agile down because agile can evolve into whatever you want it to be.
Literally, what is the difference between "agile" and "whatever works"?
Many places say "We will be agile according to this rigidly defined process". The contradiction is obvious.
Look at the difference between the principles of Theory of Constraints, and what most organizations do. Most organizations want to remove constraints, they know they're a problem (since, well, they're constraints). The difference between ToC and what most people do is that ToC emphasizes a deliberate focus on the singular constraint in the process. I've seen teams and offices optimize the hell out of areas that aren't the bottleneck. Meanwhile, they still spend 3 months at the end of their projects running their massive, manual test suite. Do the tests bring value? Yes, immense. Do they bring value at the right time? No, because they take 3 months to run so they're delayed until the end of the project, so the project schedule leaves 3-6 months of buffer after that testing to conduct rework. A deliberate focus, then, should be brought to the testing activities and identifying how they can be conducted more rapidly, ideally automatically, and feedback brought in at an earlier stage.
Agile, as described in the manifesto, brings in a set of values that are intended to provide focus on what's important for both the team and the customer. To a large extent, it means "whatever works", but with focus on the values you share with your team and customers, and the tradeoffs that will be incurred based on the choices made.
>Henrich still had a process where success was measured by reports, not outcomes. It was the same reporting process used to measure projects that used linear, step-by-step Waterfall.
What are reports, if not documentation of outcomes? What are outcomes without evidence that those outcomes happened?
I can see what he's saying, but the more accurate description of the problem "the reports included unnecessary and distracting information." But that doesn't make as good of a blog post as casting it as "paperwork vs understanding outcomes."
The whole thing feels like it needs to invent a "paperwork = evil" boogieman just to allow a process change that throws away useless paperwork and allow a team to focus on the paperwork that you just don't call paperwork anymore.
Email is paperwork. Version control is paperwork. Change management is paperwork. It doesn't matter if it's software updating a database instead of a secretary updating 3-ring binders. Most of development is paperwork. In fact, online databases mean we do way more paperwork. If we were bottlenecked by that sea of file cabinets in the basement containing all our the legacy projects, we would have way less paperwork just out of necessity.
Standups are paperwork that just happen to not use paper.
If it isn't written down, it didn't happen!
(Memory is not an infallible record.)
There's a common misperception in the industry that agile is 'fast' but waterfall is better at handling risky situations.
Subtle difference I think: Waterfall is perceived as being more predictable, which is why it would seem desirable when stability is needed.
Fully agreed that while Agile is meant to deal with change better, it's perceived as being a silver bullet to "Speed things up". I have yet to meet a enterprise manager who uses the word correctly :(
Pushing agile as faster rather than better is the biggest problem agile ever had.
Majority of my managers have heard the word somewhere, tried to infer its meaning from dictionary definition, and started using it arbitrarily. Most frequent phrase is "We need to be more AGILE!", which 98.72% of the time simply means "We need to whip the workers more and have them work harder, faster, longer.... somehow".
The time to build the whole might be no faster or even slower - but then again you might find out that after delivering only half the 'required features' that you have done enough to meet the customer need, and stop there. Big projects rarely want to stop though until the shopping list is complete.
Where it goes wrong is inflexible scopes. The business obviously wants - needs - to tie down all three parts of the triangle, and the fact that this rarely works isn't going to stop them.
The process name doesn't matter, the results of the process should be the focus of every product company.
The daily scrum is a planning meeting. It's meant to help the team understand the big picture scoped to 1 sprint while helping each other achieve a shared commitment.
I personally try to correct from this problem by asking questions whenever work may relate to me, and being very familiar with the backlog to know how things will coordinate.
But also, I rant when people tell me about meetings without any conclusions for the team.
Have you considered the possibility that your temperament or personality might be the reason that multiple people on your team feel they have to pre-emptively justify themselves to avoid angry criticism? A meeting where people have to stand in front of their peers and potentially admit their failings can be quite anxiety-inducing if team members can't be trusted to respond in a professional manner.
It is sad to me to see a non-manager saying this is somehow the part they _like_.
The best experience I've had with agile processes was also one of the least stressful and most relaxing workplaces I've been in. I knew what I had to work on, I could trust that it was the _right_ thing to work on that wouldn't be a waste of time, and I had no pressure to "push myself", what could get done in work hours was what could get done in work hours, and sometimes you had to schedule some R&D/exploration time too. (The analogy of "sprint" is actually alarming when you think about how it's used -- you can only "sprint" for a short distance, you can't do back-to-back springs for 20 miles, the human body doesn't work that way!)
I think that may actually be one of the fundamental divides in "agile", whether someone is trying to use it as a productivity tool to "press ahead" developers who would otherwise be "lazy", to squeeze out the maximum possible work out of them, being unconcerned if you burn them out cause you can just replace them with another commoditized developer slotted into the "scrum".
Inputs for sprint planning include the relevant empirical data (velocity, burn down/up etc.) to help with planning. In my experience, these data points are used to measure productivity, which is dangerous. Companies end up measuring success by outputs, rather than outcomes.
Business units need to stop weaponizing Agile and realize they too need to change in order to inspect and adapt towards building things people want.
Like I said, I've had GREAT experiences with agile practices, my _favorite_ workplaces have been using agile practices. Also terrible ones. Knowing that the terrible ones were "doing agile wrong" didn't make them less terrible.
I even said very explicitly "It turns into a process that is more like waterfall, but..." So I'm a bit perturbed by your response insisting that this is not inherent to agile. When I was pretty explicit in saying that too.
LOTS of companies _saying_ their doing agile result in misery, even worse than before they pretended to do agile. On the other hand, doing agile well can be wonderful. In order to figure out how to make more of the wonderful than the terrible, we have to get _beyond_ just pointing out that "the agile manifesto clearly states". Cause it's clearly stated that already, and it hasn't stopped the terrible.
what is not sustainable is keeping working day and night for long period of time I feel.
Slipping back into waterfall can happen because many of these issues have sequential dependencies and are one time request driven processes. It's a bit of a nightmare.
IMHO it's not actually a safe investment strategy, but I understand where it comes from.
> reviews would be about frequent delivery, incremental development and how leadership could remove obstacle
I would think that in a Lean process that outcomes and therefore reports would be focused on performance of North Star Metrics, KPIs, comparing those against goals, analytics/research to help understand that performance, and hypotheses/strategies for how to improve.
I'm a big fan of Lean/BML approach and I'm assuming this is what is happening but the article doesn't really send that message. Because of that omission, after reading but before seeing the author my first thought was this was consultant blather.
It works great! Just three more iterations, and we'll be done with maintenance!
In the end no matter what you do software and hardware development is unpredictable. No methodology will ever change that.
This is especially true with estimates. New work, new problems, unexpected difficulties, etc.
Over-planning can lead to a lot of rigidity in the process if it tries to plan too far ahead.
Or they try to compensate for technical problems by doing endless replanning meetings instead of spending the time on solving the problem.
But a good long term road map is really helpful when you have to make a decision to go one way or the other. Just accept that it’s a rough guideline and not an exact plan that’s set in stone.
Agreed. I think some people think agile means no plan, whereas I think it means frequent scheduled review of the plan as work is completed and more is learned.
This a false straw man argument. One the project plan is created, it should be updated regularly (daily/weekly depending on the size of the project) as tasks are completed and new information comes to light. The plan is to give a prediction for the likely end date and cost of the project, but must be update as new information is known.
Welcome to my world :). Just a few weeks ago I had a conversation with a project manager explaining that no amount of additional planning meetings, motivational speeches or free pizza can change reality.
I'm not saying those project managers don't exist, but I wish people wouldn't compare perfect agile against pathological waterfall.
An up-front design is sometimes necessary and valuable, in both agile or waterfall projects, but it always carries a risk of including mistakes or mistaken assumptions that result in dangerous flaws.
When such a flaw is discovered, especially late in the project, it is often very expensive and difficult to fix, because it has become foundational to the solution. This is true regardless of whether an waterfall or agile process has been adopted, and there will often be significant pressure to implement a hacky fix, instead of the truly correct one.
Where agile differs from waterfall is that it does, at least, philosophically accept that nothing is set in stone, even at a late stage. Discussions regarding how to address a flaw can then proceed on a basis of cost vs. risk vs. difficulty, and a hopefully rational choice can be made.
In contrast, the rigidity of the waterfall process is frequently used as a bureaucratic weapon to prevent these discussions from even happening. Decisions made in the project's initial stages are held to be settled and unalterable, and those seeking to overturn them usually have to fight a horrendous, uphill battle to do so, even when they're in the right.
I would liken the difference between design in agile and waterfall to that of scientific theory vs religious dogma. Neither necessarily has a monopoly on truth: A theory might be mistaken while dogma, by chance, might be correct. And challenging well-established scientific theory can, in practice, be almost as difficult and painful as disputing dogma. But science at least accepts the possibility of change, however hard it may be.
So much this. I can't count how many times I had a client tell me "We want the delivery on the proposed time, on budget, I don't care" even when analysing the client's design it shows obvious flaws and won't work as intended. Most of them usually gets fired from doing this, months later, but even so they just get another same-level job on another company, and the cycle goes on.
There's no development methology that will ever work on irrational business demands, and that happens all the time. Sometimes I think HN's users are just too entrenched at product-driven companies, or B2C ones, and never get to experience the amount of BS and CYA that happens on B2B, large-business deals.