My Foreword to “The Art of Agile Development”
martinfowler.com
martinfowler.com
I say as much in the first paragraphs of the book:
> Agile is everywhere. And paradoxically, nowhere.
> In the 20 years after the Agile freight train roared into software developers’ consciousness, the number of companies calling themselves “Agile” increased by orders of magnitude. The number of teams actually taking an agile approach to their work? Not so much. “Agile,” the easily repeated name, is enormously successful. The ideas behind Agile—well, most of them are ignored.
> Let’s fix that.
If anybody would like to talk about the book, I'm happy to engage. If you're curious about the book, the first three chapters are currently up for free on my website, and there's a lot of excerpts from the remaining chapters posted too. Follow the links in the table of contents at the bottom of https://www.jamesshore.com/v2/books/aoad2.
My negative experience with Agile has mainly been through people who are ideological about it and insist, when agile fails, that it was not the fault of the process or some other factor, but rather the fault of the people who "didn't implement agile right."
A bit of humility from the Agile community would be very welcome, as well as clarification of what types of problems it works well for and which ones it doesn't, where the dangers are, etc.
Agile works well when organizational leaders are willing to invest in changing the constraints of their organization, and team members are similarly willing to work to customize the philosophy—'cause that's all it is, a way of thinking about software development—to their specific situation.
Given those two things, I've yet to find a type of software development where Agile didn't work. People have used its ideas for web development, embedded systems development, back-end, front-end, Java, C++, JavaScript, etc. etc.
But those two things can be really really hard, which is why I think cargo-cult Agile is so common. And trying to be Agile without them is dangerous, in that it can easily be worse than whatever it replaced.
I talk about them more in chapters 4 and 5 of the book. You can find an excerpt of chapter 4, including a list of the investments needed, here: https://www.jamesshore.com/v2/books/aoad2/invest_in_agility
- Water Epic Fall. An obvious symptom of Cargo Cult Agile. This happens when the phases of a waterfall project are disguised as epics that represent a huge fraction of the development. In my own experience this is caused when the MBAs try to force Scrum from top to bottom so they can pretend that they're doing agile on paper and thus "fulfill the values(tm) of the company". This is even worse than usual waterfall because it places an artificial limitation on making multiple components of the project live and evolve together and so "world collides" when each time an epic "is closed" and the new one has to find a place to fit, somehow. Proposition: epics are a way to breakdown work and work is done by a the team and so leave it to the team doing the work. Rule of thumb: stories are defined by time while epics are defined by stories with a common concrete idea.
- Daily meetings are mostly reports to the boss. Proof of that was when the boss wasn't there meetings that normally endured for 30 minutes and more would only take 10 minutes. In none of these meetings the boss helped to resolve the blockers; other devs did. Proposition: no boss in daily standup; DSMs are a tool for and by the dev team.
- Devs are blamed for not finishing stories in time. This is usually due to a bad backlog grooming where definition of ready is left as an exercise to the reader. It is also a symptom of the Water Epic Fall described above. Devs has to deal with the fact that the code isn't simply ready to integrate the new feature but the exercise to check if it was was never done in the first place. This is made worse when it is the dev who estimated the story and is now "responsible" for it. Also made even worse when "architects" with veto power decides what is ready or not and what the stories are. Proposition: again, let the devs have the final words on the backlog and its stories.
- Asking for permission to fix problems and remove friction. This is usually a problem caused by survivor biased MBAs who think of the heroics they did 20 years ago as war medals and a good development methodology because after all "this is why we could afford our new building". This problem often forms a symbiotic relationship with a culture of fear lead by barons aka "team leads". Proposition: revolt. Seriously. "We can't work this way anymore" during the quarterly report mega-meeting in front of the C*Os has its impact. Otherwise change jobs and make sure to let HR know by naming names. It's sad to see this kind of situation but if you don't it will poison every aspect of your life.
- Nobody RTFM. Seriously. Who can say they have read at least one book on Lean/Agile/Scrum/DevOps from cover to cover? I would say most MBAs didn't because there's an entire industry based on giving very expensive courses and on agile consulting just to repeat verbatim what are on these books. And even then most of the time you will never heard of concepts such as "lead time", "feed back", "release often", "no wip", etc. Some devs do tho, but they usually quit when they realize that their conversion efforts go to waste and provoke the wrath of the MBAs and the Teams Leads(tm) because either "we're not a startup" or "we can't be like Google"
- Large teams. Scrum doesn't really say much about how to work with multiple teams. Even less so where you're dealing with hardware and firmware development.
> The number of teams actually taking an agile approach to their work? Not so much.
Classic no true Scotsman. That inept excuse gets always invoked by the proponents of Agile, Lean, Scrum, Six Sigma and Communism.
> No true Scotsman, or appeal to purity, is an informal fallacy in which one attempts to protect their universal generalization from a falsifying counterexample by excluding the counterexample improperly. (Wikipedia)
“It doesn’t work because you’re doing it wrong” is not an example of the fallacy.
(For example: “A: Light bulbs suck because they don’t light up when you smash them into your head. B: That’s not how lightbulbs work. A: No True Scotsman!” Yeah, no.)
Original version for commenter below as he is making an issue out of the edit:
"Classic double down on the Scotsman by a consultant"
(Note: thrway edited their comment after I responded.)
I think that in the case of Agile, there is a set of principles and values clearly defining what it is. For a project to be Agile it needs to follow these. (Most projects ignore them and they just follow some Scrum practices). Just saying that you are Agile doesn't make it so.
Yep, these islanders build airstrip and related infrastructure following good engineering practices, yet no cargo comes. Instead comes a consultant with his "No cargo because you're doing it wrong", and now they build the airstrip under his guidance following Agile principles. Still no cargo. The consultant continues with his "No cargo means what you're still doing it wrong" ...
The Agile is the epitome of a cargo-cult.
Edit: And then some will say "If you are stressed, why don't you just reduce the speed of the conveyor belt? Agile is meant to be customized!", or "If an item wasn't done properly you can just move it back to the start of the belt so you can look at it again!". My answer is "I just don't want that damn conveyor belt in the first place!".
We could replace agile with something else and it will end up the same way.
These false priests of technology substitute their tools for their lack of understanding and ignore completely that just having a pipeline and doesn't mean you are doing DevOps.
1. I suspect you're ascribing a level of power to "the Agile custodians" (who, exactly?) that doesn't exist. Nobody has the power to prevent people from misinterpreting Agile, no matter how much as they might want to.
2. The first chapter of my book (the one under discussion) does discuss how the philosophy has been corrupted in the real world. It's a major theme of the book.
The first chapter is online for free at present: https://www.jamesshore.com/v2/books/aoad2/what_is_agile
I think Agile fails because the manifesto says nothing about management’s role other than “business people and developers will work together.” It’s no surprise then that management fills their version of their role to be whatever they want- usually to the detriment of the developers and the core Agile principles.
God no. There’s some good training out there, but a whole lot more bottom feeders. I include all the certification mills in that category. Certifications are a racket.
> Our team was sent to a week long SAFe Agilist certification course. Our management was notably absent.
Par for the course, sadly.
> I think Agile fails because the manifesto says nothing more about management’s role other than “business people and developers will work together.”
Agile’s a big tent and a lot of people have written about agile management. Johanna Rothman is particularly prolific. (I have a few things to say about it in my book, too.) If managers aren’t learning about Agile management, it’s because they’re not looking, not because the information isn’t out there. Although I think part of the reason they’re not looking is because Scrum was so successful, and Scrum is fairly anti-management, or at least has nothing to say about it.
(Scrum’s success has been both a blessing and a curse. On the one hand, Agile probably wouldn’t be on everyone’s lips today if XP was still carrying the torch. On the other hand, Scrum’s surface simplicity and cash-grabbing certification scheme opened the door to some truly awful training.)
That or invent something better that management cannot infect
I just would also like to suggest that my experience on that team is not unique. I think that more companies might be successful in implementing Agile if the relationship between management and the development team was more clearly defined; and if the managerial role was more clearly defined (to include constraints on managerial roles). Empowering knowledge workers requires managers to relinquish some of their oversight and authority, by definition. I don’t think Agile training or the manifesto makes this explicit enough.
I understand that embedded and large systems are most likely to eschew agile as it's frequently not a good fit for the work, but I work in Python/Django mostly at the moment.
Scrum especially seems to fail in spectacular ways vs. other production methodologies in that when Scrum fails, it can be a net negative to the company. I feel as if I were to restrict my job search to companies that have anti-"Agile" methodologies, I would be more likely to work in a true agile environment.
I understand the pressures that create Scrum, as nature hates a vacuum, but I can't understand why there isn't any slack at all to experiment and fail in most smaller organizations. Perhaps we're disinclined to "hire smart people and get out of their way" nowadays, because we can only make data-driven decisions.
I'm sure that the best products are made by a small dedicated team of competent individuals communicating openly with one another, and I'm equally sure that success is rarer among those orgs. But, as someone with a degree in psychology, it's plain to see how operationalizing healthy communication patterns can have a counterintuitive result.
If I have any say in the matter, I'll never work in an "agile" shop again.
Anecdotally, Coinbase and Airbnb didn't use Agile/Scrum and I never have interviewed in a company that mentioned that. Usually how it works that there the company knows what they want and there is some kind of roadmap of projects or experiments which are based on improving some kind of goal or metrics. Teams usually work in somewhat iterative phases, were you start with design, build the first prototype/mvp, test internally until you get to a stage to run an experiment with the users. During this time, your team meets every now and then, gets feedback from others and you do reviews with leadership. Then eventually if the experiment seems promising and the feature is good enough, it gets shipped. You can have sprints or not but no-one talks about or uses story points, retrospectives, planning pokers, product owners, user stories or other agile ceremonies. You have goal and project to accomplish, and then the management/leaderships tries to estimate with the team how long things will take.
Pragmatic Engineer had article about this too: https://blog.pragmaticengineer.com/project-management-at-big...
Kanban or Lean are usually much better for smaller teams.
For a larger company, SAFe is a much better option but there’s a danger that by default people will suggest that individual dev teams follow Scrum (because that’s generally the default). Individual devs teams can use anything they want within each PI and usually Kanban or Lean will go much more smoothly for everybody.
https://www.brightball.com/articles/reality-driven-developme...
Agile for majority of devs and consultants (and not for the people around manifesto, such as Fowler) is conflated with a 2 week or so status reporting and planning process. It has nothing to do with product development as these status reports typically don't ask 'what is the status of the product' but the status report is typically about 'what everybody is doing and if they are busy'.
This has nothing to do with the original agile manifesto and if you read through it you'll notice that this modern 'Agile/Scrum' time-boxed busywork and status report is in contradiction with its principles. Keeping devs busy is obviously the goal of this consulting scrum as that is what consultants charge for.
Fowler himself gave up years ago explaining the difference. When words become popular, their meaning gets hijacked and repurposed. He explained this in a blog post of his https://martinfowler.com/bliki/SemanticDiffusion.html
I love the idea of Agile, hasn’t worked at our company since we got spotified.
A company doing Agile poorly most likely would do Waterfall poorly too.
https://blog.crisp.se/wp-content/uploads/2012/11/SpotifyScal...
Our problem at the moment is we have no processes/documentation because we’re Agile™
[1]: https://engineering.atspotify.com/2014/03/27/spotify-enginee... [2]: https://engineering.atspotify.com/2014/09/20/spotify-enginee...
I think that "waterfall" is often a straw man.
Every discussion where Waterfall comes up has someone saying this, I'm glad you've not experienced it. But it has existed, does exist, and will continue to exist (because enough project managers are idiots). One of my first jobs (I quit) was a failed project that persisted because of gov't money, but would have failed as a commercial venture (started circa 2007, failed to deliver at least through 2011, I was on board for a year and found an exit). It was 100% Waterfall, they literally delivered the wrong thing several years late and with around $1 billion spent. Great for the contractors, awful for the taxpayers.
But yes, that is what Waterfall is. It's a large scale project executed linearly with limited-to-no-feedback. Large scale is somewhat subjective, but basically if it's done in less than a few months it's probably not at a scale where your choice of development model actually matters.
Estimates are never accurate. You can’t possibly have all of the information present up front.
Waterfall's principle failing for large scale projects is that it does not incorporate sufficient (if any) feedback or delays the feedback until too late in the cycle, and plans too far out to be reasonable (do you kwon with any real confidence what you'll have for dinner in 3 years, 2 months and a day?) V-model generally also plans out very far, but incorporates V&V as an ongoing activity (it's name is a bit of a pun, both incorporating V&V and usually being charted out as a V, vice a waterfall for Waterfall). I&I may have a big design phase, but it's usually not as fully designed as either Waterfall or the V-model because the iterations are meant to provide checkpoints for validating the design and potentially changing direction (like I wrote, slow-motion Scrum).
If the company from top to bottom isn’t Agile then I don’t believe you can truly be Agile.
The requirements were hammered out in brutal detail by an experienced specialist. Each requirement agreed with the customer, and then each requirement carefully worked into a design. Each requirement clearly linked with every part of the design that fulfilled it. The design reviewed and approved. The code written in literal style (using a variant of noweb), with discussion and commentary, and that noweb document compiling into the source code to be onwards compiled and also a beautiful document listing the full source code and the various commentaries and discussions of it (such that one understood the code not by looking at the code, but by reading the document, which was like a structured guided tour of the code, and to look at the actual code was to see a one-to-one relationship with the documentation).
Each part of that document linked to a part of the design, and thus to the requirements. Tests had been written independently of the code, based on the design and the original requirements. Test documents were printed out, and an individual conducted every test on that document, writing down version numbers and so on, and writing down each results, circling PASS or FAIL, signing their name to it. Successful test documents were sealed in an envelope, with a date and signature, and put into storage against the customer requesting to see one (which happened every so often).
The original requirements could, years later, be followed through the design, through the code, through the tests; the whole chain verified on demand.
The software I worked on was developed over most of a decade (I was only there about four years), with a delivery a couple of times a year. The customer never once reported a bug. Every requirement was met. Other contractors had their part taken away from them and given to us, and when it was finished they took it and asked us if we could start working on the next generation product of the same.
I have never before or since worked on such high quality software. It was only possible because we happened to have really high quality customers, who actually knew what they wanted and had the ability to say it and agree to it.
8.3 There was once a programmer who wrote software for personal computers. "Look at how well off I am here," he said to a mainframe programmer who came to visit. "I have my own operating system and file storage device. I do not have to share my resources with anyone. The software is self-consistent and easy-to-use. Why do you not quit your present job and join me here?"
The mainframe programmer then began to describe his system to his friend, saying, "The mainframe sits like an ancient Sage meditating in the midst of the Data Center. Its disk drives lie end-to- end like a great ocean of machinery. The software is as multifaceted as a diamond, and as convoluted as a primeval jungle. The programs, each unique, move through the system like a swift-flowing river. That is why I am happy where I am."
The personal computer programmer, upon hearing this, fell silent. But the two programmers remained friends until the end of their days.
If only that were the rule. It's not. It's a rarest of rare exception. As it is, no one has time. If they miraculously clear that hurdle, they are either too scared to commit or quite frankly don't understand their business' needs and can't commit. Agile is simply a reaction to that cronic ambiguity.
Because waterfall generally has milestones as a coarser grained way of ensuring evidence of forward progress, all deliverables generated around the time of a milestone just become candidates for delivery. Do you want this version that is missing two features, this version that is missing one but the other might be buggy, or to wait another two weeks to get the whole thing? If you have anything that looks like QA or acceptance tests, those can be run on deliverables, keeping people a little more comfortable with the process than they would with strict Waterfall.
This explains so much...this was something they enjoyed, and somehow conflated that as some kind of proof that it would work for all software development.
As someone who has never worked in the industry (only in academia), I'm curious: what is it that turned the tide?
Could someone elaborate (a) why agile/scrum fell out of favor; and (b) what is its modern replacement? Thanks!
"Every great cause begins as a movement, becomes a business, and eventually degenerates into a racket" -Eric Hoffer
I think another important thing to point is that most* software projects fail. Trying a new methodology is a good way to deflect blame and extend life into a bad business, team, manager, product.
b) I don't think so. I think companies are still chasing agile. I personally hope it makes a resurgence. Unfortunately Waterfall is perceived as an urban legend vs the monstrosity it is/was on the industry by the new generations
* I thought it was 50%. [1] suggests it's 75%, and this[2] suggests 90%+
[1]https://www.geneca.com/why-up-to-75-of-software-projects-wil...
[2]https://www.quora.com/What-percentage-of-software-projects-f...
from the darkness outside and within
by dreaming of systems so perfect that no one will need to be good
but the man that is will shadow
the man that pretends to be
The name sounds cool tho