Poor Man’s Agile: Scrum in 5 Simple Steps
scottporad.com
scottporad.com
Diana Larsen and I sat down last year and wrote down all the different ways we saw people doing Agile, what worked, what didn't, and why so few teams were getting the promised benefits of what the agile movement promised, lo, those many years ago.
The nice version is here: http://www.agilefluency.com
The nasty (but much shorter) version is here: People care more about slapping the "Agile" label on their door than they care about doing the hard work of changing their organization's work habits. Change is hard. Doing Agile well takes change. It's easier to get a certification and call it a day.
(Not bitter. Nooooo... not bitter at all. But so very tired of the bullshit.)
Ah good - somebody else go there first ;-)
To James' link I'd add reading the Agile Manifesto values (http://agilemanifesto.org/) and principles (http://agilemanifesto.org/principles.html - everybody forgets the principles).
If folk want to learn a bit more about Scrum the Scrum Guide is a nice summary and not that much longer than Scott's mild oversimplification (http://www.scrum.org/Scrum-Guides).
(And since he's too polite to pimp it himself - James' book co-authored with Shane Warden is one of the ones I recommend to folk wanting to learn about agile http://www.jamesshore.com/Agile-Book/ - well worth a read if you're interested).
It also made me depressed a bit on how my team is doing after a year and a half of development =/.
Basically, the article is a backlash to things like http://programming-motherfucker.com/ , which is a backlash to people doing Scrum, XP, etc. so wrong that it seems no better than the beaurocratic monstrosities that the agile manifesto was a backlash to.
The kinds of issues that I've seen are things like:
* Teams not knowing about some methods that you can use to estimate large numbers of stories quickly - instead of spending N hours doing rounds of planning poker for each one.
* Teams trying to breakdown too much of the backlog at a time rather than relying on keeping distant
* The PO not coming to the meeting with a list of prioritised stories so folk are trying to prioritise and estimate everything at the same time (I'm not saying that prioritisation can't change during the planning meeting - but the PO needs to come prepared IMHO)
* Stories coming in at the wrong level of granularity so that everybody has to spend time breaking stuff down to implementable chunks.
* Teams lacking any kind of definition-of-ready for stories that are at the stage where they can be implemented.
* Teams lacking common agreed definitions of done for stories
... and so on...
(this is for one or two week sprints mind - if folk are running old-school one month sprints then I guess longer sprint planning meetings are unavoidable.)
Googling around that will find you some variant methods that I'm too lazy to dig out ;-) Ping me if you need more.
For an organization, that can be fine. Maybe Scrum is too simplistic, maybe it's missing a few pieces vital to your organization. Experiment, season to taste. But I believe that there often isn't enough thought put into how badly that seasoning is needed. Rather, someone along the chain says "but we don't have that thing from the old way", whether that old thing is vital or was nothing but a procedure weight around the neck of the team.
As for a 478 page book, someone must have worked really hard to come up with enough fluff, diagrams, and humorous anecdotes to fill that many pages on something that's supposed to be "agile".
1) Scrum has been getting buzzword traction, so there is demand for books about it.
2) A book has to have a couple of hundred pages to justify the price tag.
Chances are that you don't know what suites you, that's why you are looking into scrum in the first place. First you do it strictly by the book and in time you will learn what works for you and what doesn't. Then you can start customizing become mature agile team/organization. OP has probably achieved this, but I think it can be harmful to suggest "pro" techniques to someone just starting out :)
If you are going to roll-your-own solution, start with retro and build from there. Retro is the most important part of the process. There's a reason it's the only explicit activity mentioned in the manifesto. As long as you keep reflecting on past performance and continually try to improve you'll probably end up with a good process.
It encourages everyone to ignore/bottle up issues until the magical retro day where everyone can spend a couple of hours venting their frustration. At which point nothing gets done and the process repeats. Project issues should be resolved immediately, directly with the people responsible and with actionable outcomes. Not during a glorified weekly/fortnightly 'meeting'.
If that's what happening then the retros are being terribly run. Especially if you're not coming away with an actionable outcome.
If you can deal with it immediately - you deal with it immediately. Nothing in any agile process says you should save it up for the retro. Many quite forcefully say the opposite.
The retro is for helping you spot larger scale issues or systematic problems that you miss / don't have the room to address in the day-to-day project.
(For example, we ran one last Friday where it was pointed that I'd been a bottleneck on several different tasks. I'd not been bright enough to spot that myself. Since the bottlenecks were with different folk nobody had spotted it at the time. Bring everybody together and - boom - problem identified and some options for solutions spotted. No angst or venting involved. There was cake though ;-)
That's what I mean by retro. Two parts: self-reflection, and change of behavior.
Sounds like your problem with retro is that behavior isn't changing. Maybe it's management stopping things from changing, or teammates who refuse to go along with what the team wants. A team that won't change won't get better.
I can just imagine some recruiter or HR person looking at resumes:
Applicant A - dozens of accepted to commits to the Linux Kernel, or leader of some 'known' open source project.
Hmm.
Applicant B - Scrum
OMG! We have a winner!!!
Unfortunately, candidates who take this kind of approach seem to list a lot of technologies they have only heard of or been in the same room as. If someone lists a technology in their CV I do rather expect them to know what it is and what it is generally used for and good and bad points.
However - your comparison is pretty much a straw man. I'm not looking for good Scrum / Agile experience instead of code experience. I'm looking for code experience and good agile team experience. Because - over the last ten years - the teams I've seen using agile approaches have been darn effective.
When I'm helping build a technical team you need great coders and you need great team players working in a way that makes everybody happy and effective. Ideally you want to recruit folk who are already great coders and are already used to working in effective team environments.
If you think either of those is a natural skill or simple then you're mistaken or very lucky. I can mentor folk up on both sides - sometimes it's actually easier to turn a good coder into a great one than it is to turn an asocial fuckwit prima-donna coder into an efffective team player ;-)
6. Once a week, everybody gets together for a few hours and roughly plans out the features that the product is going to have. The features are compiled into a list with the ones you're going to do first at the top, in the most detail.
Without having the feature set already groomed, those step 2 planning meetings are going to be brutal.
And the next person that says to me "derp derp derp, well it's better than waterfall herp derp" can go fly a kite. If you think we live in a world where there's only two ways to manage/build software, then you're a fool and have no business being in this business. Either start applying critical thinking or go sell crazy some place else.
Maybe there is a perfect project out there, somewhere, and they're doing Agile right, and it makes the product better, and everyone on the team is happier and more productive. But I've never seen a project like that. In five Agile projects I've had the misfortune of working on or alongside, All I've seen is stakeholders making implementation decisions in ivory towers, developers being told to not implement that "delete" function because it's out of scope (it's out of scope because the designer and UX people forgot to put it in, but fuck it, we'll pretend you're right and it's meant to be like that), and teams pretending stuff works and everything is ok when in actual fact, it doesn't, and it isn't and if people could just stop looking a Jira and start looking at the code then maybe we could start to fix this hunking pile of crap and actually start being proud of our work.
And before anyone weighs in with "Well those project clearly weren't doing Agile right", please, save your breath. Every project has been infested with people who have stacks of books on the subject. They love it, they suck it all up, they obsess over the minutiae of how to implement scrum, retrospectives, and sprints. They can talk of nothing else at work, at home, or in the pub. If they can't do it right, who can?
And herein lies the point, there is no way to do agile right. What there is, is good project managers and bad project managers. There are good devs and bad devs. Good designers, bad designers. The good people will make good things in spite of having Agile forced upon them. Then the agile non-thinkers will jump on the bandwagon and lap up the good peoples success as if it were their own. Then someone will write a blogpost about how Agile made it all possible and it will all be lies piled on top of lies.
Do I sound angry? Good, because I am. Agile is a con. People like agile because it absolves them of the responsibility to think and do their job. It makes my job a misery and it sucks money out of a clients pockets and to my mind is no better than theft.
If you think I'm wrong, prove it. And I mean PROVE AGILE WORKS, WITH EVIDENCE or shut up. Extraordinary claims require extraordinary evidence. I didn't come into your life and tell you how to suck eggs, but if you're going to come into mine and do that, you'd better have data to back you up.
The latter is riddled with "process" and is just as bad as waterfall.
The former is a hodgepodge of "whatever has in the past worked for this team and was flexible enough to keep everyone happy while letting us have a modicum of an idea what stage the project is in".
You want to work on the latter. Because it doesn't matter what you call your methodology, what matters is that it works and that issues users care about get fixed.
As opposed to throwing a hissy fit because I've not taken time out of my busy day to fill out forms in whatever third rate piece of junk project management "tool" they've adopted so they can just run a report and pretend they've done their job.
Ignoring that though, I'd say, if anything, Agile promotes using simple tools like wallboards (physical wallboards, with physical cards). I don't really think you can pin using a bad tool on the Agile community when they largely prefer not to use these tools in the first place.
I absolutely can pin it on the agile community, because if you take the Agile argument away from them, they're left defending a crappy tool with no arguments at all.
As a bug-tracker shoehorned into doing double duty as a Kanban board, does Jira have rough edges? Sure.
However, used as a slightly more feature-full variant of Trello, as a wall of virtual sticky-notes, as a way to help see if the team improves over time like it should, it's pretty useful.
The moment it becomes an impediment it, well, is an impediment, and should be dealt with accordingly.
They exist in all walks of life and all project processes. Agile is not immune from this, but nor it it any worse.
I would agree that anyone who blindly follows what someone else has written in a book without bothering to consider whether you really understand it, whether it actually makes sense, and whether it needs to be modified to suit your particular circumstances is clearly not much of a thinker.
But taking and developing ideas originated elsewhere doesn't seem an intrinsically bad thing to be doing.
In the early days (2004-2005) it started out as a good thing, young-developer me was so impressed by its "no-bullshit-only-programming" attitude that I even included it on my CV (in my defense I also had XML written in there).
I noticed it changing into a religion about two or three years later, when I had joined a new team and saw the members of said team following its "commandments" with no critical judgement whatsoever. The crazy rule of naming teams ("Wolves vs. Vampires" etc) didn't help it either.
More to the point, this blogpost is not like that, it only takes a few minutes to read and understand..
Thanks for sparking the debate. Allow me a request, a comment and a question in response...
Request: I ask you to distinguish Agile from Scrum. Agile is a set of principles. Scrum is a methodology, or process, based on those principles.
Comment: For my teams, Scrum has worked more effectively than other processes.* The point of my post was not that "Agile works, if only people would do it right". That is a misinterpretation. The point was that over-complication (by way of, for example, a 478 page book) makes having it be an effective process for a team more difficult. Perhaps the reason it's worked for my teams is because I have focused on keeping it simple.
Question: Obviously, you think Agile and Scrum are nonsense. I'm not going to try to convince you otherwise because I can think of lots of scenarios where Agile or Scrum would be counterproductive. But, I would like to learn from you: what approaches/processes/methods of developing software have been most successful for you.
Let me clarify...I'm not asking "what is better than Scrum?"...I'm asking, "what works best <i>for your teams</i>?" (And, since some processes are better for some jobs that others, what type of work is your team doing?)
Scott
* - Per this post, http://www.scottporad.com/2013/03/14/the-best-developer-team..., I write:
"The very first thought that comes to mind about a building a development department structure is this: an organizational structure is a just tool for getting a job done, so use the right tool for the job...
...My experience has almost entirely been in building early stage web sites which is a very specific type of job. I am strongly biased toward Agile/Scrum teams for this type of job. If you are building software for banks or airplanes you might need a different tool...er, structure."
My original comment wasn't an attack on your article specifically, but I appreciate it looks like that. However, my comment "it's the same nonsense in fewer words" is targeted squarely at the "if Agile doesn't work, you're not doing it right" mentality. And your article has one foot firmly in that camp.
On one hand, I can appreciate why people take that position because on paper, Agile looks brilliant. But as I mentioned in a previous comment, it doesn't matter that Agile was meant to be some other thing, because Agile has become what it is today, and it's this reality version of Agile that I have a serious problem with.
You can say Agile and Scrum are two different things but it doesn't matter. If you came to where I'm working right now and said that, you would be laughed out of the room. It doesn't matter if you're right. The reality of Agile is that Agile and Scrum are now the exact same thing in the eyes of the many. And the more you look, the more you'll see that, to these people, Agile means strict, regimented adherence to inflexible procedures whose supposed merits are totally baseless. And when you call people out on this shit, they'll just scream "Waterfall! Waterfall! WATERFALL!!" in your face until you shut up. (as an aside, that's pretty much what happened to me when I tried to point out that our approach to this project wasn't working, except they didn't scream it at me.)
You ask what approaches/methods/processes work best for me. I don't know how to answer that. I'm not comfortable with the idea that just because something worked well on one project, it will work well on an another.
Take for example one of the projects I'm most proud of. From the email you sent me earlier I know you've looked at my website so I'll point you towards the Info graphics work I did for that very famous newspaper. The first project we did for them was the "Health of England". The organisation approached The Agency at the 11th hour with a vauge notion about creating something interactive to support some of the articles for their soon to be released iPad app. That project was hellish. The iPad had yet to be released, all we knew was that we could put stuff in a UIWebView. I worked with the designer to cobble together a few prototypes of things that might work and we slowly and clumsily worked our way towards a finished product. There was a ton of waste and dead ends and revisiting things we'd already done. There were no meetings, just shouting at other people over the desks or standing behind someone and arguing the toss over tiny, seemingly inconsequential things. We worked late. The stress levels were through the roof and by the end we were exhausted. But we were all immensely proud of what we'd achieved, and that work went onto win two awards and generated a lot of interest form big clients in a little agency that had been flying under the radar for a long time.
So... should I use that as a template for future projects? Should I say that every project from now on should be completely unstructured? That the best results can only be obtained by working late and burning out the team? That we're not doing our best work if the designer and developer aren't ready to rip each others throats out?
That would be insane. And this is where the Agile people jump in and say "Ahah! But that was Agile, don't you see!!"
No. It wasn't. It's called "building the thing as best you can in the time you have available".
But the Agile people will say "But... but... that's Agile".
No, it isn't. Agile is* scrum to all intents and purposes. Agile is stories and story points on the mind numbingly whimsical fibonacci scale. Agile is not doing what you should be doing because some other idiot forgot about that bit and is covering their tracks by saying it's out of scope. Agile is sprints that dictate the thing be built ass backwards because the person planning the sprints sit's in an ivory tower pretending they know what they're doing. Agile is time boxing something that really has to be in there, but won't be because there's not enough time in the sprint. Agile is reaching the finish line with a third rate version of what you set out to build, and using it as validation that Agile works. Agile is about sacrificing the one thing you should never sacrifice if you want to hold your head up high as a designer or a developer: Quality.
I'm sure that will irk a lot of people, that I'm brazen enough to claim that Agile = Poor Quality. But I have the anecdotes to back me up. And in the Agile world, anecdotes are evidence. My anecdotes are as valid as yours. Live by the sword, die by the sword.
And maybe all the Agile supporters are right. Maybe Agile can be done right and it's brilliant when it is. But in my actual real life experience, all I've seen Agile do is validate low quality people's low quality ideas and bad practices. Agile is held up a shield, protecting the holder from professional criticism. Maybe the Agile community truly wanted to give the world a gift, but all it;'s given us is a disease.
They should slap those things on the grill!
But I think you're vastly underestimating how much of an effect a good process, or a bad process can have on a development team. The good process doesn't need to be textbook scrum (In fact, "textbook" scrum is usually not a great idea), but it's a bit laughable that you're criticizing Agile advocates for not having critical thinking, while in the same breath minimizing process to "Well, there's good people and bad people, and the good people will get things done better."
What do you think makes a good project manager a good product manager? Do they just magically add +5 to development speed to a project? Of course not, they build and adapt a process that makes sense for that team. That doesn't necessarily need to be Scrum, or even Agile, but in my experience those methodologies seem to work better. The only caveat is that they need to be built and adapted for a companies particular needs.
In regards to proving Agile works, I can give plenty of anecdotes:
I could talk about when my startup had pressure from upper management to commit to a fixed roadmap several months out, moving from our "Scrum-like" agile process to something much more akin to Waterfall. I can talk about how it led to overly defensive estimates, projects taking roughly double the time they used to, and inevitable "death marches" at the end of the project, when we were faced with the prospect of integrating huge projects into our main codebase, with the QA nightmare that resulted from that.
I could talk about how, in the many tech companies I've worked for, there's an almost perfect correlation between "agility" (which I'm defining as the ability to re-evaluate project plans and iteratively develop software) and quality, ease of deployment, developer productivity, and satisfaction with the dev team.
Finally, I can talk about a recent example. We recently experimented with moving to larger, pre-planned projects. One interesting thing about the way we did the transition is that we packaged up stories that we had already broken down into stories appropriate for Scrum, and moved them into a larger project. We ran two projects using this approach. One project that was estimated (in terms of story points) at about 1.5 weeks ended up taking 5 weeks to deploy. One project that was estimated around 5 weeks took 12. I wish I could give you more evidence, but this transition was such an abject failure that we moved back to Scrum before we did any more damage.
But they're still both turds, and you shouldn't eat them.
Agile does not have a monopoly on process. But Agile apologists repeatedly offer up that concept. That somehow, a world without Agile is a world without process. That's an absurd thing to think.
If you want process, tell me what you want to do first, and we'll work together from there. Agile supporters tell you how you're going to do something, then ask what you want to do, then tell you what you want to do is wrong.
As for what makes someone 'good' or 'bad', you offer up a straw man argument that tells me more about how you think than how I think.
And anecdotes aren't evidence. They're anecdotes. You may have had just as much success with those projects if you'd made everyone wear cowboy hats on Thursdays. You can't prove that you wouldn't, just as much as you can't prove that Agile works.
And your final point is ludicrous, bordering on delusional. If I understand you correctly, you estimated how long something would take to build using Agile methodologies, then moved that stuff into an environment where Agile wasn't implemented, and it took longer than you estimated.
All that tells me is:
- You're terrible at estimating
- or unable "re-evaluate project plans" (your own words)
- or both.
I get what you're saying about "tell me what you want to do first, and we'll work together from there". I mentioned a few times in my post that it's always important to not blindly implement Scrum, but to see what works well from you. But having Scrum defined is still useful for a lot of people, since:
- It gives you a starting point to work from. Every project is different, but there's certainly common elements to projects as well. It's a bit like design patterns in software - applied blindly and religiously, they're impractical and can lead to pitfalls, but that doesn't they can't serve a purpose.
- It takes a lot of time to learn what works and what doesn't in regards to project management. Telling new managers and developers to figure out what works for them with no guideposts is just going to lead to a whole lot of inefficiency as everyone stumbles towards a (mostly) shared goal.
In regards to projects taking longer under a waterfall approach vs. an agile approach, there's a lot of different reasons for this sort of thing happening.
- Working on small sprints (or whatever you want to call them) instead of month-long projects saves you a tremendous amount of integration time. Show me a project under active development that's existed outside the production branch of a codebase for more than a month that isn't going to be a nightmare to integrate.
- Forcing yourself to make decisions up front, especially when you have a strong product management component to your team, can result in drastically more time working on requirements instead of starting with some unknowns and letting them work themselves out.
- I can probably point to several bits of these projects that we would have avoided doing or would have done differently with Agile, but the requirements said otherwise. It's amazing how much power requirements documents have, and how much inertia they can add to a project.
- Stuff generally gets done close to the due date on a project. This is as close to a universal truth as I've ever seen in project management. By keeping everything in tight iterations, you can control the flow somewhat by putting pressure to complete releasable software every week, rather than one massive due date that everyone scrambles for at the end of a project.
- Being able to "re-evaluate project plans" is a hallmark of Agile, not Waterfall. I agree, being able to re-evaluate and iteratively arrive at a solution outside of rigourously documenting things up front would make things a lot more efficient. That, far more than planning meetings and retrospectives, is what Agile is all about, so I'm not sure why you're so intent on trashing it.
I'm not sure why my example is ludicrous. If the process used doesn't have any effect on how long it takes to develop a project, what's the point of any process in the first place? Of course using different methodologies will have different results - even if you aren't an Agile fan, that's completely obvious.
The only credible evidence I can find about Agile's supposed benefits is the Voke report, which warns against using agile. With only 28% of participant reporting success with an Agile approach. That's bad. Plain old fashioned bad.
It's like I'm debating a creationist. Your logic is circular. You want me to believe the Agile works, and you're more than happy to tell me how wrong I am, but you shoulder none of the responsibility to prove your claims. At best, you're suffering from confirmation bias, but I think that's being too generous.
And stop talking about waterfall. It's a false dichotomy. You're the only one making that comparison.
Your argument, as far as I can tell is two fold.
Some process is better than no process: Well I think every sane person would agree with you there.
Agile is process, everything else is no process: Wait... what? How do you keep on arriving at that conclusion over and over again? And don't even think about saying "waterfall" because I swear to god, I'll punch my computer so hard it'll knock Google off the internet.
Find me evidence as compelling and credible as the Voke report, supporting agile, and we'll talk sensibly. But I'll go out on a limb and say you can't, and you won't. Because if I were to draw on my anecdotes and present them as evidence, Agile is universally bad and dooms everything it touches to failure or 'third-rateness'. That's all the experience I have to draw on.
And I suppose that's my biggest bug bear about Agile. Teams always finish things wether some fool is ramming agile down their throats or not. There's no other alternative. But Agile takes great ideas, shatters them into a million pieces and churns third rate approximations of that good idea out the other end. Without Agile, I've seen projects that were horribly mis-managed but the output was still great and once all the screaming was over, I was proud of my work. There's something about Agile that makes for crap products, and I feel so ashamed of the output that I wont even put my name to it.
As far as I've been able to ascertain, in practice, "Agile" means "not waterfall". Fullstop. As in, any process that isn't waterfall, is called "an Agile process."
You can be "agile" by just goofing around with no big plan; if the work gets done, you can look back at the points where you made little plans, and say "well, see, we were doing Agile." Any [successful] process other than waterfall, when post-hoc analysis is applied to it, will look like the classic definiton of an "agile" process, no matter whether you were trying to execute capital-A-Agile at the time!
And I'm not saying that in the "the Agile people are right" sense. I'm saying it in the vacuous sense: that the term "Agile" is meaningless other than in that it means "not waterfall." Waterfall is doing design once, at the start of the process. If you ever do design again at any point, ever break your design up into pieces and postpone some of them, or design this feature at a separate point than that feature--then you're not doing waterfall. And so, the Agile people will say, you're "doing Agile." That's it. That's all they mean. You've designed more than once.
And your response should be "...so what?"
(But it should also be, in a more diplomatic sense, "well, then we're all doing Agile already anyway, aren't we? You've long won this Quixotic crusade against the Waterfall windmill; we all agree that 'Agile' (to this vacuous definition) is the right thing to do. Now let's all go out for a pint.")
Too bad you haven't made it work for you, maybe it is simply not for you.
Is it better than waterfall? Maybe... but Agile is not the only alternative to the classic waterfall model. There were several iterative software building models long before 'Agile'.
Maybe other people are abusing Agile and missing the point entirely, but look at the literature! The seminars! The blog posts and endless stream of beady eyes consultants. It's all self help guru nonsense and it doesn't matter that Agile was meant to be some other thing, this is what it has become. What I call Agile matches reality more closely than what you call Agile.
That’s exactly what the Agile Manifesto suggests:
- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a planThe first and most obvious failure is that most consultants sell a process and offer little to improve the "individuals and interactions" part. In fact to the contrary I find these contractors spend a lot of effort making "Agile" palatable to managers and upper management. They try to create tons of layers of bullshit metrics for them to be happy (or unhappy) about. Instead of fixing dysfunctional management and team structures they reinforce them. The roles have new names but everyone just falls back into their old behaviours. Nothing really changes we just have new meetings and new buzzwords.
Agile is like communism in that way. On paper, it's great! In real life, everyone is miserable and everything is falling to bits and there's some other guy enjoying the fruits of your labour.
Too many people try to attribute the principles of the agile manifesto to the practices of "Agile". I don't see the relationship most of the time.
I’d consider that OK, it’s how software development works. I don’t have that much experience to make general judgements, but in my opinion agile does not try to offer you a recipe for a successful software project. Because such recipe simply does not exist. Agile just tries to guide you away from some obvious and tried wrong turns (like relying on processes too much). The rest is up to your team and your customer, and there are bound to be both successes and failures.
Scrum is scrum, if you like it fine, I don't. XP is XP, and so on. Each does in fact try to offer up a recipe for successful software projects. But there is no such thing as an "Agile" process. So, try to embody the principles and it may help you to be a better, smarter software developer, but even if you do it may be of little to help to cure a dysfunctional team or business.
Best of luck.
Unfortunately, you are unlikely to find such information in most of the literature around Agile methodologies, blogs or the army of consultants that roam the earth much less your average Scrum Master. Cargo-culting is rife in most the companies I have seen, even where the adoption of agile principles and methodologies have a significant positive effect on a team's effectiveness. I'd have to classify myself as a cargo-culter in the beginning too when I first exposed to working in Agile team taking an eXtreme Programming approach some years ago.
Unfortunately, herein lies the problem. For many people it does work (whether you like it or not) and as you say, it takes on the characteristics of a religion - people experience "better" and are just happy to keep following a recipe for "success", without understanding WHY. Obviously, this leads to people trying to replicate such "success" in a different context and when things go wrong they don't know how to fix it - they don't have a deep enough understanding of how to re-apply the principles and and normally they are optimising for the WRONG thing.
There are valid proven theories behind why working in iterative cycles, limiting work-in-process, shipping working code in the production regularly and self-organising teams lead to better results - you just won't hear why from most of the Agile evangelists in this world but there are answers out there if you look for them. Instead your typical Scrum Masters obsess about far more trivial and unimportant things like how to phrase the title of a story card, retrospective formats, burn down charts and velocity points.
If you can be bothered to look a bit deeper, then just avoid using the "A" word in your searches. ;-)
Here is one book, if you can get past the rather dry and academic natural to get you started. http://www.amazon.com/Managing-Design-Factory-Donald-Reinert...
P.S. If you want a really good laugh, do a bit of research on how "Waterfall" came to be and how ridiculous the whole Waterfall vs Agile thing really is.
One thing has nothing to do with another. If you want to kill bacteria, bleach is pretty effective.
If you want to kill a person, a bullet does a good job of that.
Dipping bullets in bleach doesn't make either of those things do either of their jobs better. And it doesn't make either of them any better at killing flies.
All it takes, as others have said, is that by applying critical thinking and with armed with the knowledge of the theories I mentioned you will quickly arrive at some of the principles/processes/techniques you'll find in Agile.
The fact that the word Agile itself has become a poisoned, meaningless word, adopted by the same people that used to peddle previous software process fads (and the consultancy and services that naturally go with it) doesn't mean that there are some good ideas that really do work.
What kind of evidence would count - for you?
Off the top of my head though, how about having agile and non-agile teams complete a project from the same designs, with amends and feature requests introduced at scheduled intervals. Study the results. I'm sure someone with a background in constructing these kinds of studies could work out the kinks and make sure it's a fair comparison.
It's unfeasible because you'd need a large sample and those kind of people aren't likely to take two months out of their life to work for free on a science project. And I guess that's why Agile goes largely unchallenged. No one can afford to do the science to prove wether or not it works.
And the ones with evidence and experience proving that it doesn't work are people like me who have to put up with being told I'm wrong, and I'm unlucky and it's all somehow my fault that I can't reproduce the results. It's like everyone is claiming they've cracked cold fusion, and it's my fault, not theirs, that I can't detect the neutrinos.
Well - what's your evidence then? I am genuinely interested.
Because - anecdotally - the most effective teams I've worked on or observed over the last 20 odd years have been the ones that are doing more "agile" things.
There's a fair amount of survey research (e.g. http://www.agilemodeling.com/essays/proof.htm) that seems to show agile teams are more effective. I've seen very little research that implies otherwise.
So I'm obviously going to go with my experiences on what works and the research that I've seen. You are, of course, free to do stuff based on your experiences. But I don't see why your anecdotes beat mine ;-)
As for the evidence that people are publishing (such as the stuff linked to in that proof.html link), that's no evidence at all. It's cherry picking. It's saying, "Pair programming worked well in this controlled study, and XP worked well in this controlled study, so putting the two together is guaranteed to work together flawlessly and there can be no unintended consequences". The evidence ignores more than it acknowledges.
In addition, when a successful project launches that used Agile, it is used as evidence that Agile works. But when a project that used Agile turns out to be an unmitigated disaster (Such as Accenture's NHS IT project to pick one cataclysmic example) the Agile cheerleaders are no where to be seen.
When Agile goes wrong, it goes spectacularly wrong. Take that NHS example I mentioned. 10 years in the making, £5bn of tax payers money, it's still not finished and Accenture plain up and walked away from it. Find me an anecdote better than that in support of Agile.
But of course you wont. You will say "well they were doing Agile wrong" and you will have no evidence for that.
Curiously enough - I won't say that. I will point out that Agile is only just a bit over ten years old now, so a ten year £5billion agile project sounds... odd... to me. It certainly sounds atypical for projects in general.
But yes - agile projects fail. There is no system that will cause all projects to succeed. There is - as they say - no silver bullet. No argument from me there at all.
Let's forget the "agile" word for a bit.
How do I go about getting better at building software and building teams that make software?
Personally, I do things like:
* I look at research
* I look at successful teams I've worked on / observed and try out some of the things they do
* I look at failed teams and try and avoid doing some of the things they do
* I look at the stuff I try and see whether it works or not.
* I measure stuff - tweak what works, avoid what fails
* Repeat.
That kind of process, for me, has led me to a chunk of techniques that broadly sit under the lean/agile label. I'm not going off and going "yay agile". When I first encountered XP I was hugely sceptical - I really couldn't see how it could work effectively at all. But I tried some bits of it, and observed some other folk playing with other bits of it, and saw that it seemed to improve things, so tried some other stuff... and so on.
I'm not trying to say agile solves all problems. It's not a silver bullet. It's really freaking hard to impossible to use it in some contexts. But in the last ten years my personal experience it's helped me and my teams get better and building stuff. The experiences of a stack of people I've worked with who I respect, and the experiences of a stack of other people I've talked to and listened to and visited have been similar.
I'd like to think that we're not all complete idiots ;-)
So - if stuff fails I love to poke at it. When agile projects and when non-agile projects fail I poke and try and learn from it. Generally - I think I've got better at delivering stuff that makes my clients happy during my career. A stack of the skills that have helped me do that in the last ten years seem to have come from the agile community. Take from that what you will.
You think because the awful projects you worked on called themselves Agile, that Agile is bad? I hate to break it to you, but you are not special enough for your personal experiences to define "Agile" for the rest of the world.
I learned about Agile, Scrum, XP, Kanban, and project management in general through a self-motivated quest. I didn't accept what I read until it was corroborated. I sought the authoritative texts (original works) and the underlying principles of what I read.
Least of all did I take at face value the claims of the clearly incompetent that happened to be physically in my presence.
Let's put this part to test. I hereby claim that I can teach you how to transmute lead to gold by waving your hands. I'll teach you the exact movements. Then someone can hand you a chunk of lead, you'll wave your hands and observe the results of the process I taught you. It's your responsibility not to expect explanations of observed inconsistencies -- between gold and what you have in your hands -- to be spoon-fed to you.
If we really did the above exercise, your conclusion would be that my claim is false. We could go even further than that. I could claim circumstances were not right. I could accuse you on basing your conclusion on anecdotal information. But somehow I doubt I could convince you or anyone else that I'm not a charlatan.
Ah, but I can't produce "authoritative texts (original works)" that reasonably and convincingly explain "the underlying principles" of transmuting lead to gold through hand-waving, could I?
Okay, no problem, let's change the example to "getting laid through neuro-linguistic programming". There are people who believe in that pseudo-scientific bullshit even today, so long after it's been proven to be total crap.
Do you honestly think that you could apply your argument about "not expecting to be spoon-fed explanations of observed inconsistencies" to NLP without being laughed at? I'll assume you don't and ask you why the heck you tried to pull the same argument when it comes to "Agile"?
My point was that, given experiences that do not align with prior knowledge, it's no one's responsibility but their own to understand the basis of that misalignment. They can take a shortcut and jump to a conclusion, or they can take a longer, costlier route and seek more information.
The first part of your comment supports me. I'm not sure what exactly about NLP (which I'd never heard of before your comment, and just skimmed on Wikipedia, so I might be missing the cultural context you are referring to) that I should apply the "argument" that one is responsible for his own learning, and that jumping to conclusions (as I characterize the OP to have done, by asserting that the 5 "Agile" projects he worked on must, for some reason, be representative of Agile) will leave you frustrated in the long-term.
I understand there is a social aspect to learning, and that we can all be victims of herd mentality. The question is do we interpret such a situation as one of abuse (where we look for fault), or as one of misunderstanding (where we look for deeper understanding).
I couldn't have put it better myself.
It shows that agile reduced the shipping duration of projects from 24-42 months to around 4 months. During that time Citrix Online's market share increased.
It's going to be hard to get the side-by-side comparisons you seek, and, sure, it could be that the agile-focused management happened to also be competent (hey, I'll accept that. :)). However, I actually think Scrum specifically is a science that measures production, and that its outcomes are usually pretty good.
Since I've now helped convert a couple of huge enterprises to Scrum, and since I love tracking data about those conversions (and it's good), I make money in this field. So you could say I'm a flogger of the religion. At the same time, I have a computer science PhD and I could be out wrangling code or running a startup.
And you are right, there are a bunch of religious zealouts out there. They annoy me too.
Dan Greening dan@senexrex.com http://senexrex.com
I think this article gives you a good start with creating a decent development process for your company.
At the end of the day successful projects happen because of good communication, clear requirements and hard work. No amounts of tricks can save you if you don't have all three.
For example, "we know agile better than the project managers" often comes from people who find themselves in yet another cargo-cult agile environment with a Scrum master who not only doesn't have a clue about Scrum, but is also pretty much convinced that it's a wrong idea and therefore doesn't give a shit about what he's doing and whether the latest process-related brainwave coming from above is one step forward or -- more likely -- two step backwards. In such circumstances, you really do know more about agile than managers when you 1) say that it defeats the purpose of Scrum to have people filling out how many hours they've spent working on an item alongside how many they have left, 2) point out the inefficiency and banality of hour-long discussions on whether we move the user story from one sprint into the next one or create a copy of it, 3) complain about having 1-2 hours of sprint planning, a 1-hour mid-sprint review meeting and 1-2 hours of sprint review on top of 30-minute daily meetings in a 2-week sprint and 4) wonder why nobody seems to have read the bloody agile manifesto itself.
So when you hear "if only we were truly agile" coming from a dev, maybe it's not an incompetent egomaniac on a power trip looking to shift the blame for the things that don't work. Maybe it's someone like me who is sick to the core of managers who can't hear you because their head is up their ass -- yet somehow there's always enough place in their for yet another "Agile" consultant's shaft to get cozy with.
Yes, I'm bitter. That doesn't make me wrong.
Because the vast majority of teams that I encounter that label themselves as "doing Scrum" are not even vaguely close to actually doing it.
Fundamentally, scrum is a recognition of certain risk-bearing concerns in projects, and the assignment of those separate concerns to specific roles. The division of concerns sets up a structural tension that drives attention to risks and which are necessarily resolved through open negotiation. To truly reap the benefits of Scrum is to appreciate the value of this tension.
All the steps listed fall from this basic understanding.
"What are you going to commit to delivering in this project?"
"points This stuff is pretty much guaranteed, unless you change your minds on the priorities."
"But that's only about 40% of the dev time we've got until our planned release date[1]."
"Yes, that's all we can commit to definitely doing. They'll be more, don't worry, but what depends on what the priorities become at a later date and where we get led with customer feedback[2]."
"But I can't go to my management team with this, they won't understand why we're not committing to much!"
"They should understand, they told us to do Agile in the first place[3]"
Project starts...
"Can I get copies of the detailed design docs?"
"We don't have any detailed design docs, you can have a look at the design docs we've got but just remember that they're 'Barely Good Enough'."
"But I can't go to...[4]"
1. Ok, we can handle an upfront release date.
2. Ha, like we'll be able to get any of that.
3. With little or no actual understanding of how it works. But it's a nice buzzword to spray around with customers...
4. etc, etc, etc
A medium bug appears in your production environment. Will your customer really wait till the next sprint to get it prioritized, then fixed?
Or, your customers expect functionality, time, and dollar estimates in a proposal before they start a project. Sure, some teams think they can do scrum under the covers, and give them what they really want, but in the end do you have the right language in your contract to address changes made during the project? Will the customer really care that feature X didn't make it because it wasn't prioritized high enough at the last grooming?
Until you get this buy-in from the people paying you, there is a huge conflict about how you want to do things vs. what the customer expects. And generally, they'll win until you convince them. But why should they change? They're happy with how things are going the way it is.
Wade's lightweight process didn't add the bullshit layers of Scrum's painful "retrospectives" and standups. Scott's approach may be "Scrum lite" but it still isn't helpful or productive. Standups are painful and often about as effective as asking your kids how their day at school went. Retrospectives are practically like water-boarding your entire team once a week.
Didn't get everything you planned done this week? Welcome to the real world, get used to it and figure out how to get the most important things delivered instead. Teams need ownership and accountability not babysitting and handholding.
In that way, Agile is purposely vague and ambiguous and it's hard to really say Agile doesn't work for you. What you can say is those aren't your values. There's nothing needed to be proven that way, it's just a disagreement based on what you value.
The agile methodologies imho are comming from someone that had luck and instead of winning 7 on national lottery hired competent and motivated people by sheer luck and due to his own self centered line of thought (egocentrism, etc), he thought his "inovative" way, brought in the success and started to market it.
The mgmt is still far too stupid to realize that you cant streamline software development, you cant get revolutionary ideas on requirement and you cant JUSTIFY THEIR POSITIONS!
(oh and btw, my cv consist of 18+ years of c/c++ software developent from world largest IT companies to startups and i think i have seen all the crap the white collars can offer, they just cant comprehend that the creativitye cant be streamlined and are always searching for a silver bullet to pay less and get the breakthrough technology for peanuts)
If you do a Product you want fixed budget, fixed releases and flexible scope, scrum/agile is the best fit.
If you do Projects you want fixed release and fixed scope and flexible budget. Use waterfall and don't close the budget until the end of the design phase.
Of course, this definition was imbued on me many many moons ago before the consultants hijacked the language.