We've invented waterfall
epicenterconsulting.com
epicenterconsulting.com
So here's what you should do. Over the shoulder review of a representative employee of each department. HR, AP, AR etc. Actively watch and understand the processes they go through. FOR AN ENTIRE 8 HOUR SHIFT.
No, seriously.
Now, you've got a good idea of how its done current.
Then do the stake holder interview to find out what they want to do differently.
Then plan, develop and deploy your software.
I seriously don't get why despite there being a recognized need for understanding the problem scope which everyone one will invest hours in meetings and conference calls talking about, no one actually invests any (significant) time doing, or reviewing the actual process. It's the fastest most efficient way to understand the problem. It would totally diminish months of back and forth "here's what you asked for" "no, that's not what I asked for" development review process that almost _always_ occurs with consultant projects.
/rant
I did this on a project a few years back. I replaced a paper workflow process that was taking up two people each in three departments with a web-based workflow that increased visibility, dropped turn-around time from days to minutes, increased accountability and accuracy and trimmed those 16 person hours of processing down to 1-2 per department.
Everyone who directly interacted with the new system loved it. Numerous edge cases that would have been lost in high-level review were caught and integrated from day 1 due to my actually watching people do the job for a day or two per department. The solution has been rock solid (minor maintenance only) for five years.
And I almost lost the job.
The people who sign the checks were furious. The balance of political power between departments were thrown for a loop. One head in particular treated the thing as a near-existential threat. His entire concept of his job revolved around being the authoritative interface for retrieving and maintaining pieces of data that were no longer exclusively under his control. Another flipped out because middle management saw the results as cause to reduce his headcount and budget, and thus importance.
These two departments fought for months, refusing to contribute their shares of budget that were pledged toward modernizing this system.
On a technical and practical level, it was the single best experience I've ever had as a consultant. On a personal and economic level, is was one of the worst. It was some of the hardest money I've ever tried to collect. It was some of the most time and energy I've put into the political and 'sales' side of a job (the part I treat as a necessary evil, but very much evil). The corporation has made out like a bandit in the long run. But I paid the price.
It's simply too easy and financially rewarding to allow a client's political nonsense to screw up every stage of a project. I have less stress, the people who pay me are happier and I bill far more hours.
As with most software, internally developed software included, you don't see better projects more often because the incentives are horribly perverted and stacked against it.
Why? The company had a long-standing policy of manually setting speed and duplex to ensure that there were no duplex mismatches. The was a team of 6-7 people who would audit servers and check switchports, and they produced a report was issued every week, and reviewed by the CIO and other luminaries. People would be shamed for errant configurations, as only the auto-negotiation team could configure NICs.
The director who controlled the auto-negotiate squad was politically powerful, and got all sorts of power out of it -- including banning GigE. As late as 2010, this org was deploying dozens of 100MB NICs to ESX hosts to get sufficient bandwidth. They also purchased 100MB NICs for desktops.
But the technical problem wasn't the problem -- the issue was that they created an inquisition that had outlived its purpose.
Besides, it shouldn't cause cascading failure so the worst case scenario is a tech call, the same as for an unplugged cable.
Forcing duplex and speed with ethtool is still relevant.
"It is difficult to get a man to understand something when his salary depends on his not understanding it." (from I, Candidate for Governor: And How I Got Licked)
The developers often get blamed for "not understanding how real people use the system". Most features are requested by C level people who have never actually used the system.
While consulting, I've actually been told NOT to talk to the users who will be using the system. In a large enterprise it is often about power and politics, it is rarely about building value and increasing productivity.
Vote with your labor, work for a small company.
The other side of the story is that we ran into huge political problems. Every day, someone was trying to kill our team because we were making the traditional IT groups look bad with our turnaround time and solutions that worked. On the other side, we were automating things that people didn't want automated. If we didn't have support from the CEO down through a specific chain of command, we would have been killed within months, if not weeks. We didn't even call ourselves developers. Our official position title was business analyst because then IT would bug us less. We weren't even part of the official IT department because those guys couldn't get outside the box. So we were just a small group of business analysts who tried to hide until we were too successful to hide anymore. That's when the attacks really started coming. And then we continued to be successful, and then they started trying to copy us instead. Last I heard, that team's unfortunately been swallowed up by the bureaucracy now, and is a shadow of what it once was.
Fascinating experience.
edit: added end of the story
This is why I'm far happier now consulting to startups than to large corporations or the govt.
Ultimately, it comes to corporate Darwinism... the companies that evolve reasonable processes prevail. Others will only leave a fossil record. Unfortunately, we have governments as well as corporations.
This is now one of my favorite comments from this site. Very well put!
Are you a consultant because you want to earn a big paycheck without much stress or because you want to make teams better? For me, the latter comes first.
I even go so far as to tell the client when I believe that I can no longer add value equivalent to my cost: when they no longer need me. My job is, ultimately, to become superfluous: to make them better then get out of the way.
Now if I was engaged because of that delusional thinking, that's another problem...
I admit to liking waterfall myself (it suits the kind of work I do) and if well done by competent people and for certain types of project its the way to go ...
It sounds dumb at first but it makes perfect sense. What you did at that place was to apply a bandaid to a fracture where it really should have been a cast. But you didn't know it was a fracture because you didn't look past the swelling to the root cause of the pain. This is nothing new, consultants have been loathed for this very reason for a long time now.
That was me telling you how to do your job. If you wanna know more, google it. If ya got offended, have a nice day!
And, for what it's worth, the C-level types loved the project too. What I did was replace a wooden peg with a modern prosthetic and upset the guy who was selling wood oil for conditioning the peg and the guy who sold the leather straps [1].
edit: That said, sure, it'd have been less stressful if I worked in the politics up-front. I said as much. But I've done it that way enough to know it would inevitably have compromised the system. And my post was a response to "why aren't the systems better?".
[1] and him, only because he couldn't see past the loss of the peg-strap business to see the gain of business making, maintaining and adjusting straps for the modern prosthetic.
Everything was couched in "We're really happy. We're not sure why Bob isn't. But we'd really like you work with him to sort that out."
But thats the thing. You should count politics as part of your requirements too, or thats what the theory states anyway.
For example, lets say you digitized an inventory form for a sales dept of a plastic company. By doing this, you effectively removed use of a pen from the 'system'. Now what politics could you possibly be affecting by eliminating use of a pen? Well, you could cause an uproar between the finance dept and the sales dept. Finance says that the organization's major stock holder is a pen company and the sales department must use pens (for whatever reason...lets pick advertising and promotion).
Now your system made the world efficient but it failed to account the current work practices of your client. You, sir, just screwed up.
Ever read Stuart Sutherland's Irrationality?
on the other hand, sometimes people need a good kick in the ass, so who the hell knows!
Interesting stuff.
Upvoted.
Yeah yeah yeah...but I did come bearing gifts!
Call me bias, but a lot of smaller companies overlook tried and tested standards. Proper requirement elicitation was documented in IEEE Computer Society SWEBOK prior to 2004 and even that is a summary of important knowledge to Software Engineering profession. Why is it that when a new company comes along with a truly innovative concept they feel they can innovate the rest of the process as well?
It made the initial sales much harder, but the results are much stronger and we've gained such a strong reputation that the software almost sells itself.
We're designers.
And we do have & run consultancies. Places like IDEO, Frog Design, Adaptive Path, RED, Doblin (Part of the Monitor group now), Gravity Tank, Teague -- and many, many more.
User centered design processes should be backed up by rich user stories, that come from real user experiences. The only way to do that is to go out in the field and get exposure to the who, what, when, why and how of what the users are doing, want to do/need to do and how they use your product/service.
And if you do it right, you can make a much more successful product that sells a lot better while actually doing some good for the users.
Not a bad line of work, IMHO.
After that we tried to reconcile the differences and usually made the in-software process a mixture of the two (where possible)
To the site's point (or missing thereof), the notion of "don't start coding until the analysis is complete" is a gross misunderstanding that coding IS analysis.
The inanity of reinventing the waterfall model is obvious to HN readers.
Every project is different. We should try and not be too dogmatic about these things.
I think a workflow approach is a better way to go for the client, but even our approach was inspired by other companies who have embraced a transparent workflow methodology such as this.
It's painful to see a company claim it as a first.
An average project experience around 25% change in requirements. Hence waterfall is highly unsuited for the majority of projects, since it does not handle requirement changes well.
However, in a few areas such as aeroplane control software, space shuttle software and similar, waterfall is indeed the optimal development process.
Different projects DO require different development processes. That's not wishy washy, it's the reality of software engineering.
Mindlessly applying your favourite development process to all projects, be it waterfall, scrum, evolutionary prototyping, spiral development or what have you, will very likely result in failure, if the process does not fit the project.
It's wrong to say that the paper is a description of how not to do software development. Rather, it's suggestions for a set of improvements to the process, while retaining the sequential nature. In the paper, Royce considers waterfall to be a good process, but with flaws which result in high risks. These flaws is what he addresses in his paper.
His suggested approach is derived from pure waterfall, and is very much similar. It falls into the "modified waterfall" category of life cycle models, together with a group of other similarly waterfall-derived models.
His proposed solutions that you call "modified waterfall" are modified quite a bit. (As an aside, he never used the word "waterfall.")
The paper is here: http://www.cs.umd.edu/class/spring2003/cmsc838p/Process/wate...
The quote "is risky and invites failure" is correct. However, another quote, even on the very same page:
"However, I believe the illustrated approach to be fundamentally sound. The remainder of this discussion presents five additional features that must be added to this basic approach to eliminate most of the development risks."
These suggestions are to 1) Add an additional (preliminary) design phase, after requirement analysis. 2) Much greater documentation. 3) Perform the entire sequential process twice (not x amount of iterations, just two times). 4) Additional activities and documentation during testing phase. 5) Involving the customer.
His diagrams require close scrutiny, as on first inspection it seems the process is markedly different, when it is really not. It is still an entirely sequential waterfall process, just with an extra phase (preliminary design).
If one is interested in software development methodology, I strongly recommend taking 15 minutes to read the actual "waterfall" paper[1].
Poor guy, Winston Royce is one of the most misunderstood folks in the industry.
(By the way, curiously enough, Royce's son William was a major contributor to the Rational Unified Process.)
[1] http://www.cs.umd.edu/class/spring2003/cmsc838p/Process/wate...
The graphic where he shows cycles going back stages is used to exemplify failure, which he proposes to fix via the strict practices above.
Just in case someone from the future misunderstands this: I don't mean that agile drops analysis, documentation, planning or monitoring (they are essential to it) but it leaves the "intensive" and "before" parts out.
Not everyone drank the agile koolaid.
My experience is that for the majority of software projects, this assumption is horribly broken. And even in fairly well known problem domains where it might work, for the majority of business users wanting new software, the ability to communicate every detail of what it is that they want is horribly lacking.
Scope creep, architecture problems, unanticipated requirements etc. can and does break waterfall projects; this was well known 10 years ago. The probably of it going wrong rises with the size of the project, and it's a curve that's well above linear.
There are two ways out: You can plan for change or you can make the penalties against change more severe - roughly, do even even more big upfront design and penalise change requests; or do less of upfront design, iterate and embrace change. In many cases the "big upfront design" way is impractical in time and expense.
Edit: I found the old article on this subject: "Theory P and Theory D of software construction: Which theory fits the evidence?" http://weblog.raganwald.com/2007/06/which-theory-first-evide...
This was well known 40 years ago:
The most deadly thing in software is the concept, which almost universally seems to be followed, that you are going to specify what you are going to do, and then do it. And that is where most of our troubles come from. The projects that are called successful, have met their specifications. But those specifications were based upon the designers’ ignorance before they started the job.
--Douglas Ross, 1968 (http://homepages.cs.ncl.ac.uk/brian.randell/NATO/nato1968.PD..., p. 32)
I also have the choice to ignore all the methodology snake oil salesmen and actually write software. Every "methodology" is bullshit. The new ones are not any less bullshit than the old ones. The fact that your response assumes I must love the old version of "follow our stupid manager nonsense" is really pretty sad. You honestly can't even imagine a world where I might think all of the bullshit is equally worthless? I have to be an adherent to one of these religions?
Books and seminars, we got these, but good agile is done by adding in some open source software tools, a whiteboard, pens and cards, and people with enthusiasm and an open mind. The last one is most important.
If we're still talking about the original post, then absolutely it's a belief system. Not a very well thought through one, though.
Pure[1] waterfall is doomed from theoretical point of view. You cannot control a system without a feedback loop, unless you have a 100% accurate knowledge of system's state and future evolution. Waterfall lacks feedback loop => you can't really expect much from it.
[1] - in real life there's always some kind of feedback loop in the process; I'd argue that the primary reason why Agile methodologies seem to work better is that they have a much, much tighter feedback loop than waterfall.
Whether or not one drinks the koolaid doesn't matter much.
15 years ago, 1-3 year release cycles were the norm. Now they're uncommon, and becoming less and less so. Some well-regarded shops (e.g., Etsy) now release many times per day. This isn't driven by what process is fashionable; it's happening because of improvements in technology and the way the Internet has shifted everybody's expectations.
Whether you're officially using some Agile(tm) method or not, you'll have a lot in common with those folks if you meet user demand for faster delivery. There are some niches that are lagging on this trend, but my pals in those areas still feel the pressure. They'll still be able to be waterfall-ish for a while longer, but I doubt it will be forever.
But imagine "agile" development for commercial airliner flight control software.
I can be done; you have to optimize for what's important to the project at hand.
They designed educational games for early elementary education. Every game activity was hashed out with a working committee consisting of an educator, a technical writer, an artist, and a programmer. They'd take a few hours and talk about the concept that the kid playing the game was supposed to learn (say, measures of weight), then they'd brainstorm ideas for potentially entertaining scenarios in which the concept could be explored (say, a flea circus), then they'd start to storyboard various activities and interactions.
Then the tech writer would produce notes, people would review them, and 2-3 weeks later, they'd do it again, refining concepts and interactions.
This process would stretch out over months, and by the time it was finished, the tech writer's notes essentially comprised a complete state machine for the activity described in English on paper. And the artists/media people had the assets for you in the right media formats. Usually sensibly named. Programming was generally pretty straightforward.
You know how a lot of the time that non-technical "idea guys" seem to have this terrible attitude? "Hey, I've already done the hard work of coming up with the idea, all you need to do is translate it into code."
This was one of those rare situations where it was often more or less true. But it was because the people coming up with the concepts were really part of a creative and disciplined process where they had to come to grips with the details in a way that usually only programmers have to.
Oh, and after things were done with programming, they had a QA team compare it against the spec and an educator's review.
They produced a pretty good product. Sometimes I think it's a shame I didn't stay longer.
As a teacher myself I do often incorporate a game (computer based or actual paper, dice, pieces), an interactive Web site, a video or other objects into a lesson. I also direct students to Web based interactive content as part of homework along with 'traditional' work.
In effect, my lesson planning is the 'interface' between the (no doubt well coded and engagingly designed) objects your team designed.
I suspect, from my own interactions with the business applications used in the largish educational institutions that I work in, that it might be harder to use this model when the interactions between the individual activities get complex...
While it fails sometimes, there are plenty of success stories about the waterfall model, like it or not.
Then again, I doubt these guys do this since it's all marketing.
Some people think of "agile" as just going through their full waterfall process every month or every two weeks.
Other people insist the core of "agile" is TDD, others claim nonsense, TDD is just as much part of "waterfall", and yet others don't see any point in TDD for their "agile" process.
tl;dr:
It's all bullshit, all the way down.
One can only imagine what actually happens in communication once they have a client. If the marketing is a complete lie, then I can't imagine the company itself can be trusted.
They claim to have developed the best process, just like all the agile con artists out there.
I had myself a nice little nerd slap fight on twitter earlier today about this post. "Agile" without up front planning and research is just dangerous.
So please, save your criticism that you know what is and clearly I don't. You can't define it, and if you think you can I know you don't follow it.
No, I agree, dammit! So take that!
Although I can see where you may have misread my honest question as rhetoric to indicate conflict. That was, as I said, an honest question--though perhaps missing a few words. I was inquiring whether the "agile" consultants you've known start coding on a new project from day one of the project indicating that, in those cases, they aren't taking much time at allfor research or planning.
Agile is a broad label that doesn't make a lot of sense. I've been in the biz 16 years. If there's anything that I preach it's "it depends" and "YMMV". Please don't assume we're all out to get you or that we all believe that we know better than everyone else.
The long and the short of it for me is this: There's two ways to solve problems. Agile helps immensely when you are figuring out how the solution would work. When you're solving a problem that might be solved in manual processes largely, Waterfall can help a lot, but not alone.
For unknown solutions, agile gives you a lot of iterative benefits.
For known solutions, waterfall with some agile feedback loops gives you incremental benefits.
What's the difference?
Sometimes, in a business, they have a great process already. It's in paper, or multiple systems, but they have their way of doing things that works. I believe it's called their competitive advantage.
Othertimes, a business is facing a problem for the first time, and wonder "how do we solve this". In this case, doing a big upfront design is a waste of time, starting small and staying flexible is most important.
For me, this mindset of doing both is critical, not one or the other at all times. You simply miss out on too much if you don't use agile, and you run around too much if you ignore waterfall when a solution exists.
Waterfall works where it's a known process, like building a house. You can have agile loops in there when building a house and improve it as you go.
When you're trying to build a new building that's never been built before, an iterative, feedback-centric approach like Agile is a lot more beneficial.
If a company focuses on taking something like Waterfall and introducing feedback loops (agile) like this epicenter thing, is it not a bit clearer than the black-box voodoo out there? Maybe it's just me, but I applaud anyone trying to make software easier. It just shouldn't be a dogmatic, or religious dismissal.
0.02.
Step 2: "we design an interactive prototype" allows them to get the feedback that really makes more agile methods work. You don't need to constantly iterate during development if you iterate properly during prototyping. I certainly wouldn't advocate this method but its not a vanilla waterfall method.
http://www.cs.umd.edu/class/spring2003/cmsc838p/Process/wate...
And frankly, if you're working on a massive project that has strict requirements (either regulatory or functional) then up-front analysis has a lot going for it.
But maybe you know this method as these steps (which are the same): Concept -> Functional design -> Graphic design -> Building -> Testing -> Deploy.
Nothing strange about that. Take a look at the IBM Rational Unified Process http://en.wikipedia.org/wiki/IBM_Rational_Unified_Process
I just think this is a hyped site trying to sell there book.
Unless you are a consulting company the size of IBM. With the same business model.
But they're probably running a computer model of the jet (and its systems) under various simulated scenarios.
They're also probably building some full-scale mock-ups to test in wind tunnels.
And they also have test pilots to fly things which sometimes never make it into wide scale production.
So, I'd argue that there is some iteration going on there.
In the 1950s, test pilots were being killed at the rate of about one a week[1]
That's why I don't think the discussion should really be Waterfall vs. Agile, it should be about feedback loops in the process. There are circumstances in e.g. military or medical projects where you absolutely, positively, need to work to an approved spec with time & budget analysis, etc.; it seems to be common outside of software (I don't see much of agile-built bridges or jetplanes). There are those parts of the process that can be predicted (and have their cost estimated) relatively well - those you plan in advance (aka. waterfall). There are others, where you only have a somewhat vague and changing final goal - there your only real solution is a process with a tight feedback loop (aka. agile).
Now you can look at software process on different levels - what might look like waterfall from the high level may be in fact a collection of very agile sub-processes. Which may contain waterfally sub-levels (like, isn't a single iteration of TDD red-green-refactor cycle a bit waterfally? ;). So I think instead of thinking about "agile vs. waterfall" we should consider what feedback loops we need where, and how tight they should be.
This is freakin' unbelievable. ROFL
How are we missing the -
Paperback + Consultation for $250 Complete 45 page book (45 pages?) Ships within 2 days 1 hour 1-on-1 consultation Expert implementation advice
They sound like experts to me. 37signals book Rework only has 288 pages $11. Page count doesn't mean everything, but clearly 37sig is stuffed with fluff. All you need is 45 to change to the game up.
I'm a non-profit and small business owner, I'm sold!
In such a situation, it is in fact entirely rational to develop a lightweight, low cost UI that a non-technical person can commit to, i.e. sign a contract. The developer can then go off and implement it with a very clear legal case if the client then chooses not to pay because "its not what we wanted".
Its not what I do, and I know that many here take a more involved approach to their clients, but its undeniable that it is a successful business model, and in some cases I believe it can produce some result where otherwise there might be none. Equally I'm sure it produces results that are ultimately not used, but where the company still gets paid.
"Hal was the first to introduce the concept of building an interactive demo as a key part of the software development process"
Really? I can't imagine that's true
What's going on there? Hal Helms home helplessly hacked? Or just poor page design?
If they have an arrow that looped from Beta Testing back into Interface Design/Usability/Engineering/Coding, you'd probably end up with what most shops do anyways, speaking in broad strokes.
http://www.youtube.com/watch?v=lIQIZ0NIkxk
He talks extensively about the need to work directly with the end users of the system, instead of the management. The realization that specifications are usually nothing more than mutually assured destruction.
Pretty much the only thing that's changed in five years is that the scale of our company has grown, and now we do indeed have contracts with our clients to protect us from bad things.
Anyhow, I highly recommend the video if you're looking for an in-the-trenches perspective.
We should direct our efforts towards the bloodsuckers, not the people giving blood.
I'm sorry, but the people who claim the one true method is keeping the programmers locked in the basement without real world input or feedback are the bloodsuckers.
And look, I'm not Clark. I don't speak for Clark. This is just the experience I had working with him and getting to know him for a year and a half.
I never realized just how insane waterfall is until I saw this diagram. Wow.
Let's just ignore for a second that a waterfall process sent men to the moon and continues to build skyscrapers, with each project having a set of stakeholders the size of a small city and all while adjusting to a shifting landscape of nearly infinite variables. Especially the first few moon missions, when no one had done it before. It's why all of the original astronauts were test pilots - not just because of their bravado but also their experience in co-designing unfinished machines along with the engineers to bring it to a workable finished state. The roots of the modern HCI and human factors industries comes from analyzing and optimizing the usability of cockpits. The astronauts even had their own usability jokes about a model of training jet that was so user-unfriendly that the emergency instructions for ejection were printed on the side of the canopy, which is nuts because step one was to blow the canopy (along with the rest of the instructions) up off into the sky.
These weren't dumb monkeys strapped to a giant tin can and blown into space, and neither were the engineers that sent them there. With waterfall.
So you use agile/xp/scrum/whatever and swear software is way different than building bridges. I've worked on and with several of those teams, with varying levels of strictness, and guess what? It's still basically waterfall. But instead of PMs owning the process and spec'ing things out in advance, engineers own the process and the spec is written and re-written during development on a bunch of index cards with sharpies like a grade school art project (or the digital equivalent), designers get barely a week ahead of each sprint or run around like caffeinated janitors cleaning up after each sprint, and documentation gets written last or not at all.
And before anyone does ad hominem attacks on whether my teams were doing agile correctly, this is the same dynamic I witnessed at the most famous agile and rails development shop you can think of. You probably only need one guess to figure out who it is.
What matters then, if it's not process? It's called expertise - a broad collection of various techniques and the context-specific awareness of when to use each one. Instead of blindly following a process and trying to hammer it into every situation, take a tools-instead-of-rules approach to using whatever techniques are appropriate for each project.
For anyone interested in this kind of flexible thinking instead of linear GTD productivity tools that leave you stuck in robot mode, there's an excellent book called Pragmatic Thinking And Learning that goes into a lot more detail and pulls inspiration from how the nursing industry overhauled their education system in the 70's.
http://pragprog.com/book/ahptl/pragmatic-thinking-and-learni...
I know, from the book publisher that sells the pickaxe Rails book and stuff on agile. Oh the irony!
The essential difference to me is in length of feedback loops. Microsoft releases a new OS every few years; some place like Etsy releases new code many times a day. A developer or product manager at Etsy can know within a few hours whether some change is really a good idea; Microsoft not so much.
You seem to be saying that because there are the same sorts of people involved, and because one has a giant spec document while another has a stack of index cards that might be called a spec, then they're the same. Which isn't making any sense to me.
It's a knock on all rigid processes and dogma, not agile in particular. I trimmed the rant down by taking out the sections that rip into UX firms, with their Big Upfront Design and cocky rockstar designer personas, but it could just as easily been about them. I only cut it because I thought the parallels were obvious with the diagram on the Epic Consulting website.
What I'm saying is that if waterfall means people are grouped together by specialty - programmers with programmers, designers with designers - each group is run by one dominant process suited to only one of the specializations, and each group tends to work in different stages, then both traditional UX design and agile development are run as de facto waterfall teams, whether by explicit hierarchies or simple stakeholder buy-in.
Consider this - what's the difference between:
1) a PM spec'ing things out in advance and basically telling everybody what to do and in which order.
2) an agile shop coding things up on the fly and telling the designers to run after them to clean up the visuals without making major changes, so they might as well be building to a spec.
3) a UX firm mapping out how the product works before handing off to peon engineers that can't make major changes and might as well be building to a spec.
To end on a positive note, there is one other solution to expertise, by the way. It's grouping teams by product, not role, and figuring out an interdisciplinary way to work together, rather than running the entire team by the methodology of one specialty. That way you're picking good ideas from across different processes and adapting to the company culture to boot. For example, TypeKit does daily stand ups at 10:05am every day and that's an aspect of agile that they've liked enough to use for the whole company, but I'd be surprised if their designers were checking their art assets into Pivotal Tracker or the programmers were writing up UX journey maps of each user persona.
Steve Jobs always liked to describe Apple as the biggest startup he knew and a more accurate description would be that it's a collection of startups, with each one focused on a single product. Seems to work pretty well for them.
EDIT: fixed a few formatting errors.
The crux of the paper is that you should be "grouping teams by product, not role, and figuring out an interdisciplinary way to work together, rather than running the entire team by the methodology of one specialty".
If the word "Agile" freaks you out, then by all means don't call it that. And certainly there are a lot of bullshit artists plying their trade out there. Indeed, if someone tells me they are "Agile", by default I don't believe them. But as this 1986 paper shows, there is a lot of good data and good theory out there if you but look.
[1] http://www.sao.corvallis.or.us/drupal/files/The%20New%20New%...
I agree that "agile" teams that group people by specialty are clueless. Done right, the teams are, as you suggest, cross-functional. That was the original intent for the people who coined the term Agile, but much of that has fallen by the wayside as larger organizations decided Agile was fashionable and consultants rushed in to sell them 3% dilutions of XP or Scrum.
I've written more about how that happened here, and a lot of the early Agile types I've talked to agree: http://agilefocus.com/2011/02/21/agiles-second-chasm-and-how...
I wish I had the courage to write up my disappointment like you did, but I'm happy that I've moved on to a meta-process of borrowing from different techniques as needed.
Whereas in practice, if you find something wrong or confusing with the specification, you just hunt down the business analysts and ask them what drugs they were on when they pooped out the spec. And then after walking them through their mistake, you get a new and better spec.
If you do it often enough, they will start to catch on and actually do their job better. Don't think of that as a poke at BA's either, everybody puts more effort in if they know that someone gives a crap about what they do. They may not like it at first, but you're actually empowering them.
The Agilistas are correct.
The reason for this is that Waterfall was constructed deliberately as a strawman. The very point of constructing Waterfall was to draw a distinction between the process as it was being presented to the VP, or in this case as it is presented to the customer, and the fact that in practice it doesn't work at all and therefore it could not have been what actually happened. Therefore, given that it isn't a possible process anyhow, we should embrace an iterative approach, because ultimately there's no alternative. We should face up to the truth and thereby learn to take proper advantage of it, instead of sneaking it in the back door, hiding it in shame, and deliberately doing stupid things that can't work because management thinks it makes them happy. And in this case, there's no possible way that that is an accurate description of what happens in any significant project that company undertakes.
If you don't understand that Waterfall is not and never was a "real" methodology and don't grasp the history, which it seems still almost nobody does, then you don't really get what's going on. And attempting to rehabilitate Waterfall into a real or viable methodology is metacontrarianism [1] run amok. There's no point in performing CPR on a scarecrow.
[1]: http://lesswrong.com/lw/2pv/intellectual_hipsters_and_metaco...
For example, large-scale construction projects like skyscrapers are run by gantt charts and committees, which sounds unbelievable and like a disaster waiting to happen. It's like the opposite of every startup I've worked at in the last 4 years. Except it actually works and they keep the everything running smoothly through:
* (at the top level) real time adjustments to huge room-filling charts that shown all the dependencies.
* (at the grunt level in the trenches) an abundant use of checklists to keep work mistake-free and to avoid groups stepping on each other.
* (at all levels) constant and real time communication. Lots of it.
So yeah, I think it's hard to write off waterfall as both ineffective and non-existent if a large industry uses it to such a degree.
Even granting your point, though, that doesn't tell us much about software. Nobody expects to start a building as a small house and work it up to something the size of the Pentagon. You can't automatically replace every screw in a building in minutes. Buildings aren't made out of words, and they don't have requirements that change continuously. Buildings aren't expected to respond quickly to competitor's new amenities. The construction of buildings is not 100% automatable with modern technology. So I wouldn't be too hasty in suggesting that the Waterfall approach is made more plausible by a similar-sounding process being used in an entirely different industry.
And honestly, real buildings aren't even much like what you're imagining. Stewart Brand's book "How Buildings Learn" examines actual buildings and how they adapt and change in ways very different than what people with GANTT charts think should happen.
The problem is that construction has a completely, radically different cost model than software. If they could possible throw up one building today, tweak it, and throw up another tomorrow, they'd go iterative too. Instead, they have horrifying implementation costs that have no analogy to software; if Waterfall "works" for them it is only because they have no other choices. That's hardly a ringing endorsement, as wpietri observes.
I have also been meaning to pick up that book...
When Agile became super popular they started saying that 70% of all software projects failed.
Hmmm....
And by using a methodology you are saying just that "we have no fucking clue, so we will try it this way that person X sol us".
I have debated agile proponents about whether agile would work in their respective organizations on multiple occasions. Usually I argue it would fail just as much as any other approach?
Why? Because people try to adhere to it strictly and completely miss or ignore their current context, which is usually THE problem they are trying to solve in the first place.
And you are dead wrong about waterfall. Waterfall is not even a methodology. Anyone selling himself as an "methodology expert" while trying to sell you on "waterfall methodology" is a quack and a fraud. Anyone talking about waterfall as a possible methodology is a quack and a fraud who knows nothing about methodologies. Because waterfall is not a methodology, just like adding rat poison to kids candy is not a business model. Waterfall has always, since the very beginning of the term, been an anti pattern a "anti-methodology".
But all jokes aside, have you ever tried to explain to a self-described 'technical-enough-to-be-dangerous' corporate middle manager what an iterative and incremental process is?
First we meet and discuss the goals for the sprint. Then I design (usually some class diagrams on a sheet of paper or board), then I code and test it (unit tests of course) and finally check it in. Each 4 to 6 week sprint is internally structured as a waterfall. :)
Not that I know what this proves, other than Linked In's shitty use of URLs.
> Helms and Valberg lay out an innovative methodology that finally delivers on the promise of Agile Development.
Unfortunately humility does not sell so well.
I don't talk to anyone.
I look at the crap you're using and make something new.
I give you the URL.
Figure it out on your own.
Invoice due on receipt.