The Waterfall methodology was a historic accident and they knew it
beza1e1.tuxen.de
beza1e1.tuxen.de
My observation is that Agile is all about software development, tuned for software development, and for the benefit of software developers and the customers of software.
Obvious, right?
What’s not so obvious is that a surprisingly small portion of what people think of as a software project is actually about software development.
Interacting with 3rd parties, writing contracts, hiring developers, planning how your project interacts with other projects and the wider organization, responding to regulations and external enquiries, managing the demand for certainty from stakeholders, obtaining funding, maintaining financial control...
These are the issues that real software projects have to deal with, and Agile is silent (if not actively hostile) on how you should deal with them.
If your project is truly just about developing software, sure, be Agile about it. But it’s not through ignorance that people “fall” back to Waterfall in the face of real world constraints.
No, it's the opposite! Agile is bad news for software developers:
1) hardware developers will insist on having predetermined requirements and will stick to them; at most, they'll throw the mantra: "Software will fix it!";
2) software users will use Agile as an excuse to avoid at all cost deciding what they want and limit themselves to bikeshedding after the release;
3) those idiot project leaders who manage by ticking cases on an Excel printout will explain you how Agile is a wonderful panacea which will solve all your problems: you will "just" have to write the code and you'll have no excuse to miss the arbitrarily imposed deadline.
No no no, not at all! Agile is for those who ponder about developing software, but not for those who do the job!
More seriously, doing big upfront design was a really bad idea. Agile and more specifically Scrum brings a number of ceremonies to keep the "managers" happy and allow the transition. When and if ever companies understand that they can trust their software engineers they can let them to their job. These ceremonies are more like a security blanked for the non software engineers who are frequently clueless about how software is developed.
I prefer to focus on the Agile principles. My favorite is "Continuous attention to technical excellence and good design enhances agility."
That's not my opinion. I'll quote Alex' worlds:
In the past decade or so, we’ve seen a lot of “fad development methodologies” crop up. [...] They all promise the same thing – cheaper, better software built faster – and they all generally fail to deliver. Like fad diets, these methodologies try to avoid a simple, painful truth: you have to know what to build before you can start building it. (http://thedailywtf.com/articles/The-Great-Pyramid-of-Agile)
If the Scrum cycle were called "iterations" rather than "sprints", things would be a lot better. Language matters.
Eng: “No, we can’t move any faster. This is literally the best we can do. Doing this right, and doing it as fast as possible, involves care and intentionality.”
Mgmt; “Can’t you just work more hours right now?”
“No, that will not help.”
“You guys are lazy. We regularly pull 70 hour weeks to get all the meetings and emails in.”
“You know what? Next week, we are going to deliver x,y, and z. Promise. It may mean the overall vision takes longer to execute, but I want to show you how hard we work. We are getting a ton done. We are moving as fast as we can. We are sprinting! Now leave us alone and let us do the work necessary to execute the vision.”
Mgmt: “Hey other managers, look how much MY team is kicking ass. They are delivering x, y, and z ON SCHEDULE AND AS PROMISED. They are sprinting, and they basically never tell me ‘no’! Sure, we don’t need y or z to execute the vision, but how else were we to know that? If we hadn’t tried it out? They just sigh and go back to their caves, EVERY WEEK. I’m the best manager.”
It’s basically a way to educate about iterative development while giving management a way to save face, no matter how inefficient it may be. A healthy org could do it without the deception, but corporations are krakens.
Agree. To me, the real value of Scrum when done right is the focus on delivering increments of value in a sustainable, repeatable way. Scrum provides a means and the existing hype/buzz/literature gives me some leverage in dealing with non-technical managers and getting them to buy-in to the process.
Of course, it never works that way. The sprint becomes an arbitrary schedule, and the work of the sprint becomes mutable. Ugh.
Sudden, OMG THIS IS IMPORTANT WE MUST DO THIS RIGHT NOW changes in direction are the bane of productivity.
If you dont then the painful truth mentioned will only apply to utopia (never been there but I hear everyone is rich and thin with infinite throughput).
Agile relies on having very capable product owners who understand what needs to built. Waterfall relies on design documents. Which will be all wrong anyway if you didn't have a capable product owner driving the vision of the product.
These aren't part of Agile 101, but the Agile community has been discussing these things for years. I first heard Mary Poppendieck discuss Agile contracts in 2006 or so [1]. Pat Reed has been talking about Agile accounting since at least 2014, and I doubt that was the beginning of it [2]. My book discusses techniques for making commitments to stakeholders, and it was written in 2007 [3].
Literally Google the phrase "Agile" + anything you wrote and you'll find a wealth of resources. If you're not finding that information, it's because you're not looking.
[1] http://agileconsortium.pbworks.com/f/AgileContracts.pdf
[2] https://www.slideshare.net/ITRevolution/wednesday-530-pat-re...
[3] https://www.jamesshore.com/Agile-Book/risk_management.html
Point is, that info is certainly not included in any of the Agile resources I've seen. The only one that handles some of this is Ambler's D.A.D, but I didn't bother reading that for a simple reason: after seeing various Agile methods (mostly Scrum) fail so much, I've come to the conclusion that Agile development processes do not represent a real-world solution.
There's always an excuse why Scrum fails in practice, but in the end it requires such a paradigm shift from our typical development approaches in order to succeed, that it's about as useful as a diet plan that only works on super-athletes.
XP is in zombie mode for a decade, Scrum is only good for mobile apps and small websites, Kanban is not even a way of making software. What are we left with? Scaled Agile, LeSS, which even the original Agile people bitterly complain about? These won't work either, but I know some consultants make a crapton of money selling them to corps.
I think one of the problems regarding this whole discussion, whenever it stirs up, is that Agile (as defined in the original manifesto) is explicitly not about any particular processes, but still most of the critique against Agile seems to actually be against different kinds of "agile processes". The manifesto explicitly states "Individuals and interactions over processes and tools".
In real world, one of course needs actual processes, and all kinds of consults are trying to ride the wave to lucrative contracts. But since the Agile Manifesto is about values, not processes, you cannot steal it from its creators. You can pervert the meaning and utilize some braindead processes in your company and call them "agile", but that doesn't lessen the worth of the original values.
I know that that's a highly unsatisfactory answer, a kind of a cop-out, but I think applying more to Agile than it's meant to be is part of the problems surrounding it today.
Edit: those "you"s are meant for passive voice. No personal offense or commentary meant.
There are two things undermining "the values" in practice:
1) The economic and social environment doesn't encourage an engineering/scientific approach to SW development. Nor does it encourage eating well or having an active lifestyle.
2) People need a lot of help to implement those fuzzy, high-level values and will always reach for some process / diet / plan.
All things considered, it's probably better to be agile, but how do you actually deliver a quality, successful SW project? Let's talk about that instead of fuzzy old Agile.
Because we shouldn't infer that eating well is bad because a lot of people try to eat well and are still obese, we shouldn't infer that Agile as in the Manifesto is bad because lots of companies get burned by Scrum. We should be coming to conclusions like "hey, maybe this Scrum thing we tried is shit, let's try some other process that seems to align with these fuzzy agile values we believe in".
The problem I have with the current discourse around the matter is that as long as we keep bashing "The Agile Values" when we mean "agile processes as implemented in framework ZZZ", we are not having the discussion of "hey, which process would align better with these values while letting us ship quality software rapidly".
Scrum quite clearly can't tolerate the peculiarities and pressures of a large scale mixed HW-SW projects for instance. Those will turn the whole sprint / PO / independent SW team arrangement into mush. A long time ago I did some waterfall-RUP mix which didn't feel worse compared to the Scrum I did recently, but it did result in better documentation and traceability.
If I had the chance I would take bits and pieces from several processes and experiment to see which work for a particular company and department, perhaps even at project level. Things that come to mind: Kanban-like task management, regular retrospectives, an exploratory design phase at the beginning, documentation (FFS!), history-based estimation + careful team-private time tracking, a project manager that doesn't also manage the team and doesn't have too much power, regular customer-feedback cycles, CI/CD pipeline, etc.
So let's take Scrum, which is probably the face of Agile, whether its parents want it or not: Scrum doesn't work either for a lot of projects. Typically the implementors are blamed, but if a method consistently fails to be implemented correctly by a large segment of projects, I put the blame on the method myself.
Maybe the real problem with software is that it's become too unstable because of market forces and all these methods designed to tame that are just a band-aid. Releasing a new version of an OS yearly seems to be a pretty silly idea...
At the current level of complexity and churn, everything seems to deliver less than impressive results in the end, but people are kind of used to perpetual crappy software.
I have heard that sentiment many times but it doesn't really make sense to me.
If a doctor tells patients that going for a run a few times a week is a good way to increase their stamina, but most patients are too lazy or busy to do it, do you blame the method of running then?
Or do you mean that the method should be designed in a way that only companies who will succeed will adopt?
By your standard, is there any successful method to this area?
I've given actually a similar example regarding the "eat well and do sports" advice. It's not bad advice, it is correct, but it's not working for a large segment of the population, causing massive problems for health care and later (or now?) society.
Instead of cheap advice maybe the doctor should prescribe physical therapy with a specialist where the patient can be educated on the what and the how. Or the appropriate gov organisation should ban or heavily tax soft drinks and crappy foods and make good, healthy food affordable and available.
If a particular method is not compatible with duties and needs of a normal employed person with a familly, then that method constitues a bad advice for such person.
Yes, if a medical intervention largely fails in real-world practice, whether or not it is through non-compliance, that's a strike against it as an intervention. On which is more successful in practice, even if worse in the ideal case of perfect compliance, is a better intervention.
Similarly, a development methodology which fails with real world orgs, even if the failure is due to “incorrect” implementation, is a worse methodology than one which produces better results in practice.
Agile methods aren't really particularly usable across orgs for the same reason prior methods weren't; the Agile Manifesto states “individuals and interactions over processes and tools”, but the Agile community is all about processes and tools at the same level, if with different content, than the processes and tools the Manifesto was talking about. How do you prioritize individuals and interactions? The Agile literature doesn't really address this. (Lean, for instance, does through higher-level processes and tools by which teams establish ownership and control of the main processes and tools, which meta-processes are subject to themselves so that he team owns the meta-process as well as the lower-level process.)
Because it works for them. Right tool for the right job maybe.
I see companies trying to switch to Agile without even considering that they're applying it to the wrong projects, like big infrastructure ones. It's a buzz word and they have to go for it to look good. That costs.
They are able to deliver software in spite of waterfall rather than because of waterfall. They just keep injecting time and money until they 'declare' their projects to be complete.
The UK has a big infrastructure project called Crossrail, which is running about £2bn over budget is two years late.
You can't just walk away from a project on that scale. Nor can you get 10% of the way in, realise you've made some fundamental errors, and start again.
In reality project management comes down to three problems - finding and optimising competence, dealing with politics and corruption, and minimising danger from external threats.
When projects fail it's almost always because something went wrong in at least one of those areas.
Agile can sometimes help with optimising competence, but only if it's run competently. It won't magically transform any org or situation that lacks fundamental competence and/or has problems with the other two areas.
Agile itself doesn't seem to recognise this.
In that case, you could concentrate effort on key stations and only add the remaining stations to the timetable when they are available.
(I don't know the actual details so I may be missing something. All I know is that there are Barnett consequentials for Crossrail that hands money to Scotland, Wales and Northern Ireland but nothing for we English who don't live in the South East).
No one has solved contract writing, and of course developers need to contribute sometimes, but the less time they spend on contracts, the more energy that is put into delivery of working products. Working products soothe stakeholders better than developers detailing why there's no product yet.
A lot of success with Agile comes down to having great administrators/managers/etc who are capable of dealing with as much of the surrounding efforts needed to make software in an organization as possible.
No, it is not. It is definitely not for developers benefit. It is sorta kinda good for junior developers I think, but mostly it is for management. It effectively ignores human psychology, treats developers as machines and individual developer is mostly powerless with no responsibility, no accountability and zero autonomy. It effectively amounts to large dose of micromanagement.
It may sound weird given agile rhetoric and some its processes are really good and improvement, but in it is does creates pleasant work environment in practice.
> These are the issues that real software projects have to deal with, and Agile is silent (if not actively hostile) on how you should deal with them.
A number of attacks on Agile I've read have made precisely the opposite claim: that Agile is "too generic" and not focused enough on the specifics of software development.
My experience is that an Agile approach can be effective at all the things you mention, and you don't particularly need to do anything different on the Agile side of things.
What is true is that Agile relies on being able to iterate quickly with the end customer (in the sense of "whoever is giving you this requirement"), and only adds value when the requirements change quickly and/or the customer isn't actually quite sure what they want.
In my experience this is a lot more common than you might think even in non-software, and creating the conditions for success can be easy once you try. E.g. if you ask your regulator whether you can take a more collaborative approach this time, have a frequent back-and-forth where you show them the current state of things and see whether they're getting what they want, or have any changes to what they want from you - you might be surprised how willing they are to try, and how happy both of you are with the resulting process.
But yeah, it is a substantial change, and I can believe there are domains where the costs outweigh the benefits. If you're working in a domain that's very well-understood, where the customer really can figure out the requirements up-front without even a prototype to go on and those requirements really are going to stay the same for however long it takes to implement them, then fair enough, maybe waterfall will work better. I do think very little of business actually works that way though; in my experience even "successful" waterfall projects that deliver all the requirements that were originally identified often turn out as something most stakeholders aren't very happy with. "It's just what we asked for, but not what we want" is very real.
--
The only durable thing I've learned about methodologies:
Alistair Cockburn "Characterizing People as Non-linear, First Order Components in Software Development." Oops. https://alistair.cockburn.us is "under construction". Sorry.
Meanwhile, here's a pretty good TL;DR: http://www.johngoodpasture.com/2009/11/people-puzzle-non-lin...
Anywho. My current position is "just tell me what you want, I'll adapt." As a recovering methodology junkie, I can no longer stand talking about it. If everyone is on the same page (roles, rules, expectations, etc), I'm happy. If someone dictates this ceremony by fiat, minimizing discussion, all the better.
By coincidence, some of it applies to software development.
This article is making the point that a "Waterfall Method" has never actually been proposed by academia, so it is incorrect to speak about an Agile/Waterfall dichotomy. In other words: The Waterfall Methodology is not a thing and only seems to be used as a "straw man" foil.
Personally I think we should also be a little more critical about who defines "Agile", how it differs from common sense, and what research has actually been done on it, but thats another matter.
Waterfall is a 'thing' because it existed long before software or computers.
I view it as essentially the most obvious form of project management, which is simply a phased approach to solving a problem: gather requirements, plan, design, do, test, fix problems etc..
The problem was accepting that custom software doesn’t work in the same way and reconciling the huge differences in how much customization people request versus what they are willing to pay.
> Agile, how it differs from common sense
I must confess I haven't read the article but had to post about Agile :-).
It's mostly devolved to ritualistic bullshit performed dogmatically by corporate idiots. Every shitty third rate software sweat shop does agile/scrum these days. You need to look pretty hard to find companies that are not very eagerly proclaiming they do agile.It means absolutely nothing until you qualify what that actually means for you.
Those claiming they are doing it "right" or by the book are the worst because that means they read some stuff without grasping the meaning of what they read. Basically, the agile manifesto signers have come out arguing against that. That was the whole point of that manifesto. They basically provided us with tools, practices, and ways of working that we're supposed to adapt to our context as needed. They don't say: do this dogmatically and follow my instructions by the letter. They actually say don't do that but if you are getting started you might as well do this and then adapt as needed.
That's sort of the same thing that went wrong with waterfall. The article points out a few historical twists to historical context of that. I read the paper by Royce long time ago and found it to be quite modern and insightful. It doesn't mention waterfalls at all (as the article points out). It also points out that, waterfall as we know and love it, basically was institutionalized bureaucratic government organizations using it to sugar coat heavy processes that they came up with that got subsequently adopted as "the way to do things" by equally bureaucratic large software houses they worked with. It got a bad reputation for said corporations making obscene amounts of money with obscenely bad software delivered late and never on budget.
This pattern is something that is a constant and was referred to as the software crisis in the 1960s already and lead to Margaret Hamilton along with other famous pioneers in computing coining the notion of software engineering in the 1970s. She was of course working on the lunar program at Nasa (speaking of bureaucratic organizations). The software crisis is alive and kicking. Many scrum/agile organizations continue to peddle atrociously bad software developed. That doesn't mean that nothing has changed but you should beware of anyone claiming to know the one and only way to do stuff.
There's a lot of common sense that inspired agile and waterfall. I'd argue the modern take on this is to replace manual processes with automation to facilitate rapid flow of changes from inception to production. Another trend is to decouple decision points to facilitate asynchronous processes and communication. For example, you don't bundle features into releases. Instead you ship pull requests straight to CI and CD pipelines. That means that there are no central planning bottlenecks for delivering new features and they can be developed on their own schedules. This happens continually. Most big brand websites you use, update many times per day. Iterations and sprints are kind of obsolete as a way of orchestrating release schedules.
That practice is now creeping into the enterprise. If you are doing scrum and being told not to do this, that's something that you need to challenge.
In areas where changes to the spec are expensive, like building a bridge, waterfall methodology makes a lot of sense. Now that dev tools are so good and computers so powerful, spec changes are cheap, so waterfall is inappropriate. However, there may be times where it is appropriate; maybe a controller for a pacemaker or an oil refinery.
I see it as using the right process for the right job.
> it was a well-defined methodology with proven effectiveness in areas like construction and manufacturing.
If you look at major construction projects, waterfall approaches don't have a great track record. E.g., the San Francisco Bay Bridge had enormous cost overruns: https://www.citylab.com/equity/2015/10/from-250-million-to-6...
In contrast, the Empire State Building was not built with a waterfall approach. It was delivered on time and 18% under budget. I saw Mary Poppendieck, who has written some excellent books on software development, give a fantastic talk on it. There's a write-up of a related talk of hers here: https://chrisgagne.com/1255/mary-poppendiecks-the-tyranny-of...
My belief is that the waterfall approach is common not because it's effective, but because it conforms to certain naive fantasies people -- especially powerful people -- have about projects. I think it's basically King Canute writ large. (Large, and into a zillion GANTT charts.)
I do agree that it's inappropriate for modern software projects. But I think this is also suspect:
> maybe a controller for a pacemaker
I've talked with a number of medical-device people, including Nancy van Schooenderwoert, who has been involved in the Agile world for at least a decade:
https://sdmdglobal.iqpc.co.uk/speakers/nancy-van-schooenderw...
They believe that Agile approaches are way better even for life-critical systems. You do need much more rigorous testing, but given the way Agile approaches deliver core functionality early, you end up having way more time for that testing, and many more opportunities to discover ways to improve the testing approach.
However, what the article describes is fundamentally a waterfall methodology. The innovative part was breaking the large project into several overlapping stages of parallel projects, but the project management was still much more waterfall than agile.
The waterfall methodology is not a product of "naive fantasies" but of business realities. In many cases changing the plan after you get started is phenomenally expensive. AN agile approach of building three stories and then deciding you want to try a different support structure or to add another sub-basement would be absurd.
We all have a natural bias to assume that people in the past were foolish for not doing things the way we do, but for the most part, they were every bit as smart as we are and were making perfectly rational decisions.
In terms of the pacemaker example, I've never built one myself, but I suspect that the constraints are not only the need for additional testing but that the functionality and hardware itself would require regulatory approval, and so a waterfall approach to the project might make a lot of sense.
The developers could, of course, use an Agile or iterative approach to deliver on those requirements, but I think a waterfall approach might be a better fit for the overall project.
Something that works quite in the same way (not as strongly, but still significantly so) in software by the way is re-modelling of the underlying entity schema.
Which by the way is my favorite beef with process-driven agile methods and why I‘m leading my projects with an initial phase of data model and UX sessions until it‘s exactly clear _what_ needs to be built and _how_. Only then does it make sense to build any working code IMHO.
Scrum and friends come with a complete disregard for distinguishing the stuff that‘s easy to change later on vs. the parts that are not.
There needs to be a mindset that everything is temporary, even the underlying entity schema, because it is.
That doesn't work with huge software systems, which is why they mostly stop being developed and can only be somewhat maintained after some time.
You can either work to minimize the amount of change you'll have and identify the places where it's more likely, so you can add flexibility there, or you can be caught by surprise every time and pray that the next change won't suddenly throw years of work away. Both are perfectly sane attitudes on different contexts.
That's not to say you can't use agile methods, but I think the sensible way would be to use agile to explore the solution space and learn what to build, but continue using a formal methodology in parallel construction to document the development of the device.
And that's one of the parts that is fantasy: total knowability. Bosses want that so they can indulge another fantasy: complete control. Perfect plan, perfect, obedience, perfect outcome.
> I've never built one myself, but I suspect that the constraints
Right. Note that here you are indulging in fantasy, in the face of me pointing to expert practitioners saying otherwise. This too is a characteristic of the adoption of waterfall. As the original article says, the paper that defined it said it was a naive, bad idea. But that stopped nothing. People ignored that and made it a template, ignoring the experts.
My company recently opened its first retail store in a new Caruso mall, and it was a considerably eye-opening experience. The pace (and cost) is frustrating, but the end result is generally fantastic and that's more than I can say for most software projects.
Even for a simple job, the number of things that need to align to allow for a dump truck to deliver gravel to a construction site is non-trivial. Along with permits and what times the truck can arrive depending on whatever bureaucratic bullshit is at play, you also need to make sure the rest of the crew is in place to receive the delivery and put it to use. Don't forget about the things that need to be done even before the gravel can be delivered, and things dependent on the delivery that begin on that task is complete.
This isn't a software project. A simple truck delivery being late can require dozens of people to get involved to reschedule everything else that needed to come after it - this is non-trivial, expensive, and can impact hundreds of people depending on how much the schedule slips since we often dealing with hourly labor.
"But it's only changing one delivery" you say? The delivery was probably handled by a contractor, which has other clients and other responsibilities and may not be able to reschedule when it's convenient for you. Multiply this by every contractor or 3rd party you're working with and suddenly Metcalfe's law is punching you in the face.
The waterfall approach exists because it works - on average - considerably better than everything else that's been tried before it, regardless of whatever cosmic forces we think are at play.
Looks like security people telling ops people they'll need to keep some extra data, and then coordinating with development how they are even going to collect it. Or developers moving ahead on their tasks while somebody later discovers that a schema migration will break a different application.
It's not exactly waterfal that solves this kind of problem. It's the other way around, rigid specifications make the problem worse.
I don't understand how you can say this with a straight face. Material costs for software projects are nonexistent (and don't come at me with AWS blabla, it's still way less than when building physical things). Hitting delete on what you've done so far costs nothing. Construction is pretty straightforward in that even if there might be different approaches to building a certain object (like a wall), these different approaches are agreed upon (as in people agree each is a valid approach and will choose which based on the pros/cons of each). Unlike in software where there exists no agreement in how to approach a problem and each programmer out there will probably provide a different solution.
For example, there might be many ways to do the same thing in the software industry, but you might have already dedicated yourself to proving that a given solution works "good enough". So hitting "delete" on that solution would require unmaintainable at that step amount of coordination between many teams who are using your code.
Construction doesn't deal with constant changes to the laws of physics, but software industry has a churn rate of APIs of about 1 new version every 6 months, all of which need to be constantly overseen and accounted for.
Did construction industry have anything like GDPR happen to it in the last 20 years?
You mean like regulations and building codes so that we don't have incidents where apartment buildings collapse like in China?
Some tests to get certain architecture certification require them to literally memorize different rules from everything to how close an AC unit can be to certain things. There are a LOT of rules to follow.
The complexities of construction projects arise at a much larger project volume compared to software projects. The equivalence to a single-family home (which agreeably bears considerable material costs, and already requires a couple of people to build) might be a simple RoR application. Very straightforward, and to some extend every professional agrees on how to execute on that.
But if you accept a certain multiplicator in project costs (say, two orders of magnitude) before you compare a construction project to a software project, you will notice a lot of challenges commonly attributed to software projects.
That a construction isn't constantly scrambling to cover expected-unexpected incidents like a machine breakdown or shortstaff or a contractor screwup?
> That a construction isn't constantly scrambling to cover expected-unexpected incidents like a machine breakdown or shortstaff or a contractor screwup
They are, but thanks to great tooling and process they've been able to deal with it pretty well all things considered.
So your theory that waterfall always works considerably better is demonstrably false.
As an aside, my brother builds houses, and I've spent a great deal more than 30 minutes talking with him about this. I agree houses are not software. But he agrees that waterfall thinking is often much more a problem for him than a solution.
> So your theory that waterfall always works considerably better is demonstrably false.
Uhmm, no? You pointed to a single building and you want to use that one exception to prove a point? Constructed during a time with considerably less federal & local bureaucracy? Seriously?
I'm open to the notion that waterfall processes work well somewhere. But my experience is generally that person in industry A says, "Oh, it really works in industry B." When I talk to somebody in industry B, they say, "Oh, I'm sure it works better in industry C." (Or, often, "industry A".)
Your theory seems to be that the fact that waterfall is the now-dominant paradigm means it must be good. That was the theory in software, and still often is, despite theoretical and practical demonstrations otherwise. So it shouldn't be inconceivable that it dominates in other industries even though it doesn't work well there either.
That is itself pretty naive. There is a reason why waterfall used to be popular, but it no longer is. Is it because people's ego's have ebbed? Doubtful. It's more likely that something else happened in the interim.
The obvious change is the proliferation of instant communications. If communicating is hard and expensive, it often makes more sense to change things as little as possible. If it's easy, then change becomes less expensive.
The internet gave us instant communication (e-mail, video conferencing, github, etc). Tools like these made agile more attractive. Humanity didn't just become enlightened.
Agile doesn't start at the corp level. It starts at the individual and team level (think scrum) and maybe penetrates to the large strategic decisions on a department/corp level. Communication is cheap for teams in the same room or even people in the same building.
A far greater effect is the internet itself and software/services that is/are distributed through the internet. This made shipping new versions cheap, gave an abundant access to metrics and fast feedback cycles. Cheap shipping and fast feedback enable frequent iterations.
When you had to burn your software to CD and mail it to the customer releasing minor updates multiple times a week or even just every few months was not an option. You also got far less feedback and had basically no usage metrics.
Updating software quickly is difficult without internet. Getting feedback quickly and cheaply also isn't.
So you're right that agile isn't just enabled by instant communication among developers, but I still think it's enabled by fast communication generally, including to and from the end-user.
In the past, you got an app and you were lucky if the dev provided a marketing phone number for a drone to talk to. (Here is where IBM and such made bank, they did actual on site support.) Or perhaps an email address that was read manually in a year.
Nowadays, a crash is automatically detected and loses some send feedback going is included in many apps for additional reports, preferably landing in a CRM directly accessed by consumers.
The one problem left to tackle is marketing selling things that are not ready nor even tried out.
Good transparency and bug reporting tools are only possible with instant communication.
If you need to call a meeting and have senior devs fly-in somewhere to report them, bug reporting is difficult, and addressing those bugs are even more difficult. If all it takes is instantly opening a github issue, you can find out about the issue and address it nearly instantly.
In contrast, modern Agile shops release early and often, with product choices for an initiative being informed by the real-world data generated by actual use. As an example, last I did a startup, our team of 6 was releasing a few times per day. The product manager would have an idea, find the minimum essence of it, and we would build and release that. User tests and live user data would quickly tell us whether the experiment pointed in the right direction; we'd then either double down on our bet and build a little more or we'd change direction. The average cycle time between having an idea and testing it was under a week for us. In mini-waterfall shops, that number is generally months.
The reason the 18-36-month version of waterfall died out was not modern communication tools. When my dad was a developer in the 70s and 80s, you had nothing like remote work. There was a special room that the computer was in, and everybody had directly connected terminals in an office they went to every day. Some places were pure waterfall. A few were doing what we'd now call Agile, releasing new software every few days and seeing how it helped.
From the way my dad tells it, the difference in the waterfall and non-waterfall shops was the extent to which executives preferred a sense of ultimate control over trusting expert staff. I agree that hasn't changed. What did change to force many companies to a shorter cycle was the change in expectations created by the rise of the internet. 18 months seemed a very long time on the web. (Although as late as 2004, I knew about one prominent brand name that did annual releases for their website.) So you'll see that consumer software now comes in smaller chunk sizes. That's sometimes true in the enterprise context and sometimes isn't. I know of 18-24-month projects at large, traditional organizations with controlling executives, captive user bases, and low-competition business environments.
So waterfall hasn't gone away entirely; it may have gotten a facelift and been forced to retreat to the margins, but it's still definitely there.
So you're saying that the reason shops have moved to agile is because executives have changed their psychology and are now far more comfortable with relinquishing control than in the past? Hard to accept.
What's much easier to accept is that communication tools have reduced the cost of changing specs and building consensus, allowing the same number of people to make far more changes in the same amount of time with the same amount of effort.
The incentives of an executive (or generally, of a human) haven't changed for a few thousand years.
I am explicitly not saying that. In fact, the very next sentence from the one you quote is agreeing that the variation in psychology has not changed.
If you finish reading that paragraph, you'll see I'm saying what changed is consumer expectation given the rise of the Internet. People were ok with new versions every 18 months when they had to come in the mail and be manually installed. But the web got people used to frequent improvements and fast responsiveness. This caused pressure on executives to shift at least to mini-waterfall.
That change in expectations has differing impact based on the industry, and how much executives are isolated from that. Large companies with captive customers can more easily get away with longer waterfall processes, especially when the product is for an internal audience that can't switch tools. Startups and consumer-focused products tend to feel the pressure more, and so stick with much shorter cycles, some of them even doing what I'd consider true Agile.
So in sum, I'm saying some shops became Agile and many more just went to mini-waterfall with Scrum labels, the variation being partly due to the existing psychological and cultural variation, but mainly due to the extent to which customers and/or users demanded more responsiveness.
That's why I've found Agile so much more effective - if you get a system that works end-to-end at the earliest possible stage it becomes so much clearer what's possible and what's not, and your design discussions are much more productive. You still have to make design decisions and tradeoffs, but you get to make them much later on when you have much more information to work with.
There has been a bit of not understanding the requirements. But more is in not understanding current personal capabilities or needs.
Think of it as training for a marathon. At the end of the day, it is knowable how many calories you will need to burn and how far you are going to run. You are still far more served by running several five and then ten kilometer races, than you are studying the marathon track you want to run. Similar goes for software. Make a few customers happy. Then more. Then more. Don't stall out thinking you need to be at Twitter scale for day one. Not even Twitter did that.
Twitter only scaled well when they started over rather than trying to use Agle to bolt on add hock solutions.
The difference is, I think, more that we are for whatever reasons better at predicting our true bridge requirements than we are at predicting our true software requirements.
Which the OP discusses some.
I‘d like to differ. Changing the spec in every iteration is going to end you up with a suboptimal architecture, because at some point devs will stop tearing it all apart and just refactor „a bit“. Of course, that’s different for various project sizes - a mega project like FB obviously has to allow changed requirements, all the time. But I‘m not sure that should actually be called a project.
An atomic project should be of finite duration, and it should be finishable. Ideally not longer than 6 months. For those, requirements should not change much after a short cool-down phase. Just like tasks/stories/features/epics, projects are made up of different levels with very different aspects.
I find the ideas presented in the article appealing, because they are refreshingly realistic even for one-off freelance projects to pull off.
If your developing software that manages your countrys core ip network getting the design right is key.
In the above case we did waterfall design (nine months) and then used RAD/Agile to develop the software in 12 weeks
I think the golden middleway is somewhere in between, where you do the upfront analysis and figure out what to do, why you’re doing it and with whom, but don’t lock everything down tight in an unchangable requirement specification.
I think methologies like service design can be really great, if you don’t know what you’re trying to do. I think agile methologies are decent for the actual development of software, but a software project is so much more than that, including benefit realisation.
I think it’s embarrassing that we can plan and build a organic house for thousands of employees, and deliver on time and within budget, but we can’t build a piece of software to handle the digital workflow of labelling and storing documents.
Hell, we plan city infrastructure and succeed, every year without going over budget. But building a piece of software to digitise vacation registration? Fuck no, that’ll go one year over time and cost four times what we expected, even though you could rightly build the same thing in a few months as a single focused developer.
I’ve seen all sorts of project management models in action, none of them work for software. But agile is especially stupid from the buyers perspective. When I buy a new house I wouldn’t sign a contract for an unknown amount of finished rooms within the agreed upon timeframe and budget either.
Mostly though, I think software developers just need to do better work, and stop blaming the project management for their own inability to deliver good work in a timely manner.
Even if this is the case, PM still responsible for blown deadline, because they did not estimate their teams delivery velocity correctly.
They should either:
a) give longer estimate, if their team quality is low.
b) pay more to hire better, more productive developers.
A good PM will also account for uncertainty in devs estimation, by padding etc
Predictability of delivery is what ultimately counts, not giving optimistic deadlines.
You hope you don't. But halfway into construction, everyone might realize the blueprints attached to that contract don't cover some detail you really need. Then if you want that done, you'll have to pay extra. That's why change orders are so common in construction.
In a pre-internet era, where every change could take weeks to permeate an organization, change became extraordinarily expensive. With faster communication, the cost of change decreased, and made agile development more attractive.
The author of the post does a good job summing up the issues. I only skimmed it but have to agree with everything I read.
I can add that there's a time and place for Waterfall: when requirements are well-known and static and when lots of oversight is needed. I can also add that no methodology has successfully dealt with the mega-projects we read about in software engineering case studies.
Building stuff, whether it's software or anything other product, is hard. It's messy. Often the requirements are not well thought out. In the situations I normally deal with, a requester can state a high-level strategic goal for the organization, but not how the software can get us to it. That's where my team gets involved. One engineer I respect highly once said that we need to get something in front of the requester quickly so that we can find out what (s)he doesn't want. By process of elimination, the idea foments in the requester's mind and requirements become more concrete.
Again, though, some organizations are very static or they deal with software whose failure can be catastrophic. In those situations, the highly bureaucratic Waterfall method can be helpful at least for slowing down the process.
1. It lets everyone involved cover their ass. "All I'm responsible for is my function. There was a problem with my inputs, so no wonder my outputs suck. Don't blame me, my inputs are all right there in black and white, and you signed off on them."
2. It gives people the comforting feeling that they can predict the future, and then do things like plan budgets or negotiate fixed price sales contracts. Predicting the future is irresistible for people who handle money. As fictional as it might be.
So those two things, I think they have a momentum of their own, and without constant vigilance and effort, for most domains, waterfall development is a sad, inevitable default.
Agile prescribes so many things that are absolutely pushover, toothless concepts in the face of basic corporate bureaucracy that it really is a case where it’s the tool’s own fault that everyone uses it wrong.
I’ve seen Waterfall used successfully. Not often, but on a few defense research projects it was great.
I have never once seen or heard of any Agile process anywhere actually working with any degree of success.
I’ve heard plenty of people indulging some No True Scotsman nonsense about how a “real” Agile process can succeed, and then proceeding to write a diatribe about their “real Agile” workplace that is clearly so fucked with dysfunction that the person writing can’t even realize they are only shipping software in spite of Agile and not the least bit because of it. But I’ve never ever seen or heard of it actually causing success, not across any of the countless blog posts or anecdata on HN, and not across 6 different jobs at companies of various ages, sizes and capitalization levels all using Agile withvarying degrees of formality, pragmatism, Agile coaching and consulting, etc.
None of it made any difference. Agile just hinders people. It is not capable of doing otherwise.
Now, having said all that, I do agree that for processes that can work, processes that are not overly prescriptive, let different teams do things different ways at different times based on different goals or different personnel, let teams ignore the customer when needed, enforce long-term planning when needed, do pure research with no time boxing when needed, etc., then for those kinds of common sense, non-Agile processes, we have to change the incentives so that people are aligned to care about the objective truth and objective outcomes required to get the job done.
I ran a team were we shipped (firmware!) every 3 weeks.
After the first year of tracking all our numbers we were able to hit our estimates within a few hours +/- for the entire team.
3 week sprints, 2 weeks writing code, 1 week testing and bug fixing.
It wasn't perfect, and learning how to do it was painful, but once we got going it worked great.
We didn't adhere strongly to any one guiding principle, but being able to use sprints to align work across multiple teams, and as a planning mechanism for management was incredibly useful.
"Our breakdown of that feature means it'll take 3 sprints in total, the last of those sprints the work is split across 2 devs, if one of those devs gets reassigned it'll take 4 sprints."
Being forced to actually plan work out, and then being held accountable for the hours, meant that over time we got good at estimating, and knowing how far to break work down. There is of course a trade off between gains from breaking work down, and time spent doing the breakdown! Several months were rather painful when we went too far one way, and then inevitably the next few months swung too far in the other direction.
But in the end? Yeah it all worked out. We figured out what length of sprint worked for us, what type of estimation system to use, and how much work to put into our planning sessions.
FWIW one of the huge benefit of doing the work breakdowns is being able to see opportunities to parallelize work, and also to see natural stopping points.
My team was lucky to have someone almost full-time dedicated to managing our workflow, without that the entire process wouldn't have been tenable. The tools just suck up too much time.
We built Bioshock using agile.
We didn't use any prescribed agile methodologies like SCRUM, but we developed a process that worked for us based on agile core principles.
update: Sorry meant to reply one above.
The risk of No True Scotsman applied to Agile is that you go and find a process that works, and because it works, it gets labeled as Agile, even though it may (likely) have nothing to do with anything that could be said to resemble Agile.
That's more consistent with the Agile Manifesto than calling any of the concrete development methodologies that come out of the Agile community “Agile” merely because they emerge from that community.
Choosing the process that works in practice for your team in their concrete situation is pretty much the whole of the Agile Manifesto. The failure of the Agile community is that it has almost no literature or body of knowledge oriented toward how you do that, just a bunch of concrete development practices without tools for adapting them or choosing among them.
OTOH, tools operationalizing essentially the values of the Agile Manifesto do exist in the Lean community. (Which is why I sometimes, when feeling particularly snarky, call Lean “Agile, but for engineers” or “Agile, but usable”.)
It reminds me of a great LessWrong quote, “If you can explain every outcome equally well, you have zero knowledge.”
No, it's just the definition is not in the space you (and to be fair, lots of other people) seem to want it to be in.
> If “Agile” == whatever works, then Agile’s not a thing, it cannot differentiate between what works and what doesn’t, and it can’t predict in advance what will work.
“Whatever works” is definitely a thing; however, the kind of thing it is, as you note, is not a tool for predicting what will work.
“Agile”, though, while not that kidnd of tool, is not exactly “whatever works”, either (and here I'm talking about the Manifesto; it's worth noting that the Manifesto, the Agile community, and the concrete methods labeled as “Agile” are not precisely in alignment); it is a critique of (accusation against? hypothesis about?) the status quo at the time it was written attributing its disappointing performance in finding what works to three errors:
(1) Failing to recognize that what works, at the level development practices were generally recommended, is not a universal across software development, but a function of the particular members of the team and their relations, which not only vary between efforts, but vary over time within one effort.
(2) Prioritizing written artifacts of the development process more than is justified by their utility to software development.
(3) Preference for arms-length negotiation and implementation to collaborative involvement of the customer with the development effort.
> actually plan
sounds like waterfall with extra steps (like a lot of corporate agile)
Waterfall is about the set of phases and the staggered project transitions. See https://en.wikipedia.org/wiki/Waterfall_model
Agile software development is about 'advocates adaptive planning, evolutionary development, early delivery, and continual improvement, and it encourages rapid and flexible response to change'. See https://en.wikipedia.org/wiki/Agile_software_development
For a very large feature, agile deliveries will split that larger feature into chunks and deliver those. That reduces risk and can get feedback ('adapt') to how that 'chunk' has gone.
Edited for grammar.
Typically this results in a mishmash and never finishing. (instead the thing is dropped) Bioshock itself still has game breaking bugs nobody will ever fix - and its story wasn't too iterative.
The process is not immune to building the wrong thing at all - devs always want to side with easier vs better and agile pushes them even more by forcing things to be independently chunked...
I’m skeptical of this, because even working in a team that had many person-decades of experience of estimating software delivery, the consensus opinion was that nobody’s estimates are worth anything and that the business would actually be worse off using estimates even as mere guidelines than just working on things, hitting blockers, and adjust expectations as you go.
But regardless it sounds like your process was fairly different from Agile. The only superficial similarity was having some notion of a regular sprint cadence, but cadence based scheduling is common to many project management strategies. It sounds like you had something very pragmatic for your exact situation, which is great — that’s the whole goal, the very thing Agile paradoxically does not conceptually support.
Also, XP was very into feedback. "How much did we get done in the last two weeks? How much did we think we were going to get done? Let's use that to adjust our estimates going forward." This was actually done with story points, so that when we "adjusted", the units of estimation stayed the same (that is, an item that was 4 story points before was still 4 story points), but we realized that the team was only doing 12 story points per 2-week sprint, instead of the 20 that we expected.
And yes, it worked very well for us, and it was truly agile.
And I disagree with you that agile doesn't allow something tuned for your exact situation. Or rather, agile allows that, even if Agile doesn't. We'd sometimes say "For the next two sprints, we're going to tweak our approach in this specific way. If it works, we'll keep doing it that way." That is, our process was one of the parameters we could adjust.
That's not really all that agile. The agile teams I've worked on try to break things down much smaller, ideally half-days or less. Any estimate of how complex some task is, is often off by a factor of 2, sometimes much more. Only by breaking down tasks into small bits can you really start getting a handle on your velocity and complexity of a project.
It’s part of the problem with Agile. It means all things to all people, very much like denominations of a mainstream religion.
You’ve got your small estimates people, you’ve got your big estimates people. You’ve got your no-meetings people, you’ve got your tons-of-meetings people, you’ve got your every-iteration-has-to-be-shippable people and you’ve got your what-about-fundamental-research-(and-don’t-you-dare-say-spikes-or-timeboxing) people.
And all of them believe their way of doing it “is Agile.” So the term pretty much has no meaning in practice apart from some juvenile Ten Commandments sort of phenomenon where the only commonality in all of it is time-waster meetings like stand-ups and backlog grooming.
Have you ever tried alternative approaches to software development? I suspect that the main reason you're frustrated with Agile is that you have never dealt with it's predecessors. There is undoubtedly wasted time in Agile, but with other processes, it seems like the goal is to maximize the wasted time.
I do believe smaller chunks are better than larger ones, but I'll happily take larger chunks over going back to a waterfall process.
What I have seen really work are self-organizing policies where teams decide on their needs on a rolling basis. No prescriptive requirements.
For example, Agile might suggest teams can determine what type of sprint length matters for them, within some generally narrow range of sprint lengths.
But a good process will have no concept of a sprint or regular cadence, since it will be allowed to be different for each person and vary by project.
Another example may be that Agile offers flexibility to do research tasks as “spike” tickets or time-boxed regular tickets, so that the scope of a research task is forced to fit in with the iterative nature of Agile.
But a good process will not define a limit on the scope of a research task and will trust the engineer or implementer to choose when to curtail the scope based on the that person’s trusted understanding of the trade-off of probing more vs doing work on some other area, and that other people are likely not in a position to know better if more research investment is better than context switching.
Agile might suggest that at the completion of every sprint, the overall project is in a deployable / shippable state.
A good process would instead defer to the engineers or researchers to own the decision on a case by case basis about whether extended “partially implemented” states are better than the intermittent overhead of needing each change to satisfy delivery constraints.
Basically Agile pays lip service to infinite flexibility, but in reality is quite dogmatic about prescribed process. With Agile you are “free to customize” a series of prescribed global properties like fixed cadence, iterative delivery, etc., but you’re not actually free to arbitrarily disregard those things at whatever moments you need to disregard them.
The minimum condition of any good process is that the engineers and implementers are empowered to ignore and override any and all aspects of the process’s prescribed requirements at any time for any reason, and that the principle thing the business does is to trust the engineers and implementers to selectively disregard fixed processes at the exact moments when doing so is what’s best for the product & customer.
For example one way that Waterfall permits this where Agile doesn’t is to allow engineers to refuse scope change requests from customers. It can be crucial for making a customer happy to tell them “no” and refuse to change your existing plan in the face of new requests. You won’t always do this, of course. Often you will make changes that customers demand. But a good process leaves that freedom to veto a customer request up to the engineers and trusts that they won’t abuse that freedom. Agile on the other hand intrinsically revokes that freedom and takes the attitude of not at all trusting engineers to make that decision.
Agile, despite the lip service, categorically disallows this by its very definition, all the way back to the Agile principles themselves.
This also ignores the fact that there are evolutionary dead ends and digging yourself out of such a hole is hard.
You don't have to be perfect at dividing up a big project, but you should at least try. Breaking down big pieces of functionality into smaller pieces is one of the core tenants of Agile.
In fact, your approach has far more risk. By refusing to try to break things down into smaller units, you risk not shipping anything because every feature is deemed "critical".
#ifdef is a thing and it solves this problem.:)
For more modern teams, branches also work!
I usually agree, but we were in a bit of a special situation:
1. In house designed and built custom RTOS
2. In house designed and built custom hardware platform
3. Our entire UI framework was highly customized by us, for our specific needs
It turns out, when you control all the variables, making estimations becomes possible. It isn't easy, but it is possible!
> the very thing Agile paradoxically does not conceptually support.
Early on, a few of the more senior engineers wanted everything to be done as one big design up front. That is after all how embedded hardware projects are normally delivered. (And for hardware that makes a lot of sense, lead times are a thing!)
The idea of shipping embedded firmware every 3 weeks is considered insane by many people. Heck I think that throughout the industry it is still considered insane.
But if we had not pushed for "agile", users would have suffered. They wouldn't have gotten the incredible stream of features that they did, and the teams wouldn't have been forced to maintain constant stability and code quality.
That idea of "always ready to ship" is one of the tenants of agile. Of course having milestones is common, but "our code is always good" is not common, and it is one of agile's differentiators and adhering to that leads to systematic changes in how teams function.
2. In house designed and built custom hardware platform
3. Our entire UI framework was highly customized by us, for our specific needs
It turns out, when you control all the variables, making estimations becomes possible. It isn't easy, but it is possible!
That's an interesting set of observations there. I've been wondering for a while whether we -- as a field -- have been systematically underestimating the cost of using libraries. To my eye, the anecdata definitely seem to be on the side of projects going more smoothly when the dependencies are few and well understood.
Does anyone know of anyone doing research into this kind of thing?
Especially for experienced devs who have written framework code before. "How long will it take you to write a graphics rendering routine that you've previously written on another project" is easier to estimate than "does this feature in the graphics library we are using work?"
The library probably works, sure, but when it doesn't the problem can become an almost unbounded time sink.
I also think there are some break-points where adding libraries can change the nature of a project significantly. Think of old-school Javascript, loaded directly with a few script tags. In general "new style" JS (with many small libraries, a bundler, and probably some other tooling) brings some noticeable advantages (at least for larger applications), but you definitely lose something relative to the "no tooling, just hit reload" edit-test cycle.
I worked on two internal tools at Etsy that used the most agile of agile processes (no product manager or project manager, no estimation, no timelines and burndown charts, just the people who used the tools and the people writing the tools, a big physical board with cards on it representing the work that needs to get done in priority ordering, and daily standups with the manager of the engineering team. The whole team would have retros every two weeks that were kind of social, kind of project management, kind of technical.
They were the two most wildly successful projects I've ever worked on and the tools were well loved and last I checked a couple years ago, at least one was still being used about 4 years on. They solved the users' problems extremely well.
Maybe I received wrong data, but according to their reports anyway, it sounded like Etsy would be almost a case study in bad Agile processes.
Anyhow, I've used these approaches very happily in a startup context. In particular, I've used processes derived from Extreme Programming, so we used practices like pair programming, frequent pair rotation, test-driven development, small units of work, continuous deployment, weekly retrospectives, continuous process improvement, and a number of other things. A+++, would do again. And all the developers and business people said the same thing.
Part of the problem you may be having is that most of what people call "Agile" is what I'd call "mini-waterfall". It's where a waterfall-oriented organization decided Agile was fashionable so they borrowed a bunch of labels, cargo-culted some rituals, turned up their release frequency slightly, and declared themselves awesome. I agree this is shitty nonsense. But we didn't trademark the term "Agile" at the beginning, so anybody can use the term to mean anything they like. I think this slide began circa 2007, and in 2011 I wrote about what a mess things were: http://agilefocus.com/2011/02/21/agiles-second-chasm-and-how...
Agile prescribes exactly one thing; adapting processes and tools to use those that work for producing working software for the actual people on the team, working in their actual context.
> I’ve seen Waterfall used successfully.
And pretty much all the concrete methods sold as “Agile” are processes that have been used successfully somewhere, too, that's usually what motivates documenting them.
Moreover, Lean (which is essentially the system in which the values expressed in the Agile Manifesto are concretely operationalized, unlike what happens in the “Agile” community where you have the Manifesto reject rote application of canned processes, and then the literature of the community is almost entirely canned processes for rote application rather than discussion of how to adapt processes to teams and situations) shows how Agile values can work, as well as “Agile” canned processes.
Agile is a gimmick to sell. That’s what motivates documenting different examples of so-called Agile.
It’s similar to the age old scam of sending out mailers to 2^n different sequences of buy / sell advice for n different investment decisions, and then only focusing on the people who received the advice that turned out to be correct in every case.
Say that some organization “uses Agile” (whatever that means), but that the engineers deliver a product in spite of the processes that exist “because of Agile” and would have delivered better results with less risk had those processes not existed.
In that case, Agile will still be labeled as a success story and trumpeted around as a documented example for consultants to sell, despite Agile having a negative effect.
It boils down to a causal inference problem: what would have been delivered in the counterfactual situation of some methodology that did not impose any “Agile” processes?
I’m just pointing out that in every implementation of Agile I’ve personally worked in, heard about from friends or read about in HN comments (including this thread), all of the details appear to suggest it really is just this counterfactual problem.
Agile has a subtractive, hindering effect on a process of delivering software, and due to engineers doing extra work to overcome Agile and deliver things in spite of it, the end result is that, no thanks to Agile, organizations still ship things and make customer’s happy. Just not as happy as if they did not approach work with Agile process management.
"In 1985 the US Departement of Defense wanted a documented process for software development: DOD-STD-2167. Since governments and companies reused these requirements, it radiated across the world from there"
I contend that there are more fundamental reasons for this, so even if this quirk of history had never occurred, we'd still see a lot of waterfall development.
The historical philosophy of government procurement is not just an outcome, but a competitive process. In many circumstances each part of delivery let to bid, and bidders performing design work cannot implement.
Even today, design-build (aka agile) is fairly limited in most procurement scenarios.
The thing that’s hard for people to grasp is that equity in procurement often yields increased cost and timelines. If vendors don’t price risk into bids, they lose money and fail to deliver. The government compensates with more stringent requirements that make it harder to bid, which shrinks the bidding pool, etc.
In my opinion, these concepts are fundamental to the successful functioning of a monopsony like the government, who is also subject to it's own equitable competition laws.
0: https://www.gpo.gov/fdsys/pkg/STATUTE-88/pdf/STATUTE-88-Pg79...
So the government compensates with a document heavy process -- not to prepare for implementation, but to prepare for a court case.
EDIT: rephrasing for clarity
In some cases, there is a downside to incorrect behavior that is worse than "oops we'll have to patch it." This category includes anything that handles money, industrial equipment, scientific data, or anything that isn't a computer game or adtech.
Additionally you can have rollbacks - strategy applied by many banks when things break.
Funny enough, safe reverts and rollbacks require specialized up front design, it does not happen on its own. Safe automatic updates with no state loss? Same.
Placing bids is (potentially) spending money. A bug in your adtech platform can absolutely bankrupt you in the very direct way of spending more money than you have.
"...the general approach for us, empirical but pursuing this monstrous goal, monstrously difficult to like building and trying empirically, and we’re approaching it evolutionary-wise because we feel that it’s a whole system problem, you need to get a person in that environment, working and looking at the many aspects of his working system that are involved in his effectiveness. That’s many more things than just these computerated tools. And in a large system like that, we need to do it evolutionary-wise because we can’t be analytic enough about it at any one point, to decide what best our next thing should be. We can only decide from here as well as we can analyze it, where we can invest our next resources to get the most return that will increase of the effectiveness of the system we have and this item down here is the term boot-strapping applied in a slightly loose sense. We’re applying that to our approach".
[1]: http://dougengelbart.org/pubs/video/fjcc68/Englebart's-1968-...
Other sayings are "The spec is in the tickets, there is no user manual." "We have unit tests." and so on.
All this makes trouble - Especially when you consider in order to change a thing, you should aim to understand it's salient parts in the first place - but perhaps that is not agile or the scrum way according to some.
In the end it comes down to constantly adapting to problems after openly admitting these problems.
In software, people usually don't know exactly what they want, let alone be able to precisely define it. This situation lends itself perfectly for rapid-prototyping to discover what works fast... which is basically an agile approach.
In my experience, though, what was unreasonable was the sequence "build, then debug." This meant spending huge amounts of time in debugging after the code was written and made it impossible to keep projects to a schedule because the debug/fix/rerun cycle was open-ended.
This, in my view, is where Agile has rightfully distinguished itself--by having tests developed in parallel with code, thereby making debugging an "every once in a while" activity rather than a whole endless-seeming cycle.
Which is not possible when the person giving them is not a programmer...
The fallacy of all CASE tools stems from this.
On the "waterfall" projects, we wrote lots of documentation. LOTS. Business cases for every major feature. Requirements documents with serial numbered acceptance criteria. Design documents explaining exactly how we were going to implement the requirements, with every REQ bullet needing to be accounted for explicitly. Test plans showing how each DES bullet was implemented correctly and how it proved the REQ was satisfied. And it all had to be correct and fresh. We weren't allowed to have anything disagree or slip in or drop out. If it wasn't in the REQs, it didn't go into the software.
That meant if we discovered constraints that necessitated a change in design, we had to change the requirements. If you had to change requirements, it meant you had to reevaluate the business case.
And that meant we were prototyping while we were writing the documentation. The documentation wasn't done until the software was about 80% complete. By that time, we were mostly just working the test plans and resolving defect tickets to get to 100%.
Some of the best, longest lasting software I've ever written.
All of the "Agile" projects I've been on are what I call BTSOOP development: "By The Seat Of Our Pants". There was no attention paid to documentation at all. It was hell getting PMs to write any tickets, let alone ones that were in scope. And of the tickets that were there, most of the devs ignored them, even if the PM didn't. Some people would mark tickets done that they just didn't want to do, hoping the PM would just blanket-accept tickets at the end of the Sprint. Which he most frequently did.
The difference wasn't actually process. It was accountability for failing to adhere to the designated process. The Agile shops all had corporate policy on how projects would be ran, and nobody was ever taken to task for it. The "waterfall" shops didn't lack continuous integration and improvement or responsiveness to change, we just wouldn't sign off on a particular document until we knew it represented the final product, with linear dependencies on when different levels of assets could be approved (not when they could be worked on).
[0] our product manager called it "the NASA way", and we fairly closely followed their documentation standards, if not their engineering ones, but we were all pretty junior at the time. IDK why I was made senior other than being the best programmer there, which at the time wasn't saying much.
So, if you have a ton of issues while using Scrum, that's actually kind of the point. The real issue is not the framework that unveils your issues, but rather the level at which a team is empowered to fix their own issues.
Scrum is both powerful and useless depending on how empowered teams are to manage their own work. It's really that simple.