Hacker News Reader

Comment by jl6 | Hacker News Reader

I work for a corporation that has worshipped Waterfall for many decades. We are now in the middle of a years-long large scale “switch” to Agile, which is a vast cultural change.

My observation is that Agile is all about software development, tuned for software development, and for the benefit of software developers and the customers of software.

Obvious, right?

What’s not so obvious is that a surprisingly small portion of what people think of as a software project is actually about software development.

Interacting with 3rd parties, writing contracts, hiring developers, planning how your project interacts with other projects and the wider organization, responding to regulations and external enquiries, managing the demand for certainty from stakeholders, obtaining funding, maintaining financial control...

These are the issues that real software projects have to deal with, and Agile is silent (if not actively hostile) on how you should deal with them.

If your project is truly just about developing software, sure, be Agile about it. But it’s not through ignorance that people “fall” back to Waterfall in the face of real world constraints.

View on HN

> My observation is that Agile is all about ... and for the benefit of software developers ...

No, it's the opposite! Agile is bad news for software developers:

1) hardware developers will insist on having predetermined requirements and will stick to them; at most, they'll throw the mantra: "Software will fix it!";

2) software users will use Agile as an excuse to avoid at all cost deciding what they want and limit themselves to bikeshedding after the release;

3) those idiot project leaders who manage by ticking cases on an Excel printout will explain you how Agile is a wonderful panacea which will solve all your problems: you will "just" have to write the code and you'll have no excuse to miss the arbitrarily imposed deadline.

No no no, not at all! Agile is for those who ponder about developing software, but not for those who do the job!

Paraphrasing here. "Agile is the worst software development process, except all the rest"

More seriously, doing big upfront design was a really bad idea. Agile and more specifically Scrum brings a number of ceremonies to keep the "managers" happy and allow the transition. When and if ever companies understand that they can trust their software engineers they can let them to their job. These ceremonies are more like a security blanked for the non software engineers who are frequently clueless about how software is developed.

I prefer to focus on the Agile principles. My favorite is "Continuous attention to technical excellence and good design enhances agility."

> Agile is the worst software development process, except all the rest.

That's not my opinion. I'll quote Alex' worlds:

In the past decade or so, we’ve seen a lot of “fad development methodologies” crop up. [...] They all promise the same thing – cheaper, better software built faster – and they all generally fail to deliver. Like fad diets, these methodologies try to avoid a simple, painful truth: you have to know what to build before you can start building it. (http://thedailywtf.com/articles/The-Great-Pyramid-of-Agile)

"Faster" (and "cheaper") are the problems here. I believe Scrum embedded this flaw in its very language by calling the iterative cycle "sprints". You don't run a marathon by breaking it into a bunch of sprints - sprinting in a marathon is foolish. But the language suggests speed.

If the Scrum cycle were called "iterations" rather than "sprints", things would be a lot better. Language matters.

This is an interesting language problem. I think part of the reason the words are “agile” and “sprint” is that it’s essentially a way for engineering to communicate iterative progress, convince management that yes, work is getting done, and that really, we’re moving as fast as we can. That there is deception in the wording is, imho, a pragmatic response to management never seeming to understand software development at face value.

Eng: “No, we can’t move any faster. This is literally the best we can do. Doing this right, and doing it as fast as possible, involves care and intentionality.”

Mgmt; “Can’t you just work more hours right now?”

“No, that will not help.”

“You guys are lazy. We regularly pull 70 hour weeks to get all the meetings and emails in.”

“You know what? Next week, we are going to deliver x,y, and z. Promise. It may mean the overall vision takes longer to execute, but I want to show you how hard we work. We are getting a ton done. We are moving as fast as we can. We are sprinting! Now leave us alone and let us do the work necessary to execute the vision.”

Mgmt: “Hey other managers, look how much MY team is kicking ass. They are delivering x, y, and z ON SCHEDULE AND AS PROMISED. They are sprinting, and they basically never tell me ‘no’! Sure, we don’t need y or z to execute the vision, but how else were we to know that? If we hadn’t tried it out? They just sigh and go back to their caves, EVERY WEEK. I’m the best manager.”

It’s basically a way to educate about iterative development while giving management a way to save face, no matter how inefficient it may be. A healthy org could do it without the deception, but corporations are krakens.

>If the Scrum cycle were called "iterations" rather than "sprints", things would be a lot better.

Agree. To me, the real value of Scrum when done right is the focus on delivering increments of value in a sustainable, repeatable way. Scrum provides a means and the existing hype/buzz/literature gives me some leverage in dealing with non-technical managers and getting them to buy-in to the process.

My favorite feature of Scrum on paper, as designed, is the idea that any change to the requirements for a sprint means the sprint ends right there, and a new sprint is started. It's great for "Just leave us the hell alone for two weeks, and we'll deliver what we promised we'll deliver over these two weeks".

Of course, it never works that way. The sprint becomes an arbitrary schedule, and the work of the sprint becomes mutable. Ugh.

Sudden, OMG THIS IS IMPORTANT WE MUST DO THIS RIGHT NOW changes in direction are the bane of productivity.

Add to that: you have to be prepared for sudden changes in functionality and scope.

If you dont then the painful truth mentioned will only apply to utopia (never been there but I hear everyone is rich and thin with infinite throughput).

That is the thing. You rarely know what you need to build. With big upfront design your are speculating, the design is a house of cards. Agile amplifies learning. You learn as you build because you get early and frequent feedback.

Agile relies on having very capable product owners who understand what needs to built. Waterfall relies on design documents. Which will be all wrong anyway if you didn't have a capable product owner driving the vision of the product.

> Interacting with 3rd parties, writing contracts, hiring developers, planning how your project interacts with other projects and the wider organization, responding to regulations and external enquiries, managing the demand for certainty from stakeholders, obtaining funding, maintaining financial control...

These aren't part of Agile 101, but the Agile community has been discussing these things for years. I first heard Mary Poppendieck discuss Agile contracts in 2006 or so [1]. Pat Reed has been talking about Agile accounting since at least 2014, and I doubt that was the beginning of it [2]. My book discusses techniques for making commitments to stakeholders, and it was written in 2007 [3].

Literally Google the phrase "Agile" + anything you wrote and you'll find a wealth of resources. If you're not finding that information, it's because you're not looking.

[1] http://agileconsortium.pbworks.com/f/AgileContracts.pdf

[2] https://www.slideshare.net/ITRevolution/wednesday-530-pat-re...

[3] https://www.jamesshore.com/Agile-Book/risk_management.html

The agile community is currently an amorphous blob of people connected by words like "Agile!", "Scrum", "TDD" and so on. Agile was stolen from its creators and mutated into a corporate version - which I have to say I find pretty darn funny.

Point is, that info is certainly not included in any of the Agile resources I've seen. The only one that handles some of this is Ambler's D.A.D, but I didn't bother reading that for a simple reason: after seeing various Agile methods (mostly Scrum) fail so much, I've come to the conclusion that Agile development processes do not represent a real-world solution.

There's always an excuse why Scrum fails in practice, but in the end it requires such a paradigm shift from our typical development approaches in order to succeed, that it's about as useful as a diet plan that only works on super-athletes.

XP is in zombie mode for a decade, Scrum is only good for mobile apps and small websites, Kanban is not even a way of making software. What are we left with? Scaled Agile, LeSS, which even the original Agile people bitterly complain about? These won't work either, but I know some consultants make a crapton of money selling them to corps.

> I've come to the conclusion that Agile development processes do not represent a real-world solution.

I think one of the problems regarding this whole discussion, whenever it stirs up, is that Agile (as defined in the original manifesto) is explicitly not about any particular processes, but still most of the critique against Agile seems to actually be against different kinds of "agile processes". The manifesto explicitly states "Individuals and interactions over processes and tools".

In real world, one of course needs actual processes, and all kinds of consults are trying to ride the wave to lucrative contracts. But since the Agile Manifesto is about values, not processes, you cannot steal it from its creators. You can pervert the meaning and utilize some braindead processes in your company and call them "agile", but that doesn't lessen the worth of the original values.

I know that that's a highly unsatisfactory answer, a kind of a cop-out, but I think applying more to Agile than it's meant to be is part of the problems surrounding it today.

Edit: those "you"s are meant for passive voice. No personal offense or commentary meant.

The road from Agile values to having something that can be used to ship quality software is way too long. "Eat well and do sports" is correct, yet ineffectual advice, since many countries still have a big problem with obesity, diabetes and other modern diseases.

There are two things undermining "the values" in practice:

1) The economic and social environment doesn't encourage an engineering/scientific approach to SW development. Nor does it encourage eating well or having an active lifestyle.

2) People need a lot of help to implement those fuzzy, high-level values and will always reach for some process / diet / plan.

All things considered, it's probably better to be agile, but how do you actually deliver a quality, successful SW project? Let's talk about that instead of fuzzy old Agile.

I agree with your conclusion fully. Also your link to "eat well and do sports" is kind of exactly what I was after. I mean no one is saying that eating well is bad for your health, right? What I was trying to convey is that I think similarly of the Agile Manifesto itself.

Because we shouldn't infer that eating well is bad because a lot of people try to eat well and are still obese, we shouldn't infer that Agile as in the Manifesto is bad because lots of companies get burned by Scrum. We should be coming to conclusions like "hey, maybe this Scrum thing we tried is shit, let's try some other process that seems to align with these fuzzy agile values we believe in".

The problem I have with the current discourse around the matter is that as long as we keep bashing "The Agile Values" when we mean "agile processes as implemented in framework ZZZ", we are not having the discussion of "hey, which process would align better with these values while letting us ship quality software rapidly".

just curious -- what do you recommend?

I find the whole idea of one size fits all processes dubious. I don't have a solution for this and I haven't seen any such process work reliably in large scale projects.

Scrum quite clearly can't tolerate the peculiarities and pressures of a large scale mixed HW-SW projects for instance. Those will turn the whole sprint / PO / independent SW team arrangement into mush. A long time ago I did some waterfall-RUP mix which didn't feel worse compared to the Scrum I did recently, but it did result in better documentation and traceability.

If I had the chance I would take bits and pieces from several processes and experiment to see which work for a particular company and department, perhaps even at project level. Things that come to mind: Kanban-like task management, regular retrospectives, an exploratory design phase at the beginning, documentation (FFS!), history-based estimation + careful team-private time tracking, a project manager that doesn't also manage the team and doesn't have too much power, regular customer-feedback cycles, CI/CD pipeline, etc.

People tend to say "Resources don't exist" to mean "I have never bothered to look".

The thing is, for software projects, nothing else really works. So if your core business is software, and your organization doesn't embrace this inherent fact and instead does waterfall, there is a risk that all your examples are fiction, including maintaining financial control.

Agile is such a vague thing that it's impossible to say whether it works.

So let's take Scrum, which is probably the face of Agile, whether its parents want it or not: Scrum doesn't work either for a lot of projects. Typically the implementors are blamed, but if a method consistently fails to be implemented correctly by a large segment of projects, I put the blame on the method myself.

Maybe the real problem with software is that it's become too unstable because of market forces and all these methods designed to tame that are just a band-aid. Releasing a new version of an OS yearly seems to be a pretty silly idea...

At the current level of complexity and churn, everything seems to deliver less than impressive results in the end, but people are kind of used to perpetual crappy software.

> if a method consistently fails to be implemented correctly by a large segment of projects, I put the blame on the method myself.

I have heard that sentiment many times but it doesn't really make sense to me.

If a doctor tells patients that going for a run a few times a week is a good way to increase their stamina, but most patients are too lazy or busy to do it, do you blame the method of running then?

Or do you mean that the method should be designed in a way that only companies who will succeed will adopt?

By your standard, is there any successful method to this area?

Yes, I would blame the choice of the method if most patients are too lazy or busy. A good choice would instead work for busy and lazy patients or would convince them to be less lazy and/or busy :) Multiple options should be offered depending on the patient.

I've given actually a similar example regarding the "eat well and do sports" advice. It's not bad advice, it is correct, but it's not working for a large segment of the population, causing massive problems for health care and later (or now?) society.

Instead of cheap advice maybe the doctor should prescribe physical therapy with a specialist where the patient can be educated on the what and the how. Or the appropriate gov organisation should ban or heavily tax soft drinks and crappy foods and make good, healthy food affordable and available.

If a doctor tells patients to go for a run a few times a week with no additional resources and training advice, the doctor just set the patient to fail. Because they will train ineffectively getting little benefit for a lot of effort. Some will get injured from overdoing it. Others will stop improving after just few weeks and then end in frustration.

If a particular method is not compatible with duties and needs of a normal employed person with a familly, then that method constitues a bad advice for such person.

> If a doctor tells patients that going for a run a few times a week is a good way to increase their stamina, but most patients are too lazy or busy to do it, do you blame the method of running then?

Yes, if a medical intervention largely fails in real-world practice, whether or not it is through non-compliance, that's a strike against it as an intervention. On which is more successful in practice, even if worse in the ideal case of perfect compliance, is a better intervention.

Similarly, a development methodology which fails with real world orgs, even if the failure is due to “incorrect” implementation, is a worse methodology than one which produces better results in practice.

Yes, that shows that 'telling patients to go for a run' doesn't work

> Agile is such a vague thing that it's impossible to say whether it works.

Agile methods aren't really particularly usable across orgs for the same reason prior methods weren't; the Agile Manifesto states “individuals and interactions over processes and tools”, but the Agile community is all about processes and tools at the same level, if with different content, than the processes and tools the Manifesto was talking about. How do you prioritize individuals and interactions? The Agile literature doesn't really address this. (Lean, for instance, does through higher-level processes and tools by which teams establish ownership and control of the main processes and tools, which meta-processes are subject to themselves so that he team owns the meta-process as well as the lower-level process.)

Or you get away with selling bad software at exorbitant prices. Alot of big companies do fine (financially) with the waterfall methodology.

> Alot of big companies do fine (financially) with the waterfall methodology.

Because it works for them. Right tool for the right job maybe.

I see companies trying to switch to Agile without even considering that they're applying it to the wrong projects, like big infrastructure ones. It's a buzz word and they have to go for it to look good. That costs.

I guess. Ime. all project are are combination of waterfall and agile. Wrt. infrastructure the initial architecture plan needs to be almost all-encompassing. On the other hand you need to be able to react to new information and changing circumstances. Being religious with methodology is going to backfire 99% of the time.

> Because it works for them. Right tool for the right job maybe.

They are able to deliver software in spite of waterfall rather than because of waterfall. They just keep injecting time and money until they 'declare' their projects to be complete.

Very insightful observation. As someone who has worked in companies just like the one you described, fully agile startups and horrible hybrids invetween, I think that the biggest failure of the application of agile is that persons outside of the development bubble see agile as a solution to old late, overspent, under delivered solutions. Really the agile manifesto needs to be applied top down. Frameworks like SAFe attempt to solve this. In truth the most powerful application of ‘agile’ every entity in an organisation can apply is to fail as fast as possible. third party is slow to interact and hard to deal with? Switch provider or offer assistance. Hiring staff is tough? Look internally at recruitment team and remuneration offered. Project is late and costing to much? Cancel it, don’t pump more more resource in

Not always possible.

The UK has a big infrastructure project called Crossrail, which is running about £2bn over budget is two years late.

You can't just walk away from a project on that scale. Nor can you get 10% of the way in, realise you've made some fundamental errors, and start again.

In reality project management comes down to three problems - finding and optimising competence, dealing with politics and corruption, and minimising danger from external threats.

When projects fail it's almost always because something went wrong in at least one of those areas.

Agile can sometimes help with optimising competence, but only if it's run competently. It won't magically transform any org or situation that lacks fundamental competence and/or has problems with the other two areas.

Agile itself doesn't seem to recognise this.

I thought the problem with Crossrail wasn't that the track wouldn't be ready, it was that the stations wouldn't be ready?

In that case, you could concentrate effort on key stations and only add the remaining stations to the timetable when they are available.

(I don't know the actual details so I may be missing something. All I know is that there are Barnett consequentials for Crossrail that hands money to Scotland, Wales and Northern Ireland but nothing for we English who don't live in the South East).

I think the reason for that hostility is that, since it was decided software needs to be made and software developers are the only ones who can make it, tasks that don't involve making software may be better handled by others.

No one has solved contract writing, and of course developers need to contribute sometimes, but the less time they spend on contracts, the more energy that is put into delivery of working products. Working products soothe stakeholders better than developers detailing why there's no product yet.

A lot of success with Agile comes down to having great administrators/managers/etc who are capable of dealing with as much of the surrounding efforts needed to make software in an organization as possible.

> My observation is that Agile is all about software development, tuned for software development, and for the benefit of software developers and the customers of software.

No, it is not. It is definitely not for developers benefit. It is sorta kinda good for junior developers I think, but mostly it is for management. It effectively ignores human psychology, treats developers as machines and individual developer is mostly powerless with no responsibility, no accountability and zero autonomy. It effectively amounts to large dose of micromanagement.

It may sound weird given agile rhetoric and some its processes are really good and improvement, but in it is does creates pleasant work environment in practice.

Specifications are different than constraints. Agile and constraints can go hand in hand imho.

> These are the issues that real software projects have to deal with, and Agile is silent (if not actively hostile) on how you should deal with them.

A number of attacks on Agile I've read have made precisely the opposite claim: that Agile is "too generic" and not focused enough on the specifics of software development.

My experience is that an Agile approach can be effective at all the things you mention, and you don't particularly need to do anything different on the Agile side of things.

What is true is that Agile relies on being able to iterate quickly with the end customer (in the sense of "whoever is giving you this requirement"), and only adds value when the requirements change quickly and/or the customer isn't actually quite sure what they want.

In my experience this is a lot more common than you might think even in non-software, and creating the conditions for success can be easy once you try. E.g. if you ask your regulator whether you can take a more collaborative approach this time, have a frequent back-and-forth where you show them the current state of things and see whether they're getting what they want, or have any changes to what they want from you - you might be surprised how willing they are to try, and how happy both of you are with the resulting process.

But yeah, it is a substantial change, and I can believe there are domains where the costs outweigh the benefits. If you're working in a domain that's very well-understood, where the customer really can figure out the requirements up-front without even a prototype to go on and those requirements really are going to stay the same for however long it takes to implement them, then fair enough, maybe waterfall will work better. I do think very little of business actually works that way though; in my experience even "successful" waterfall projects that deliver all the requirements that were originally identified often turn out as something most stakeholders aren't very happy with. "It's just what we asked for, but not what we want" is very real.

The Agile Methodology is to argue about The Agile Methodology.

The only durable thing I've learned about methodologies:

Alistair Cockburn "Characterizing People as Non-linear, First Order Components in Software Development." Oops. https://alistair.cockburn.us is "under construction". Sorry.

Meanwhile, here's a pretty good TL;DR: http://www.johngoodpasture.com/2009/11/people-puzzle-non-lin...

Anywho. My current position is "just tell me what you want, I'll adapt." As a recovering methodology junkie, I can no longer stand talking about it. If everyone is on the same page (roles, rules, expectations, etc), I'm happy. If someone dictates this ceremony by fiat, minimizing discussion, all the better.

Agile is all about how contractor developers should interact with their clients.

By coincidence, some of it applies to software development.

Reply on news.ycombinator.com