Study Finds 268% Higher Failure Rates for Agile Software Projects
tech.slashdot.org
tech.slashdot.org
[1] https://tech.slashdot.org/story/24/06/06/022253/study-finds-...
[2] https://www.theregister.com/2024/06/05/agile_failure_rates/?...
[3] https://www.engprax.com/post/268-higher-failure-rates-for-ag...
[4] https://www.engprax.com/work/impact-engineering-transforming...
Take it with a huge grain of salt.
Hazardous levels of sodium with this one.
What matters is not the "268% higher" failure rate than whatever they are selling. It's the 65% failure rate of projects using Agile.
That said the true question is how they define failure? Missed deadline? Project stopped? Etc.
Reminds me of the study saying "a teaspoon of honey per day is healthy" with funding from the American Honey Something Association.
Not implying it is true, but there is the possibility that Agile allows you to fail early, fail more often, lose less money, try more things, maybe?
Fun fact: I once talked to a pmo person who claimed that 95% of their waterfall projects finished on time and under budget (but only if you include the avalanches of change requests that were approved 8p% of the way through, a time that somehow corresponded with the software going into test)
Study finds 268% higher failure rates for Agile software projects - https://news.ycombinator.com/item?id=40584901 - (170 comments)
Could it be that agile projects fail more often (or take longer to complete) because people with a different "mindset" are able to work on projects like these because they are being enabled to do so due to a different "way of working"?
Example: We have a shop relaunch going on and marketing is in the lead. Project is almost double over budget and at least two months over time, they fail so set clear expectations and requirements and there's sprint after sprint after sprint to get things to a release-ready state.
Without agile project management this relaunch would have long been counted as a fail and consequences would have followed OR marketing wouldn't have been in the lead because they would not have been able to set expecatations and a clear goal from the start on.
Any ideas on that?
Software development is a lot like photography. If your work sucks it’s because the engineer/photographer isn’t close enough to the subject/user/business goal. Agile tends to silo engineers away from their ideal motivations (users/business goals) in favour of meeting the needs of the egos of management.
One case in point was a hedge fund who had increased their engineering team by a factor of 10 because they couldn’t get any work done after the Agile people took over. Their productivity had stabilised at a pre-agile /10 level. I was part of a group of consultants brought in to build them something because they were so unproductive. The consultancy’s solution was to do MORE Agile. So we ended up building a large but not very useful set of features instead of doing the smaller, more technically complex and much more useful feature. We completed the work on time only because of the generous bonuses offered to the outsourced developers. I believe it was unlikely to be worthwhile for the hedge fund.
> The argument used to refute this is usually that people aren’t doing agile right and if only the people and companies were more competent and or did MORE Agile then things would be fine.
> Generally speaking successful tech companies don’t often put unskilled dispassionate middle managers in charge of software projects and don’t often use Agile processes
But do you think this problem is inherent to the agile process itself? We (as in "more technical department") manage to get our things done on time and budget with the same external agency. It might be due to us formulating clear expectations and behaviours and iterating over improvements, but I'd say this whole process of "lots of little steps" works fine for us.
On the other hand, when I think about it right now, we might be doing fine with a more traditional project management process too, because we are able to set expectations... Hm... not sure...
No idea where you get this from. I've never seen teams closer to customers than with agile thinking. Waterfall with a programme manager at the top setting the timeline, the budget and the features, an army of business analysts getting between the team and the customers, and project managers making GANTT charts - you don't think this silos away information?
When companies have too much money, they feel they have to spend it, and so they spend it on hiring too many engineers, designers, product managers, etc. And they also feel they have to keep tweaking things when things are good enough to be left alone, in order to justify all the hiring. It's basically busy work on top of busy work.
My most productive times as a programmer was when it was just me and another programmer hammering out 500+ lines of code per day and where we were our own product managers. When you have a team of 10-12, you need to spend a lot of time creating well-sized tickets (that really are ludicrously small) that are doable in the allocated time of the sprint. Then each programmer spends, say 2 weeks in a 2-week sprint, producing about 50 lines of code, testing it, and then deploying.
TL;DR: There are too many hires (engineers, product managers, etc.) and not enough work, so we end up doing a lot of busy work. That is Agile.
EDIT: to fix typo.
if so, this avoids wasted effort, and is a feature not a bug.
There are projects for which you can't know everything ahead of time and you need to iterate and seek customer feedback. These are likely more innovative projects, and of course more risky and more likely to fail.
Not finishing on time is a very different thing from failing. Especially for agile, where you're supposed to deliver the parts that provide the most value early.
What other processes are being used?
At that point any refactor is the automotive equivalent of a recall. That’s unfortunate because when the right processes are in place enormous refactors on even large applications can be an effort shorter than a sprint planning meeting.
It's really good for a consultancy to churn through developers whilst billing hourly / daily, but less so if you want to build a product, or actual productivity.
That's not to say there's nothing good about agile, but when your average team consists of a Delivery Manager constantly applying pressure and Product Manager with no tech skills deciding what gets worked on, it's immediately obvious nothing actually changed.
I am not saying whether Agile is good or not, but there are better approaches that can yield faster trajectory with lower defects. Its a trade off between automation/autonomy versus administrative oversight.
agile had a lot of promise, though is definitely biased towards working with external devs, but in practice, in my experience, is boils down to the DM and PM roles turning the team into a feature factory 9/10 times.
That said, it isn't really about agile, it's about the specific people, and how suddenly adopting some framework doesn't really change them or their incentives.
""" Yesterday I read a report claiming that Agile projects had "268% Higher Failure Rates" than projects with more upfront planning.
As a fan of upfront planning, and a fan of Empirical Software Engineering, I read the report in more detail.
And I'm pleased to report that it is the single worst study I have ever read.
It is so bad I do not want to share it for fear of spreading a mind-virus, but I can share some highlights:
- The author surveyed 600 people and somehow got P values of 0.00004
- By how he specifically defined "upfront planning", the number of engineers he found practicing it is logically impossible
- Every major calculation table had some sort of error
- The author edited a whole bunch of Wikipedia articles to promote himself, and claims his results can also cure smoking
Today I saw the same report doing on The Register and Slashdot, who both accepted it uncritically. And the hundreds of comments were people accepting the study or suggesting a "confounding factor".
...C'mon, people. You're better than this.
I normally write about Empirical Software Engineering because I think using science to study software engineering is important. But just as important is knowing how to protect yourself from bad science. Anybody who's read a couple of research papers would see P<0.00004 as a major, MAJOR red flag.
Being in STEM doesn't make us immune to pseudoscience. And we're forever vulnerable to seeing something that confirms our beliefs and not digging deeper to make sure it's not lies. """
Thank You Captain Obvious
So when the project spec is unclear? And/or changed during project? Abandon ship??
Or, utilize a methodology that's capable in adapting to mid-project changes....?
Hmm.... I wonder....
In reality, I'm not sure I've ever seen the "true agile", only excuses of that form. It has the same feel as there being no 'true communist countries': Ease of implementation (and human nature) matters!
So you've never worked in an environment where developers were trusted as professionals who can organize themselves?
Because that's who pushes "bad agile" so much, who is targeted by all the SAFe propaganda, who tells you to shut up or be fired when you tell them their brilliant ideas about how SCRUM is to be done across the company result in unnecessary friction with no benefits. (I nearly quit there and then because of that talk)
And due to geographical constraints I can't exactly shop around as much as if I lived, let's say, on west coast of USA.
So, "working on my interview skills" doesn't change anything, except maybe grinding leetcode so I can tasteore of the fecal rainbow of corporate agile.
Communism works... at small scales. It just doesn't scale up to country size.
Every business with more than a few staff members is a communist dictatorship. Employees don't vote in their boss. Business are not democracies. Employees don't own their own means of production. They share a communal shared pool of capital resources, which is the very essence of communism.
Every functional business is a success story for communist dictatorships.
Where it breaks down is at large scales. Too much bureaucracy and misaligned incentives degrade the efficiency until it's worse than a bunch of competing organisations voting in a democracy.
I'm saying this because Agile is similar. It too works. Just... not in the way that people try to apply it, at huge scales to large inflexible enterprises that say they want Agile but insist on implementing it using every step of the waterfall process, just renamed.
Agile works, Communism works.
Just not everywhere.
There's an interesting conversation here -- the governance of the countries where it was tried didn't change very much; new tsar, same as the old tsar -- but my main takeaway was that while you might be able to change the economic system, changing the governance requires deeper changes to the underlying culture, which is usually infeasible.
Scandinavia, especially Norway, are IMO closer to marxist communism than the USSR ever was; last I checked around 40% of the economy is state-run. It seems to work pretty well, and it's democratic. But it's been democratic in one form or another for the last twelve hundred years. Even when it was ruled by Denmark or Sweden, there was local democracy. I suspect it would still be democratic even if it were explicitly communist.
=
And so we come back to Agile, which I think is similar: You get the qualities that are already inherent in the team, implementing something like Agile doesn't make a huge difference one way or the other. Except if you're heavy-handed about it, in which case you can absolutely destroy what's already there.