Scrum has failed the developers
ageling.substack.com
ageling.substack.com
People saying they're using SCRUM (by this I mean "literally saying 'SCRUM'") are a dying breed and if they're not then they're cargo culting and I'm not sure why this is more of a big news than management cargo culting waterfall or kanban.
I used to be one of these process experts selling methodologies to whoever was ready to pay, from SCRUM to lean startup.
After doing this full time for 2 years the only conclusion is that none of these processes hold long enough. Why?
1/ every product / team / output is different. some teams should be ticket driven, other exploratory driven, and each of these methodologies apply to one situation only 2/ high turnover in the industry. meaning that new people come and organically change the new flavor of the day, the dynamic of the teams and the throughput of the team 3/ all of the best teams I've worked in the industry (from seed stage to FAANG through late stage startups, and even F50 companies), just do WHATEVER that works for them and don't comply with the flavor of the day (except for the looks of it). They all use a shared core set of values (deserves it's own blog post), and you can find this core set of values in most agile methodologies, buried behind ritualistic behaviors
anyway. I could go on.
Just let it go, and if you're working in one of these last companies applying SCRUM unironically, you're probably not in a high performing team. Time to move
I worked with ActiveColab in 2007, Skype 2007, Yammer 2009, Trello 2011, Pivotal Tracker 2013, Trello 2016, Confluence 2022, Slack 2013, Google Meet, and sometimes I think, scrum became _less-relevant_ over the years as more advanced product management tools became the norm and the product manager role matured by leveraging them.
These days, it's not rare to see lead developers manage kanban-like boards very effectively, releasing on time, with grace, without the need of a scrum master to coordinate efforts.
I do like asynchronous scrum daily standups using http://geekbot.com on slack, when on-site or/and distributed and doing sprints. I seen this work well on startups going from pre-seed to series B.
Personally, I am fascinated with team dynamics and how they've changed over the years. We are definitely living the best of times as a developer and I still see sparkles of well-applied scrum every now and then that works nicely.
Sadly, it's also common to see such kanban teams endlessly winging it and slowly losing sight of what they were trying to accomplish, at the same time burning out their teams on an endless stream of tickets and testing without ever taking time to reflect and course correct on their goals.
That's sort of a tautology, right? If it's 'good enough', that implies it's a good process. In my experience, Scrum is a good enough process, with very little wasted overhead. It keeps the team focused, limits their in-flight work, unblocks them and offers regular iterations with feedback.
I'd agree that over-optimization is sometimes a problem, but when something as simple as scrum fails, it's usually down to the basics, like poor meeting practices, or micromanagement, or something outside of the development process entirely, like badly underbidding the project. No amount of process will save a project that was doomed from the start due to poor budgeting of time or money.
And yeah you said it well at the end. There are many other things that I believe are more relevant than the process, but you do need a process teams can follow.
I don’t think this is the consultant’s fault, either. Usually the company doesn’t want to pay the money or time to do this.
(I realize that many of my colleagues don't do this. Fie, I say. Fie!)
I've actually done a GREAT job. These teams shipped.
But I realized that they didn't ship because of the actual methodologies but the core set of values we implemented as a team.
ergo, these methodologies are a sham.
I was responding to @scott_w saying that process change not taking root isn't the consultant's fault, because companies don't want to pay for good process change. I disagree: I think it's the consultants job to be clear about what's needed to succeed, and not work for companies that don't want to pay for good work.
Regarding methodologies, I agree that successful teams internalize the principles behind their methods and leave the by-the-book methodologies behind. I think you're being overly cynical, though. Beginners need a concrete place to start. It's like saying cookbooks are a sham because expert chefs don't need them.
The turnover issue we're seeing in the industry definitely doesn't help with seniors leading the way. which ends up being a problem top to bottom
What agile/scrum was supposed to do was kill waterfall, and provide more transparency in how a project it product was actually proceeding with demonstrations of execution or lack thereof.
What I'm saying is that I was cargo culting too, because I personally believed in these methodologies.
But I realized after seeing many situations, that success is not linked to the usage of these methodologies
But, trying to distill all the formalistic nonsense into usable points, they actually do have a few. My personal takeaways:
- try to ship faster (try daily for a webapp), but do what feels right for you
- have some task board and scrub it regularly (once every week or two, whatever)
- learn to use ballpark estimates (1 hour, 1 day, 1 week - or whatever else floats your boat. Fibonacci, none of this stuff will ever be realistic - just get the order of magnitude)
- have short meetings about blockers, but schedule stuff for later the moment someone starts banging on for too long
- yeah, and try to break stuff down into smaller, more manageable pieces, the spec, the PR's, pretty much everything
No idea if this is "SCRUM" or not, but this has worked for me so far...
1) do you have retrospectives, and what typically happens there?
2) have you ever changed anything, as a result of the feedback given by developers at the retrospective? please provide a specific example.
If they say "no" to either of these questions, it is the bad kind of scrum.
If they say "yes", and give any non-bullshit answer to the second one (for example "we had 2-week sprints but then developers changed it to 3-weeks" or vice versa; the exact details are not important), it has a chance to be the good kind of scrum.
If someone says "we don't do retrospectives, because they are a waste of time", they are kinda right, but for the wrong reasons. The retrospective is where developers provide feedback about the process, and can choose to change it. If you have a strictly top-down managed company with zero developer autonomy, in such case having retrospectives is indeed a waste of time, because no matter what the developers might say, nothing ever changes as a consequence. But that doesn't mean that the retrospectives suck; it just means that the company sucks.
Why would scrum be a hindrance to that? I don’t know if any principle, artifact or event that is there to reduce team autonomy or to prevent the team from bouncing ideas of each other. And it seems scrum is intended to let the team procede whichever way they deem best.
By Pareto principle: 80% of times when I see "The tool X sucks" it is because people are using wrong tool or they are using tool wrong.
Then there was a change in management, and the new manager decided to "introduce scrum" to the whole company. We were told to stop doing what we did previously, and instead to do what the new manager called scrum. Unsurprisingly, it was the same parody of scrum that most companies use. The productivity plummeted, and the developers in our team gradually left the company.
I suspect that scrum works well when it is introduced by the developers, and works horribly when it is introduced by the managers.
I agree that is the way it is often, but that is to the argument to the poster before: not SCRUM anymore.
I call it the no true scrumsman fallacy.
Doing it "properly" means doing a series of mini waterfalls. Thats just how it was designed.
I'd say it functions as a poor but semi functioning compromise between an intransigent waterfall management and an agile team.
I'm not saying such companies don't exist, only that I've never personally worked for them.
I use "agile" as a filtering term these days. If the job description mentions being an agile shop, I know that job isn't for me. If they use agile principles but don't advertise it, I want to see what the process actually is before deciding to accept the position.
Because at most companies, you have a simple choice: follow the prescribed procedures, or work somewhere else. I have absolutely quit jobs that were too egregious, but that doesn't change the situation at that company.
Put all that aside, scrum/agile is actually pretty good on tracking small and simple tasks. So if the whole project can be broken down to nice and small tasks and the estimates were accurate, agile/scrum will work perfectly. Unfortunately all the project I've ever involved were complicated, some were even super large. We humans are pretty bad at estimating and no surprise most estimations were way off. Also, there were huge tasks that could not be broken down. That's exactly where the short coming of scrum/agile. Also, in most projects there were for sure huge, hard and important tasks that needed to be solved as soon as possible. But in scrum/agile, the team tends to pick the small, simple and easy tasks first as it would make the metrics nicer. So when they finally have to pick those important hard and huge tasks, it would be most likely way too late as there were already too much done in the wrong way.
I've come across tasks that were difficult to break down and required a bit of creativity.
Never come across one that was impossible though.
In reality it has been used by management to export their responsibilities onto developers while not ceding control.
With scrum, planning and estimation are now done by developers. But management still reserves the right to reschedule mid-sprint for the all so urgent pop ups they need handled.
The only thing developers can do to make scrum work is be completely devoted to the process and not the business, but that is a career limiting move; scrum masters are put in positions to either be ineffective or be ablative armor for their teams.
When it appeared it was a breath of fresh air: minimal, explainable in 5 minutes, just accumulation of common sense.
You don't even need to declare "We're working in SCRUM" - the important feature is possibility to do "SCRUM in stealth".
SCRUM went wrong when it became "process", when SCRUM masters/evangelists appeared. They started to teach "right SCRUM practices" and SCRUM became a bureaucratic ritual not distinguishable from RUP.
If you’re in a high uncertainty environment with loads of market risk, by all means Agile away.
But if you’re designing and developing something fairly well understood… something more oriented towards a better understanding up front with bigger customer delivery increments is completely fine.
There in lies the problem; building software is not like building a house.
I've seen many projects at Big Co fail to deliver on time, budget or quality because senior managers believed that everything is "fairly well understood" up front when just wasn't true.
That mistake leads to treating estimates as concrete delivery times, scaling the development team before it's ready and a refusal to fix things that hurt productivity because they're not on the unchangeable timeline.
The author is correct Scrum is usually sabotaged by outside influences trying to turn it into waterfall.
If IBM couldn't make waterfall work in the 1970s it's very unlikely that will work anyone's big project today either.
Or rather, they do go wrong, but they go wrong for reasons outside of the scope that a development process considers. Instead, I argue the issue is related more with the way that managers and employees interact, and the expectations they both bring to the table.
More explicitly, I am arguing that if the development process changed at a company with a manager running agile processes, the manager won't necessarily suck less if they were already bad. Likewise to the employee.
My suggested action is to treat this as a question around social environment, and tweak accordingly. This would be different for each company I would expect.
It helped that because it was such a different process, they largely left us alone. They loved the regular retrospectives. They were awed to see feedback from the previous retrospective made real by the next one.
By the end we delivered more for less and it actually did what the stakeholders cared about (not what people thought they wanted when we started and didn't understand it).
There is one thing in particular that no one here is addressing. Underperforming developers. Every comment assumes that all developers are fully equipped to excel if management would just get out of their way. Um, no. I've seen extremely knowledgeable and talented developers who, left to their own devices, fail miserably. You "empower" them to do amazing thing and set them up for success and they get almost nothing done. If there's no one holding them accountable, they just coast.
The truth is that some people and teams need the extreme rigidity and process that comes with scrum or they will simply not ship. Other teams will die in a fiery explosion or wither quietly into attrition if you force process on them. The best you can do is set a foundation and course-correct as the team succeeds or fails. You might end up with very little meetings or 2x scrum ceremonies. Whatever it takes for the team to be productive and happy.
There is no "One Process to rule them all" and leaving developers total freedom to do whatever they want is definitely NOT the answer. The best process for a team will likely change over time due to many factors like company stage, # of members, their experience levels, their commitment levels, their personalities, etc... The job of a manager is to tailor the team's process to what achieves both the business goals and keeps people happy. It's a difficult and thankless job which few do well.
Trying to push off HR problems on a software delivery process is the most dev take I’ve ever heard.
What do people writing realtime flight control systems do?
What do quants writing investment strategy code do?
Do they all do the same thing? Would they call their process a ritualised behaviour?
I would like to break the cycle and end my fellow developers’ misery by helping willing teams and orgs work better.
How could I do this? How is it called? I certainly don’t want to be another Agile(tm) peddler. But I also can’t go back to being a code monkey.
And before people here reply that I have to find better orgs to work at: while I believe they exist, they’re so rare that this isn’t possible for 99% of people. In my over 10 years in software, I have never interviewed with or worked at one of these elusive better orgs. I’m also not in the US which might be a factor. Anyway, I’d rather have a different solution to this problem than “work somewhere exceptionally hard to find”
I'm not sure I quite understand how it makes sense.
From my personal perspective, the sprint model is an okay idea if you're working greenfield, or at least in full control of your deliverables.
I'm wondering how it fits into any sort of dev process where third parties are involved. For the same conceptual change, some will be "send us an email with the ID from the test API call you made, and we'll tell you if it looks right" taking a few hours. Others will demand a huge stack of formal test cases, weekly meetings, and a cycle time of two weeks to even get back what they didn't like about your test requests. How do you possibly slice that into two week intervals?
So it probably means that you will have daily meetings and jira tickets...
These environments will be label 'Agile', 'Extreme' or whatever label can attract talent. No manner of 'Process X sucks' blog posts will change this. If the Hacker-News Scrum hating community comes up with a "better named process", or a successful manifesto advocating the 'end of processes', organizations led by sociopathic management will quickly adapt these slogans and then distort the environment to their likings.
If management is beating you with sticks because you missed getting your story points done by end of sprint, the problem isn't a process that measures stories by points, and has biweekly sprints where velocity can be measured; its having management that is willing to beat developers with sticks.
* Narcistic Personality Disorder
If it really worked, managers would also be subject to it.
I've been in environments where it creates fake pressure. Standups where you have to say something for fear that you'll be axed if you say nothing. Questions about whether a ticket is going to get done by Friday. I call these fake pressure because it becomes obvious that there is no teeth to them. People get away with doing nothing and making up a few sentences at standup. People say a ticket is going to be done, but it roles over through two or three sprints.
It's always been a sham for me. You can't point at the Agile Manifesto and say that we're focused on process over interactions. You can't point out that we need a Product Owner who is focused communicating the needs of real users. You can't point out that standups should be for the team's benefit and not status updates to project managers who are barely involved.
Still, I'm a firm believer that the Agile Manifesto makes some good points. I believe that there are elements of Scrum that are really beneficial if implement well. I believe all of it is subverted by management that wants a semblance of control/power. Most of the workers don't care whether it's subverted, they're just there to be paid while doing as little work as possible.
So, who's left to change it? A few righteous agitators? Who are they going to agitate? The managers who have control aren't going to give up the reins. The hoi polloi will ignore anything that takes extra effort.
Saying the scrum master needs to be accountable is true. Everyone should be accountable for something. However, there's simply no way for that to happen in many environments.
I've had scrum masters that were basically just required to gather status updates and report them back to the CTO. I've had contractor scrum masters that were hired because Scrum said they had to have that role but they ignored the part of Scrum that said there was a product owner role. I've had scrum masters that were just engineering managers who ignored their engineering manager roles to be the one who screen shared the Jira during standups, including one who was a Director of Engineering.
All of those scrum masters were installed by the people in power not to be held accountable for anything. Most of those scrum masters, like all the other workers, just want to get by with doing as little as possible while being paid.
It's just too common for a task to turn into an "epic" mid-sprint. I only see value in sprints as regular checkpoints but the expectation that all tasks are finished after 2 weeks is just contrary to reality of software development in large organisations.
...and we come back to one of agile principles: "the best architectures, requirements, and designs emerge from self-organizing teams". I think the scrum itself says to not confuse the job title with the role and I believe that's what happening in many cases. But it all looks good on paper, we have scrum because we hired scrum master so management is happy whereas in reality it is not true - I saw it many times.
I do think Scrum took off because it is the opposite of XP, it doesn't tell developers any development practices (Pair programming etc.) so developers didn't feel bullied and said "Yeah, ok, lets do your Scrum". But it backfired.
I for one am glad that there are Scrum masters and POs to deal with all the politics and project nonsense that I don't have any interest in being part of.
A previous commenter had it right imho. Its not that the approach isn't working, its that it has been co-opted by management and requires separation to actually succeed. Another note would be that agile itself says:
- non team members shouldn't be at stand ups etc.
- points shouldn't be used for evaluation of performance (good luck with that)
- if management joins anything other than the demo, they should not speak during the meeting (I've had some success here but also always been labeled a curiosity when I push management to be silent)
So some assortment of reasonable starting points, with their pros and cons described, would be valuable.
You can use Scrum, Kanban, ... as starting points. Just don't implement them mindlessly like most companies do.
These are 2 of the 4 tenets in the manifesto for agile development [0]. They go on by saying:
"That is, while there is value in the items on the right, we value the items on the left more."
Essentially, choose tools. Follow processes. Periodically and permanently reevaluate. If following the plan creates an impediment to interactions, or makes the people dissatisfied, respond to these changes. Rinse and repeat.
People first, tools second, and then process.
While scrum/agile does not = more developers, it can be leveraged to actually grow a developer's career and skills when done well.
In a world of continously created saas pursuing bits of walletshare and department budgets endlessly I would say there is a small % of tools that are specialized and uniquely required. There's also a TON of tools that are preference based selection.
Michelangelo sculpted with one set of tools and painted with another. Klimt may have had some overlap with one and none with the other set. Neither was being doggedly pursued by salespeople or having decisions about tools made by their bosses.
IMO: Its not an apt metaphor in the slightest.
And don't get pulled into success. Most anything can and will work with eager teams at the start. Life has a way of changing without you noticing.
That said.
I have never been anywhere where the process does not have room for improvement.
But often improvement seems impossible for structural reasons, for a team that is not empowered, where people are checked out, or not forthright due to whatever reasons. And like you say, people go through the motions for the sake of doing it, leading to wasted time and more frustration.
Eliminating the retrospective is the easy choice to avoid that.