Agile and the long crisis of software
logicmag.io
logicmag.io
Moreover, daily standups, billed as lightweight, low key check-ins,
have become, for some workers, exercises in surveillance.
Particularly when work is decomposed into small parts, workers feel
an obligation to enumerate every task they’ve accomplished. There’s
also pressure for every worker to justify their worth; they are,
after all, employees, who need to be perceived as earning their
salaries.
Similarly, this is why I don't believe that a fully egalitarian group of engineers would work, either: All skill and no direction doth not a project make. Besides, true egalitarianism doesn't really exist. Some will be stronger than others; some will have weaknesses that are complemented by colleagues (hence the value of diversity); some will be brazen, while others shy. Inevitably, a de facto leader will emerge. In my experience, often the loudest, rather than the most effective.
Building complex software is a social problem more than a technical one.
Striving to be agile, flexible, adaptable is a very valid and useful principle.
"Agile" as a noun, the cult-ish process-training-industrial-complex is not at all agile.
Link to the part where he recalls the creation of the Manifesto https://youtu.be/a-BOSpxYJ9M?t=384
It is immediately followed by the difference between agile (adjective) and Agile (noun).
Agile IMO has been perverted by business and middle management to facilitate bean-counting.
I think it's essentially just the default state of working for a business. Any business.
Surveillance by whom? A standup consists of the team and a scrum master, who is usually on the team because most companies can't afford a dedicated one.
I hate the title software "engineer" because engineering is a set of practices and processes that software creation doesn't come close to complying with.
The primary problem is the feature backlog. It's always been that, whether it was called the "Requirements" or the "Functional Requirements" or whatever.
Working out what the product of the effort has to do is hard. It involves understanding the actual end users (not the managers of the process/product), the process/product itself, and the context and environment in which it will operate.
The stakeholders are multiple and on multiple levels. There are managers and auditors that need visibility of the ongoing operations, there are users and consumers that need the actual input or output.
I've never met a "Product Owner" that actually understands the product they are building or where and how it is expected to be deployed and operate. They rarely filter out unnecessary or dangerous "features", but just add them into the overall bucket without knowing where to draw the boundaries.
The "solution" is multi-faceted, but it needs to be directed at this area.
Domain Driven Design is one approach, actually engaging with the actual end users is vital.
Sometimes, ignoring a manager's desire (particularly for "dashboards" and "reports") is very important to keeping the feature backlog manageable.
Understanding the context/environment is also vital. Does this product deal with individual's personal information? How does SOC affect it? Is the process required to be auditable? If it is, what evidence is required for the auditors?
None of this is "software" per se or even "programming", or, even worse, "engineering". It's process and product and understanding what is being built.
While they may not be common, actual product managers do exist. There are a lot of incompetent ones, to be sure, but there are also a number of incompetent engineers.
It’s so nice to deal with experienced product managers and project managers but it’s rare to deal with them.
Where do all these PMs get promoted to? If it is to a "manager of PMs" role, that sounds as if the company is always launching ever more products, and that promotion to a higher level with more pay while remaining a PM isn't an option.
And 15 of the 17 agreed that yes software engineering really is engineering.
This is sad. It's like a professional football player saying they never played for a coach who understood the objective of football. If that's truly the case, no amount of software development best practices or skill will save you from failure.
I have seen product owners who understand the product, and clearly those people exist, as there are successful products in the market. But if you've spent your career playing for the software equivalent of the Detroit Lions, I can see how the industry could appear pretty bleak.
1 comment at https://news.ycombinator.com/item?id=31299834 ("The way it is described in the article highlights a subtle problem - the ability to change requirements as you go", followed by a discussion of that tangent.)
1 comment at https://news.ycombinator.com/item?id=31145030 ("doesn't match my understanding of the history")
2 comments at https://news.ycombinator.com/item?id=30988096 (both concerning the high number of re-posts)
1 comment at https://news.ycombinator.com/item?id=30967432 ("Neat historical overview followed by bizzare conclusions")
1 comment at https://news.ycombinator.com/item?id=30963468 ("Sigh. Another article on "Agile" that makes it read like "SCRUM" == Agile (Sprint, Stand-Up are Scrum terms, not Agile).")
Hey - that's not in the manifesto!
Switch to waterfall, and you'll have problems with that. Try whatever process, and there will be a set of problems that comes from it. Scrum is a flawed process, but there are many flawed processes. A lot of scrum-haters (understandably) argue against scrum, but what I'm missing is the presentation of the alternatives. We are going to be miserable and stuck doing scrum for the coming century unless we can formulate and demonstrate good alternatives. And if we don't want the processes to be upper-management centric, we better formulate the alternative as developers.
The places where I have seen scrum function well (teams delivering features without burnout, without losing grip of the big picture because of constant micro-development etc.) are in teams that are high performing and highly functioning. But of course those teams would have been well functioning and productive with any process.
The allure of scrum (in my opinion) is that it makes teams that would likely have delivered nothing, deliver something, or at least not waste time building the wrong thing. It's a way for management to increase the transparency and lower the risk of software development with poor software developers and/or bad teams. And that, to be honest, isn't something to be scoffed at. Throwing a poor process onto a bad team means you'll get miserable developers and risk having nothing produced. Agile means miserable developers and hopefully some transparency too. Not great, not terrible.
So, it's kind of an "observability tool for teams' production process management"? (to reframe it in a more contemporary fashion)
Edit: I guess we're headed to having pulse, blood pressure, VOmax, weight, daily exercise and so... metrics also displayed along mood and burndown charts. :p
Absolutely, I think thats literally on the label and not in the fine print of the Scrum tin.
Why are developers who work in an agile environment miserable? If you take scrum, doesn't it include retrospectives, with the express purpose of allowing developers to reflect on their misery and to gradually improve their condition?
"No true Scrum/agile" aside, this is the reality of many. The fundamental problem still lays in the power difference between ICs and upper management. Even if it is "inefficient", so are many things done by modern corporates. Showing something can be done more efficiently isn't a guarantee the company will become leaner.
Yes. And yet that often doesn't happen. And that's where one might argue "well then you aren't doing it right", but that's precisely the No True Scotsman argument.
I see teams be miserable, have all the power to do something about it, including buy-in from management, yet they remain miserable. A key feature of any alternative methodology and organization should contain this: It's the role of management to ensure teams aren't miserable. Merely giving teams the tools to do so doesn't make it happen.
In the case of Agile, Scrum, and retrospectives, they all have published definitions, so No True Scotsman doesn't apply.
Are you valuing processes and tools over individuals and interactions? You aren't doing Agile right. (Agile Manifesto)
Does your team not consist of "one Scrum Master, one Product Owner, and Developers [with] no sub-teams or hierarchies [who] internally decide who does what, when, and how?" You aren't doing Scrum right. (Scrum Guide)
Does your retrospective not include "Decide what to do?" You aren't doing retrospectives right. (Agile Retrospectives book.)
In fairness hardly anybody does these things right, so it's tempting to lob No True Scotsman criticisms. But that's just as fallacious. A more accurate criticism might be to say that Agile & co. are too hard to do right, or to say that they require a business environment that doesn't exist in most companies.
"This isn't a One-True-Scotsman fallacy, a true One-True-Scotsman fallacy would be <insert narrower definition>, your claim about this fallacy is fallacious"
I'm pretty sure Agile, Scrum and Retrospectives have multiple published definitions, both broad and narrow.
1) They don't understand Agile, Scrum, or retrospectives, each of which have authoritative definitions (which can be found at agilemanifesto.org, scrum.org, and in the Agile Retrospectives book).¹
2) Furthermore, they don't understand the "No True Scotsman" fallacy.
Correcting a misunderstanding is not "No True Scotsman." If somebody called a car a "horse," and you said, "that's not a horse—a horse has hooves, not wheels," would that be No True Scotsman?
¹You could argue that those definitions are overly broad, or fuzzy, or many other things. But that's not what people in this thread are doing. They're saying horses have wheels, then saying it means horses can't jump.
first thing to improve, get rid of retrospectives
second thing to improve, if having retrospectives have them in such a way that nobody feels like they have to come up with some things you can improve, keep doing etc. etc. every two-three weeks.
It's an hour plus of sitting there mostly listening to other people complain about stuff, which bums me out even if I was otherwise generally happy.
It is a ceremony without actual purpose because usually the deeper things that people bring up are not within that team's power to control.
It gives leadership an excuse for not fixing things or listening because retro is supposed to be the outlet for gripes about the process (despite everyone knowing that major changes never happen coming out of retro).
> It is a ceremony without actual purpose because usually the deeper things that people bring up are not within that team's power to control.
From what you said, it sounds like there are already several things that are making you miserable and that could be addressed in a retrospective.
First, remind each other about the intended purpose of a retrospective, and compare this with how you conduct your retrospectives. Read up on what retrospectives are intended to be. Notice any discrepancies with your retrospectives.
Second, talk about what it would take to make retrospectives a useful event. Maybe, agree that you don't discuss things that are out of your control. Or maybe have a retrospective specifically addressing the things that are outside your immediate control, and ask your scrum master to figure out ways to bring at least some of them under your control. After all, that is scrum master's role, not the micromanagement of the team.
Third, if your retrospectives are an irredeemable waste of time, and there is no way to improve them, agree within your team that you will stop having them. This is probably a last resort — but it's better than to waste your time.
Case in point, I'm working in an engineering capacity with a "Big Mean Secret" pharma company on a contract that is essentially "Client will work with Vendor in an agile manner to <do something vague which could encompass everything or nothing>". The company I work for is nominally a "product" company, not a "services" company, and the contract specifies fixed payment terms and timeframe.
This morning, one of my teams finished implementing some stuff that was a good amount of effort, several moving parts, etc. It was explicitly called out as top priority, and its progress has been tracked by both parties over the past weeks in a shared Jira board.
A few hours later in a biweekly update meeting with said client, we mentioned having finished the feature. The client's project manager responded that actually the person who cared about that feature left the company, so it is no longer in scope. Then they erased it from the project tracking record, and in the same breath complained that so little progress had been made against the other objectives.
And a million other little papercuts like that. It seems that "agile" style process is best confined to small / internal teams re: the "how", but the high level "what" requires comprehensive elucidation from the beginning.
Or is your concern being able to bill for that work?
In practice, agile contracts should be time+materials, or the vender needs to be a stakeholder in defining the scope. You almost always need a sketch of the total solution up front, and a light-weight change-management process. It's been years since I was in that business, but people have been working on this problem. You're not the first.
Actually, I don't see here an issue with doing "agile" or "waterfall". It's a communications issue. Either someone at "Big Mean Secret" company did not manage communications with the client properly, or the client didn't bother to communicate properly when given the opportunity.
Either way, using a "waterfall" methodology would not have helped here. It can be said that maybe "agile" "done right" would have solved this issue, but when is "agile" ever "done right"?
(The last sentence was an over generalization of course)
What I'd like to see is a methodology which produced an objective measure of how overloaded a software dev team was and which could be used to justify an increased headcount. I've never actually witnessed that come out of one of these processes.
The author meant responsibility for the product’s impact on society, I think, but it could also apply to responsibility for technical decisions.
I've been told that gating one story behind another is waterfall. I've been told that having a design story is waterfall. I've been told that architecture documents that are eligible to be revised as the system is built but created before all the code is in place are waterfall. This sort of backlash taken to heart means that anything larger than a developer can do in two weeks is infeasible. This often happens when the team building the software is not considered among the stakeholders of the team's output.
There's a sharp focus on shippable improvements. Well, to me a plugin system is a shippable improvement even if there's no plugins for it yet. A data flow diagram from this week people can use to determine which parts to build next week is an improvement. This isn't a multi-year pre-planned development schedule. This is just having some idea how to integrate the software.
The word "anarchists" fits in its purest sense of tossing the org chart. Yet its common meaning of "bomb thrower" hardly applies.
I typically describe Agile as "A small, stable cadre of ninjas getting it done."
Projects wind up with too much scope, a revolving-door staff, and a dearth of wizards to cut through the technical thicket.
The discussion of "tech debt vs. 'green field' projects" minimizes the threat of legacy code/data to any agile effort.
We need to talk about a "software archaeologist" specialty where one tries to plow through a mound of garbage left by the predecessors and seeks to figure out what prescriptions their therapists had written for them.
Reminds me of one of my favorite quotes about software/computer systems - "we build our computer systems like we build our cities - over time, without a plan, on top of ruins."
The really successful software projects I have been involved with had very reasonable requirements. In that they was not a long list of them and the core problems had plenty of time allocated.
That's the first key in my mind. Limit the scope. Overall. Another thing is to close feedback loops in terms of making sure that there are actually users using releases.
The number one biggest thing that has helped me is to avoid having places for managers or users to add an infinite list of features they want. I usually only need one or two things to work on every 3-5 days. We discuss every thing directly in text (or sometimes voice) chat based on the users testing the working application as it progresses.
I think this is also best for small teams, although recently I have done mainly solo projects.
I know one of the ideas with feature/issue trackers is that there are too many things to keep tabs on without them. But it seems like for a lot of projects, having the tracker means people fail to communicate directly when necessary, create busywork, and do not focus on priorities enough. If there is so much stuff going on that it actually can't fit in the chat room then maybe that's too much stuff.
Obviously things are going to be different for a large team handling something really extensive like Google Chrome, which is actually a full overlay operating system.
Should we just stop developing software? Should we go back to Waterfall?
I know every person reading this could write a dozen pages full of suggestions for how to run a software team, but I really wish the author herself had spent some time suggesting what software engineers should be doing instead of "agile", because that, to me, would be infinitely more interesting.
Also, the rumblings of agile don't really point to anything to do with agile fundamentally, just the fact that people don't quite understand it or that management doesn't wholeheartedly embrace the mess.
"Agile meant that the team didn't really know what they were producing" - that is a Product Owner issue 100%.
So yeah, nothing's perfect but it's probably the least worse of the options we have!
Waterfall seems like a strawman to me. Has the tech industry ever used it? The first reference I could find for it was back in the 70s as what not to do. I don't think waterfall should be used as a defense to scrum.
As for alternatives, you can start with giving engineering teams autonomy and ownership of their work. Gergely Orosz has a good article on project management in tech.
http://www-scf.usc.edu/~csci201/lectures/Lecture11/royce1970...
https://newsletter.pragmaticengineer.com/p/project-managemen...
Waterfall is in no way a "strawman", it's very real, and many shops who claim to be "agile" are still using it today, not to mention the near-ubiquity of its presence in software shops for years prior to agile taking over.
'… we need to accept the possibility that we will iterate during the development, perhaps within a given phase (eg reworking a design until it seems that it will give the required performance), or from one phase back to another (eg to correct a faulty design decision that, during construction, is exposed by a failure to be able to implement it). The biggest iteration is the one where we complete the development and, when we see what we have produced, we decide we need to start again and redefine the user requirements in the light of what we have produced. Such iterations happen in the real world and we wanted to accept this fact and put it in our model. Royce first formulated this idea in what he labelled the waterfall model … page 31
'The "oldest" process model is the waterfall model … This shows a similar progression from inception, through definition, the various stages of design, code, unit testing and integration to final acceptance. The fact that that progression is never smooth and monotonic is acknowledged by drawing little backward arrows suggesting iteration." page 39
1990 Strategies for Software Engineering: The Management of Risk and Quality
https://www.google.com/books/edition/Strategies_for_Software...
It did. That's objective fact. It was prevalent across the industry, that's also objective fact.
Seems like you deny that waterfall model.
Waterfall definitely existed. I've been at places that worked that way. I think Agile (and corruptions of) have replaced waterfall well enough that people no longer think it used to exist.
Ever wondered why so many Open Source projects deliver software vastly superior to commercial alternatives? It’s because in Open Source, non-competent folks don’t tell competent ones what to do.
Compared with the vast, vast array of software we both need and use, open source has failed. Precisely because there’s a gap between what the developer can envision and what is needed. Hence product owners. And a gap between the resources of time and money available and what a single developer can do in their spare time. Hence management.
PS: I also think you should re-examine your disdain for “burger flippers”. Even if it’s a low-autonomy job with much repetitive labor, there are aspects – like dealing with customers under pressure – which might break the average software developer. And people are much more than how they choose or are able to sell their labor. The general lesson here is that there’s usually a lot more to every job and worker than you think.
In many areas we are already doing waterfall, except it is a sin to call it that way and we must pretend we are doing agile.
Let me give you two examples: - Contractual requirements. If you have a contract with a customer to build X, no matter how much agile lingo you use, you are basically doing a waterfall. - Legal requirements. Same as above. For instance, there is very little flexibility in being GDPR compliant.
https://www.linkedin.com/in/miriamposner/ - OK, now it is clear - the author has "immense" experience in both management and software engineering and that's why she tries to describe the situation using the knowledge she has available to herself. What a miserable failure though.
The real question is why people with actual experience in both would read such an article...
And it was not said in relation to agile.
Judging people by their appearance is extremely superficial and guaranteed to bring you tons of problems in any more complex context.
Oh is this is the new measure of success? I better get one then.
Used to be we just pointed to the successful products and companies we created, but I guess we are measuring folks now on mostly pointless pet projects that no one is using?
I must live in a sci-fi horror movie/simulation. Where I'm the only person who remembers actual mature proven repeatable project management methodologies. Like critical path, joint application development, and a dozen others. Whenever I mention this alternate reality, I'm dismissed as a lunatic.
Hello? Does anyone see me? Does anyone else remember Peter Drucker, PMI, and shipping software?
Planning and then executing on that plan in short increments is reasonable. Planning and executing on that plan over months or years, stupidity and insanity.
There is a rule in the world: no single solution can be applied to all problems alike. One-size-fits-all is always, always a lie, no exceptions.
A software project is a problem that needs to be solved according to its needs. The skills required, the communication, infrastructure and metrics always differ.
A good manager will never fully subscribe to any methodology, but instead will be truly "agile", or actually flexible, drawing inspiration from different methodologies where it's needed, knowing their team very well, and recognizing what's needed for their team to get the job done in the best way possible.
Pretty sure the author mixes things up in an attempt to create a narrative.
For "managers", substitute "customers". It's the people paying that want to know the cost up front as much as possible and when things will be ready. The managers are just trying to hold everyone to that (as much as is possible).
SCRUM was a framework that one of the original agile proponents built to make money off developers. It gained popularity not from engineers but from project managers and the like which meant suddenly every project started to use SCRUM. The main reason was that they sold a course which gave a certificate and if there something that a project manager like more than micromanaging it's certification.
In the case of Scrum, it's that "individuals and interactions over tools and processes" line. Scrum did start out as a popular Agile methodology among developers, but because it was the most prescriptive and process-oriented of the methodologies, it got adopted and twisted by management and turned into a tool to reassert authority. Now it reasserts that tools and processes are, in fact, more important than individuals and interactions, but this time with the "Agile" label.
I'd love to try working on a Scrum project that doesn't rely on JIRA. Just notes on a corkboard with a quick "need anything?" conversation in the morning.
Don't you find value in keeping the digital record of the work? For example, if you want to keep track of dependencies that a given task has; or want to get back to the description of a given piece of product that was captured in a ticket (a user story, links to designs, references to other documents, etc.) after some time has passed? Notes on a corkboard are transient; tickets in the Jira are permanent. Isn't there value in that?
BTW, why do people dislike Jira so much? Apart from it being slow or not allowing assignment of tickets to more than one person?
But there's a slight difference between keeping records for accountability (just do the bare minimum to show that work happened as requested/regulated) and keeping records for documentation/action/archival.
The difference shows in the care you take to design and operate those records.
As for Jira: 1/ it IS slow 2/ it IS bloated (visually and functionally) 3/ it does work for a very specific limited number of ways to do things - but never yours 4/ when you start to look at the APIs and data model, you start to understand that it's been designed/built by people not having a clue what it was going to be used for, and by who - and it definitely shows.
Bugzilla is old, clunky; Trac is... extremely focused; GitHub issues are limited. But I would more happily go back working with any of these and plug into additional tooling rather than with Jira.
Granted, none of these allow for a clear understanding of a ticket lifecycle, or a whole project. But that's because this belongs in a narrative, not in a dashboard. And a lot of people don't like to write or value written docs.
Maybe there is a market for "literate programming merged with project/task management" where the narrative is maintained as a source of truth and individual items can be queried (or compiled) out of it. The narrative could based on some of the ideas in this article even: https://www.joelonsoftware.com/2000/10/02/painless-functiona...
That's exactly why. Bad JIRA implementations can easily incur 5-10 seconds of wait time just loading the page alone. JIRA as a tool allows everything under the sun for management to configure, which makes management do exactly that, configure everything under the sun.
It's not unheard of having to tick 15-25 fields + checkboxes + dropdowns (most of which could've been autofilled or cut and made lean) because management likes it that way. I hope I don't have to explain why a crowd largely composed of "I hate doing the same manual work more than once" collectively sighs at this notion.
Then you add onto that a workflow which requires one to open JIRA multiple times, multiple stages etc. and these people end up spending a lot of time on something they feel is incredibly inefficient. Great way to tank morale.
"But that's a problem with the use of the tool, not the tool itself!"
And we've seen this story a thousand times. If tools allow for weird things, weird things happen. Beyond that, I don't really see why people prefer the Atlassian stack which shows similar issues all-over and is a hassle to cobble together, when GitHub and GitLab do these things out of the box. The common complaint is "well GH/GL don't have X and we really need X", when the organization has never tried doing things without X.
>Individuals and interactions over processes and tools
and I would argue Scrum is a process and a rather heavyweight one. Thinking in terms of Scrum instead of your team/members and their preferences is getting it backwards.
I've yet to find a growing/grown company that still embrace chaos as a start to find and nurture their own agile working methods.
It's very much like how companies implement "innovation strategies", being oblivious to the fact that innovation starts with rebellion against the very structure in which it appears.
Uncertainty is not something people embrace easily.
> They were all working hard, that was clear, but there was a fatal flaw: no one really knew what the project was ultimately supposed to look like, or exactly what purpose it was supposed to serve.
sound like scrum though? No product owner, who would have a vision for what the product's purpose was or what it was supposed to eventually look like. No product backlog outlining a path to get to the vision. No sprint reviews with the product owner and stakeholders to inspect whether the product is on the right track. At least not that I saw in the article while quickly skimming through.
What she is describing is some techniques picked from some agile methodologies; but lacking the proper spirit of the game.
If the result isn’t “working software” created with “customer collaboration”, while “responding to change”, then it’s not agile.
There are also twelve principles (https://agilemanifesto.org/principles.html). My favorite is: (7) Working software is the primary measure of progress.
Can't code until the architect has written the spec. The deadline estimated from a Gantt chart by the PM. No feedback from stakeholders during development or the ability to alter requirements as they become better understood. So usually delivering code late that doesn't actually meet the customer's needs.
Scrum was better.
I hear it is now horribly abused (I'm out of the game now) and that's a shame.
EDIT: I saw the most resistance to Scrum from project managers themselves, who did not understand a world without Gantt charts, or changing requirements as you went along.
But the plan, code, test cycle, largely fixed requirements and PMs running everything from a Gantt chart existed in many places to a large extent.
What I anecdotally perceive has changed in the last five years or so is that the no good lying tools that arose from Scrum's popularity have become compelling to upper layers and weaponized against engineering.
Burndown charts. Design documents. The bureaucracy that is JIRA (YAGNI IMO). These each chip away at autonomy.
There was a time when software teams didn't meticulously ticket and estimate everything. Hard for some to fathom, but stuff got done if you worked with honorable craftsman. Just differently. #noestimates feels very long ago.
Sounds like you moved on wisely. Aberrations like these vary of course, but along the rise of large tech giants that demand alignment and are threatened by autonomy, it's become more de facto.
It's sad to see small teams take it as a given for how things are done. I genuinely think it deforms and delays the solutions they're trying to make for their customers.
Enterprise software is appalling at imposing workflows on everything in sight, and they always have pretty graphs and reports for management.
I feel quite strongly that software should serve humans, rather than the other way around. These tools do not exist to help you get your job done, they make you its servant.
Imagine how much more productive we could be if we had tools designed to actually help us. Management could still get nice reports, but some manager might have to knock up a spreadsheet. I can dream...
Yet Microsoft wasn't using a (pseudo)waterfall development model. Quoting Steve McConnell in "Rapid Development (1996) Page 271:
] In addition to providing explicit support for morale, Microsoft gladly trades other factors to keep morale high, sometimes trading theme in ways that would make other companies shudder (Zachary 1994). I've seen then trade methodological purity, programming discipline, control over the product specification, control over the schedule, management visibility -- almost anything to benefit morale.
As I've pointed out before, I've read about software development projects at Apple, Be, Commodore, Data General, id, Infocom, Microsoft, VisiCorp and others. None were using (pseudo)waterfall.
I therefore agree with bdefore that it seems to have been "far from ubiquitous and mostly limited to government or very large corporations", and that the era of waterfall was a mythical beast.
My issue with the standard Agile story repeated here is the lack of any mention of pre-Agile approaches other than waterfall.
Take: "Rather than clearing the way for software developers to build, waterfall gummed up the works with binders of paperwork and endless meetings."
Why is there no mention of any other approach? We can easily point to the Macintosh project, which took 5 years (1979 to 1984). Yet, no waterfall there.
Or, take a closer look at: "Peter Varhol, a technology industry consultant, estimates that in the early 1990s, the average application took three years to develop, from idea to finished product." (That's shorter than the Macintosh project.)
What does that mean? Which industry? What kind of applications? Clearly a lot of "log cabin" PC applications are excluded from that list - just think of all the DB2 and VB applications people wrote in that era. And video games.
In neither case did we do waterfall as people describe it. If anything it was even less process than agile. Sure the gantt charts existed, and sure at a high level things had been drawn out. But in the boots on the ground teams things were a lot more ad hoc. The biggest difference I see from that experience today was that individual developers were handed larger pieces that'd take weeks or more, and had a lot more freedom of action.
Not sure I'd call it better or worse to common scrum setups, but likely a bit of both.
I think many people found software quite mysterious. You had to let the wizards perform their strange magic ;)
I think what agile gives us over that world is better feedback cycles.
He also said that agile was not a substitute for planning.
I guess those lessons weren't learned by everyone...
solve([problems]) 1. Define car(problems). 2. Plan a solution. 3. Implement the solution. 4. Test the solution. 5. Document the results. Add any problems discovered to the list. 6. solve(cdr(problems))
It augments the naive waterfall with continual testing and customer feedback among other things that might surprise you.
Your 6 month project now has to be finished every 2 weeks now working shipped bug free in 2 whole weeks. The scope has been reduced for sure but now everything you do has to be in 2-3days at tops.
We embraced the agility, gave a lot of autonomy to the developers and had good product owners. We used scrum, also a bit of kanban and mashed them up in whatever way worked for us.
The product owners shielded us from the business if they tried to interfere too much, and built trust with all the stakeholders that we were delivering. The sprint reviews were really popular events.
Scrum can work well, as can many other things. It's not so much the methodology itself that causes problems. It's when everyone misses what the actual point of Agile really is.
The abomination is refusing to understand that creative activities can't be done well in constrained setups, while mass-production work very well with a certain rigidity. Toyoda NEVER tried to make Toyota R&D a kanban activity, just the factory part was kanban. Modern neoliberal managers have decided that the same technique they like can scale, while it can't. As a result we see a kind of quality-rise in listing terms, while the final outcome is crapwares monsters no one can understand not master, witch a big batch of side techniques to keep them artificially alive to a certain extent.
Classic sw development have flaws, Agile manifesto was a tentative to mitigate them, but have not much to do with SCRUM nor with what we call agile today.
The issue is that the granularity of a system that should have product management is often larger than the granularity of a single scrum team. You can have several teams working together on a cohesive product. Yet often each team will have it's own product manager acting as PO. This leads to a lot of unnecessary horse trading and suboptimal results.
https://github.com/rayfrankenstein/AITOW/blob/master/README....
Agile was a response to management practices that didn't respect the profession of software development.
Management didn't get the joke when the Waterfall model was used to show exactly what was wrong with those management practices.
Agile was a response to the increasing unrealistic expectations of management, and the widespread (and highly predictable) failure of the Waterfall model.
Somewhere along the line, women were pushed out of the mainstream of programming, and became a minority.
The men who started the Agile movement were a bit creepy, and those in charge now have acknowledged past sins, at least somewhat.
Consultants and Trade shows have commodified the dissent that was Agile programming, slowly turning it insane by forcing it to be a Waterfall again.
As always, people who make powerful tools share some responsibility of how they are used.
The software crisis, is the notion that software is hard to produce on a schedule. People indeed noticed that early on before there even were compilers. Hidden in that notion is a thing that is problematic with software. In order to even have a timeline for "it" to be delivered, you'd have to specify what "it" is. That's a notion that is quite hard. Often such specifications are a combination of wrong, incomplete, and not very useful. And it leads you to an engineering mentality where specifying the thing up front in excruciating detail becomes a mandatory thing. If there's one thing we can learn from the last seven decades of software engineering it is that that is indeed extremely hard and probably futile with the exception of a few domains where anything less is just not acceptable.
The notion of waterfall, which was popular with people that did not read the original paper by Royce (he does not mention the word waterfall but is often blamed for inventing the methodology) in full. Waterfall is the notion of specifying requirements upfront, designing the software, and then implementing it. Royce actually called out in his paper that going through this process multiple times yields better results. So, the key paper on waterfall actually suggests we might want to "iterate". Still worth a read more than 50 years later. Among all the agile nonsense, this is actually a good paper. But you have to read it in context.
You might think of writing software as the actual process of specifying what the software does. The best specification is the source code you produce. It's executable and complete. Whatever it specifies, it does. You might be unhappy with the level of detail but it is a specification.
The contribution of the agile manifesto was acknowledging that reality and the notion that it is more productive to talk about modifying this specification than to come up with some meta specification. You have to build in some moments of reflection and adjustment. So most of the time you are actually specifying changes relative to the existing system rather than the entire system. The only moment that is not true is when you have not written any software yet.
Specifying software changes is what an issue tracker is used for. Bugzilla was one of the first widely used ones and it launched a few years before the agile manifesto got penned down. The insight at the time was a that a good issue tracker (and maybe a wiki) were really all the tools you needed to orchestrate software development. All the rest (scrum, kanban, sprints, planning/estimation meetings, velocity, etc.) was just retrofitted around that notion.
That's it. If you can wrap your head around iterating, you are agile, just like everybody else. The smallest possible iteration is: write an issue, write some code, create a pull request, wait for someone to merge it. Pull requests are great. Most modern software development features multiple pull requests being worked on independent from each other at the same time. Of course the notion of concurrent iterations that aren't coordinated centrally makes the heads of most scrum proponents explode.
There's a lot of great software that gets produced by large organizations, or even distributed organizations (like open source projects), seemingly without a lot of ceremony (scrum or otherwise). Open source is basically software development stripped to its bare essentials. There are no estimation sessions. No sprint plannings. No sprints even. Work happens concurrently with very little central coordination. All there is is an issue tracker, a code repository and exclusive access for those allowed to merge pull requests. Some great software gets produced that way. And I don't think it's accidental either. Some projects seem very consistent with maintaining a predictable frequency of releasing, high quality standards, etc. In fact they put many corporate "agile" teams to shame. Mediocrity is the norm, not the exception in our industry. The software crisis still exists; but mostly in supposedly agile teams still insisting on disguising waterfall as agile. The louder they talk about being agile, the less they generally are.
- Projects always take longer than anyone expects, even when accounting for projects taking longer than anyone expects (Hofstadter's Law)
- The project architecture will reflect the organizational structure of the entity creating the software (Conway's Law)
- The above two points are often presented as jokes but they are not, they're both actually true
- You cannot force two people who do not like one another to pair program
- Pair programming should never be forced on developers, let developers decide when to pair and whom to pair with - this is the autonomy developers seek
- Code reviews are a waste of time and present a great opportunity to destroy team morale
- Software development attracts a lot of assholes and prima donnas who all believe they are the fabled 10x developer
- It is just as important to understand the why of a feature as it is to understand the what - developers need to understand why a feature is relevant and what problem it's solving for users to be able to best implement it
- Invest in automated unit testing, application build, packaging, configuration and deployment. This is your best defense against Hofstadter's Law.
- Some teams have a knack for failure regardless of the software development methodology being utilized. The same is true for application frameworks and development languages.
- Some teams have a knack for success regardless of the software development methodology being utilized. They can also use the "bad" application frameworks and development languages and still be successful.
- Mutual respect and everyone pulling their weight are the best predictors of a team's success
- Just because someone isn't working out well on one team doesn't mean they won't excel on another team. Some people will excel regardless of the team they're placed on. There are also a few, very few, who can't seem to excel anywhere.
- Some people aren't cut out for development and since 1960 we've been trying to figure out who those people are and dissuade them from entering the profession - all to no avail. This is a sink or swim profession and it's tragic to watch people sink - especially when you can't put into words why they're sinking and how to start swimming.
- Outsiders believe developers are anti-social and don't realize modern development is a team sport. The best predictor of success is how well you were able to play team sports. If you always had lots of problems with your team mates then you'll probably have lots of problems with the members of your development team. Of course this isn't 100% accurate, but it's a good predictor. Those playing well in team sports, play in bands, or have been involved in dramatic productions tend to work well. Basically any activity having a group of people work together towards a common goal. Loners do not work well on development teams - they find success on skunk work projects.
- Finally, management is key. If you fundamentally disagree with your management then it's time to move on. There's no need to trash the management when you leave, all you need to say is you don't feel you're aligned with their goals and priorities. Some people may like the management, others may not. Don't take it personally.
- Code reviews are a waste of time and present a great opportunity to
destroy team morale
Great post, but we would have to come to agree on that one somehow, because in Capers Jones books, code reviews are measured to be the #1 most effective method at improving quality. I'd agree they are very easy to do badly, and often overused.
For the first 20 years of my career I worked at 5 different places and so have some experience with different development cultures. They all did code reviews and they all had the same flaw - they focused almost exclusively on all the wrong things. There would be endless arguments about indentation, braces placement, variable naming and so on. A little bit of the time was spent on what actually matters - the implementation. C++ (which was hugely popular in the 90's) made this even worse - now people could argue about whether a function should be overridable (virtual), whether members should be private, and on and on and on and on. After a few years of working on our own internal framework and working with commercial frameworks it dawned on me that nobody got it right. Meanwhile all these inane arguments led to time being wasted, projects missing their dates, team members arguing over stupid crap - ugh! I saw this with multiple teams across multiple organizations. Different people, same problem.
So what happened 20 years ago? That's when what we now call CI/CD tools first started appearing (remember Cruise Control?). With that we saw the advent of automated unit test frameworks, tools that could check the code against coding guidelines, and tools that could measure the amount of CRAP (Change Risk Analysis and Predictions, remember Crap4j?) and assess the security risk the code imposes. All of this is automated. I saw that automating the code reviews could get rid of a lot of the negativity, be objective, and focus on the implementation and quality of the code. This is the way.
When code is pushed to the repo I have evidence of it's quality: I know it conforms to our coding conventions (which I'm not a big believer in, but some people like to die on that hill), it's been tested (we require 40% test coverage and getter/setter tests don't count), it's CRAP score has been assessed, and it's had a security scan looking for vulnerabilities. All automated.
When working with junior developers they get immediate feedback on the quality of their code and can fix it. If they don't understand how to fix it then they reach out to one of us and we can help mentor them as to what is being reported and what the actual issue is. It's all very positive.
What I should have said then was manual code reviews are a waste of time - you want to automate that activity! I don't ever want to see a code review activity scheduled on my calendar!
In my experience the value of code reviews doesn't come from nitpicking those types of things. That's by far the worst part of code reviews and is definitely annoying but mostly becomes not a problem if you employ the abovementioned tools/practices. Rather the value comes from others picking up mistakes in the quality of your implementation that arises from some gap in knowledge that the reviewer is able to pick up on and fill in. And to that end they're absolutely essential. They're not simply a box ticking exercise either - the amount of code reviews I've both done and received that picked up on actually important issues is in the high double digit percentages. I would take no code reviews to be a major red flag.
The ‘weirdos’ in the first paragraphs were primarily nerdy men. The landscape has changed tremendously since then - Agile is terrible for many things, but it does not incite racism or bias. Perhaps the only bad thing it does not do.
I was hoping for a better ending to this read.
I personally don’t think it’s ok to refer to a person like that, it’s a rather gross oversimplification of a human being.
We ought to be more sophisticated than that.
Embarrassing.
That explains an awful lot. When people think that it's enough that they're right, bad things happen. Agile doesn't actually need a theory of society to understand that, just a little humility.
Author misses the part where the majority of the working force is also "white men" who see no benefit from this either, because it turns out, the decisions of management aren't for "white men" or even "old white men". The decisions are for "our management".
Please, if you are writing an US-centric article, state it in the title. Example: "Agile and the Long Crisis of Software in the USA".
I saw the transition of the industry from a complete waterfall process to the overhyping of agile, the pushback and now the land where agile as per the manifesto is more-or-less a given. The old ways were so much worse...
The bad sprouts from agile are the multitude of processes and bullshit methodologies that tried to ride the hype and create "The Agile Way". Scrum and others are a side-effect of this and I'm very glad to see Scrum being shunned for the past 4-6 years and instead engineers being attracted towards much more lightweight processes.
The problem with agile is that it only works when people had experiences, bad and good, and a team can tailor how their agile process should look like given their constraints. Throwing a bunch of juniors under an agile methodology and telling them what's right or not only creates friction and useless rituals, you need experienced people (PMs, EMs and at least one engineer) to cut the bullshit and stick to the points that matter: collaboration, communication, re-assessment of priorities, etc.
If you ask 10 people what agile "really" means you'll get 10 different answers, many quite incompatible with each other.
Beyond that, 10 people might find that the processes they've refined from Agile really do differ. That might be because of differing (and broken) application of Agile itself, which is a common claim. It might be, though, that continuous improvement of process has lead different teams in different directions that function well or well enough for those teams.
You could replace the entire thing with "shorter feedback loops" and it would be significantly more meaningful.
If you actually read it it really is just a bunch of words that doesn't specify much of anything at all. To me it reads as something incredibly vague and hand-wavy and how on earth modern Agile has been inferred out of the text of the agile manifesto is somewhat beyond me.
It's vague exactly because it's a manifesto, not a handbook on how to apply it.
> We’ve long known that eliminating bureaucracy, hierarchy, and documentation feels great, until you’re the person who needs rules for protection.
I think the article eventually makes good arguments for this, at least if you are willing to say that code review and pair programming are things that spring out of the spirit of the manifesto.
Basically they author is claiming that the goal of the agile manifesto is to remove oversight from developers so that they can focus on "the project". The downside is that you are losing oversight of bad behaviour, which gives space for work-place bullying. (and the other downside argued is that developers, who are also stakeholders, do not have a voice in questioning the utility of the project overall).
1. Mike (Miguel) Beedle was latino.
2. "Al Tenhundfeld, one of the manifesto’s authors" - NOPE.
I could go on but what's the point.
If you want to learn about agile I recommend reading Alistair Cockburn, James Coplien and Craig Larman. They're old school and know the history of this stuff amazingly well.
That said, however, the article is correct to assert that Agile, as implemented in real organizations, addresses the corporation's needs and only those needs. Otherwise, it wouldn't have been adopted at all, but dismissed as the snivelings of whiny insubordinate programmers who forgot just who signs the checks.
That we've gotten to this point must also be old white men's fault.
The usual rules apply, there is no silver bullet, nothing is all bad. Pick the most sensible process for the job and adapt as necessary.