In praise of SWARMing | Hacker News Reader

In praise of SWARMing

dannorth.net

45 pointsadrianhoward18 comments

> Many proponents of these methods have a religiosity about them.

Guilty as charged.

> Their method works; if you don’t believe this you are misguided, misinformed, or just antagonistic

Dismissing anyone's experiences as false, whether directly or by necessary implication, is bound to create a fair amount of ill-will.

I get lectured down to on Hacker News a lot by people who've had a bad time with scrum, or TDD, or pairing, or some version of these that they can imagine themselves disliking.

It's galling for everybody: here I am, telling them that I have experienced these things very differently. Here they are, telling me they have experienced these things very differently. Everyone subconsciously concludes that the other party is essentially calling them a liar and it kinda goes downhill from there.

Don't feel bad that you didn't know SWARM (Scaling Without A Religious Methodology) because the author made it up for the article.

> The most successful transformations I have experienced—the ones where you walk into the office and there is a tangible difference in the energy and interactions between people, where the commercial and management stakeholders are as excited and invested as the technology stakeholders, where everyone agrees on the metrics that matter and those metrics are trending in the right direction(...). And they do all this without recourse to canned methods or certification Ponzi schemes. (...) My argument isn’t that packaged scaling methods are unhelpful per se, rather that they are neither necessary nor sufficient for successful transformation. They can be anything from a useful starting point to an expensive distraction, but one thing they are not is a solution.

I am seeing more and more people involved in making organizations and teams more efficient come back to how the canned / certifications methods are not sufficient by themselves. I am on the same page, noticing how on the project level the actual project management method (like Scrum, Kanban etc) are no the main predictor of success. The main predictors are getting similar basics right, like Dan North summarized in his post. https://news.ycombinator.com/item?id=17710248

The most successful “transformations” or running of agile projects I’ve seen have been because key people understand the value in iteration. And they organise the entire operation around being able to iterate on the product.

I’ve seen organisations do all the rituals, hold all the meetings, do certification courses but they still don’t wrap their heads around the idea of creating shippable product iterations at least once a month. They just end up dividing the work they were already going to do into arbitrary 2 week “sprints” and wonder why they’re not being efffective.

The pendulum has swung a long way. It used to be that scaling a successful organization up was supposed to mean forgetting what was successful at a small scale and trusting the experience of people who succeeded at a larger scale. In other words, read a book, ignore your instincts and experience that were formed in a small-scale environment, and do what the experts prescribed based on their own experience in larger organizations. Those were dark days. People who followed what seemed like the foolish path, trusting their instincts and limited experience, making things up as they went along, not trusting the software engineering "knowledge" that had been supposedly accumulated and passed down, seemed to be much more successful than people who followed the apparently modest and prudent path of trying to stand on the shoulders of giants.

The utter worthlessness in practice of what we read in those big, heavy (and expensive) books, and the leadership failures of the people who relied on those books and touted their own mastery of them, suggests the "experts" were publishing ideation and speculation, not reality. Also their elaborate models of "maturity" were quite similar to the model of maturity we had as college freshman when we measured maturity by "purity tests." For non-Americans, these were ludicrously extensive checklists of sexual and drug-related acts that you would fill out with your friends to see who had the highest total. In retrospect, I don't see any difference between this and measuring software development maturity by the sheer number of possibly helpful practices your organization has adopted, without asking whether they're worth bothering with in the first place.

An alternative explanation for the failures of 'big methodology' is the shift in workplace, professional and career trends. Monoculture isn't possible with a diverse, agile workforce and unless everyone's on the same page those big, prescriptive methodologies just don't work. "this is how we do things around here" doesn't work if the newcomer you're preaching to is likely to up sticks in a year or two, and you yourself have only been 'here' a year or two. But that's the makeup of most organisations today.

I'm not saying diversity and mobility is a bad thing, but it does explain why those approaches perhaps worked fine for the authors in the time period they experienced it, but don't work today.

> Monoculture isn't possible with a diverse, agile workforce and unless everyone's on the same page those big, prescriptive methodologies just don't work

This is a really interesting insight. I don't have much personal reference to compare to the software dev cultures in e.g. the 80s or 90s, but there is definitely a shift towards software engineers having more autonomy, more connected with the customers, often being customers of the software themselves.

I don't know — I came into the software dev world in 2000, and we worked in cubicles and planned in slow waterfall cycles. Integration happened very late in the development process. The managers often wore suits. Programmers didn't talk to clients. In that environment, the old-fashioned processes were still not better than people just making it up as they went along, which is a pretty damning assessment of "decades of progress."

I mean, this should be tempered by Keynes' observation that "Practical men who believe themselves to be quite exempt from any intellectual influence, are usually the slaves of some defunct [he said economist, fill in whatever field applies]." The ideas came from somewhere. The people who were "just making it up" distinguished between different forms of testing, used some form of project planning, and followed some kind of waterfall process that was internally iterative. These ideas were found in the experts' books. The difference was that the seat-of-their-parts people left out 80-90% of what the experts said was required for a successful software process, and they produced the same quality software faster and cheaper than people who studied the books and followed the recommendations faithfully.

So maybe there was an incremental improvement as more and more ideas were crystallized and propagated through the development community, and the experts erred only in their maximalism.

One of the things I've always liked about XP (and is almost never discussed) is that it isn't a process. It's a meta-process. The idea of XP was to come up with the smallest number of "practices", which when executed with attention would generate a successful process. That's one of the reasons why XP signed on to the Agile Manifesto -- people over processes and all that rot. Unfortunately I can't find a link where he describes this which isn't behind a paywall. A lot of people probably don't realise that the idea of using generative patterns in programming originated from Beck and Cunningham [1] -- I often avoid mentioning it as it sometimes fuels the hate fires :-P Kent Beck was quite interested in using this kind of generative pattern language to describe things that would otherwise require ridiculous amounts of text to describe. Instead of describing the actual solution you need, you describe a series of patterns which help you generate the solution. As an aside, you can see how people have taken the "pattern" ball, run with it and ended up going in the opposite direction ;-)

And as you say, XP (and especially Scrum!) were built to help you solve very specific problems. In the case of XP, how to spread out requirements and design discovery across development so as to reduce risk. In the case of Scrum, how to decrease communication overhead on teams that were otherwise operating well (Originally Scrum had no development practices -- it was assumed that you already had development practices that were working well for you).

As I get older it's easy to see a pattern: the inception of a good idea, the successful adoption by a small number of teams, the spread of fame, the religious adoption of a version of the idea championed by the most charismatic bloggers, the loss of the original idea as the masses cargo cult their implementations based on religious doctrine.

[1] - http://c2.com/doc/oopsla87.html

Starting to see this damn near everywhere, macro and micro scale stuff. Especially true on individual projects where team members cycle out over time. Often they end up looking like that monkey ladder experiment where we're very clearly tracking towards a goal, but nobody is at all clear on why.

> It's a meta-process.

I was told this early in my time in Labs: "Pivotal is a meta-process shop".

My version is: "Pivotal Labs is a debating society which produces code as a by-product".

It works. We spend a lot of time reflecting and overall, we keep improving. It's the first place I've worked at where I can see continuous improvement occurring. Not the slogans about improvement, not the Department of the Bureau of Efficacious Improvementology, but actual hey-this-is-better-than-it-was-6-months-ago improvement.

> As I get older it's easy to see a pattern

You might find this parable familiar: https://www.ckwop.me.uk/Meditation-driven-development.html

Ha ha! That's a great read. I hadn't seen it before. Thanks. One of the things I've heard about the fabled C3 project was that before Kent Beck was brought in, the team was incredibly unhappy. He introduced XP and it changed the attitude of the team. (BTW, I've talked with a few people who were on the team and they have verified this). I've had one phenomenal success with XP -- the first one I worked on, and it was a similar situation. We removed a manager that was keeping the developers under his thumb and let the programmers write code.

Since then, I've had successes and failures with approximately XP shaped teams, but I've never even come close to that initial success (85 KLOC of code in a completely new coding environment, with between 3-6 developers over a 5 month period with exactly 12 bugs at the end. Won an award at Comdex and the core technology developed brought in hundreds of millions to the company). The team was rocking when we delivered, but big companies being big companies, they split us up ;-).

Many times I've thought back on that time and wondered, "What exactly is the difference between then and now?" Many times I've come to the conclusion, "The team was ready to work together. They had suffered so badly before and they were so happy to have their shackles removed, that they couldn't help but succeed".

Don't get me wrong. I still feel that XP can work extremely well -- after all, it did. And when I look at problem after problem after problem in other situations, I can point to exactly what is going wrong. But the reality is that some people just aren't going to do it the way they need to do it. That's a people problem, not a technical problem.

I often say when I go into a job, "I know exactly 1 way to succeed dramatically. That doesn't mean that there aren't other ways to do it. It just means that I don't know what they are". Inevitably people ask me to succeed in ways I don't know (in advance) how to succeed in. That's life, isn't it ;-)

Pivotal Labs sounds like fun :-) I hope it keeps going like that for you.

Weinberg said about consulting that no matter what they tell you, no matter what the contract says, no matter what else is happening, it is always a people thing. That's definitely been my experience.

Sometimes everything crackles. But while you can't make lightning strike on demand, you can definitely drive to where the thunder is. I think we try to do that.

Labs was a real education. These days I am in R&D. I like to say that building Cloud Foundry was a process of incremental humblings: "Ohhhh, that's what our clients were telling us about when we brushed off their questions about big projects". But we at least have the necessary lack of shame to ask questions about our oversights and get better.

We are always hiring, if you want to see it for yourself.

Just yesterday I was talking with a friend about hard skills of humans and as it appears how it isn't interesting anymore. Maybe this is out of scope of this article. Dan talks about getting the right people around a table to gain some traction, but that's it. Has the thinking about how individuals function in a team or company become out of favour? It feels like we currently just accept that we have some natural talented people person in the frontline that will figure out how to handle individuals. Scrum masters seem to be very helpless with solving conflicts or improving teams apart from random strategies they found in a cool textbook. There seems no serious education. And that leads back to the beginning of the article. Many agile people sell snake oil, scrum masters buy it and try to sell it to someone else.

[I’m the author but not the OP]

I agree team skills are important and I discuss them extensively elsewhere, for instance in this talk (https://youtu.be/lvs7VEsQzKY), but that is orthogonal to what I’m talking about in this article. Those skills are valuablein any case!

Your comment about “how people function in... a company” is key, however. That’s why I talk about building a leader-leader organisation of the kind David Marquet describes in “Turn the Ship Around”.

As you say, you can’t just rely on getting the right people round the table or having charismatic individuals who are “good people people”. You have to build resilience into the organisational structures so you can federate decision-making away from the bottleneck of senior management, and give people the autonomy to make good local decisions.

Dan, please add author names to the suggested books!

Oh, where did I miss that? I’ll go back and check.

Ah, I just spotted where you meant. I’ve changed it to a bullet list with authors. Hope that helps.

Reply on news.ycombinator.com