Why products built by large teams fail so often
blog.mojotech.com
blog.mojotech.com
To set the stage this was when web services where coming into fashion and he was a big proponent of them, he said that he likes to focus teams around business domains (e.g accounts receivable, road logistics, etc.) then let them control their destiny. He said that he liked a service based architecture because it allowed him to build Amazon scale software, a piece at a time and more importantly the nugget that I really took away from the conversation was that you cant stop bad software from being written, but you can isolate bad software from the rest of the system which allows you to measure all kinds of things that give you an overall metric on the quality of that software. Things like time to defect resolution, uptime as well as true business metrics given that they are aligned with a vertical (i.e did the software empower the core business goal). If the numbers are lacking the software is bad plain and simple. Isolation makes it easy to replace.
I don't think enough people who architect large systems give thought to how do we protect the overall system from bad code which is a certainty to happen on large projects because they necessitate large teams. As developers and architects we tend to focus on elegant designs, but the overall support of an application thru it's life is anything but elegant.
Hopefully no one is using a large team to build a small product. If so, yeah, don't do that. But you're going to get into different problems building large products with small teams.
Ultimately there's no silver bullet, the best you can do is maintain awareness of the tradeoffs, identify and deal with problems as they arise, and otherwise do the best you can to maximize the likelihood of success and minimize failure.
Sometimes people feel like they've found a silver bullet just by identifying the problems, in which the silver bullet is "well just don't do that then". In contrast, I feel like everything that's nontrivial in life consists of choosing from a set of compromises and tradeoffs. There's a natural tendency to feel the pain of the compromises and assume that if only we weren't making those compromises, everything would be great, without taking into account the tradeoffs and compromises that alternate approaches would create.
A 'small team' making an MVP, and then moving on incrementally from there, adding more people to do peripheral features as the project progress might seem like a good way to overcome some inherent 'monolithic' risks - as well as keep costs down at the start.
It takes an open mind from management to grasp that building complex software is not like building a really big building. You just can't predict outcomes, so, you have to build a good team that can forge a path, and have some 'kill-it-don't-kill-it' guidelines, and expect that many will fail.
What's that...? No, I've never experienced that situation myself, why do you ask?
Another way to look at it is to say this provided you with a job and an entire company with a revenue basis, probably for years.
I mean on the other side of the fence there are plenty of stories of software developers being allowed to "properly design", no technical debt, then after 1 year of zero result they come with a product, which doesn't even start (the tests run and succeed, though). Then the third or so feature request that comes in ... needs a rewrite.
Scaling engineering is much easier if the project is designed with that scale in mind.
When possible also have (a snapshot of) real data to run conformance and performance tests against.
Gall's Law is a rule of thumb for systems design from Gall's book Systemantics: How Systems Really Work and How They Fail. It states:
A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system. – John Gall
https://en.wikipedia.org/wiki/John_Gall_(author)#Gall.27s_la...
Having a product manager is key, even for internal products. You don't even have to call them a product manager. That person can be a senior engineer who knows what the system has to do to be useful in the company's ecosystem. Someone has to have the final say when there are decisions to be made about the direction/execution of development and feature choices.
Also, there must be a manager above the "product manager" who has the guts to kill the project or remove a poorly performing product manager. This seems to be a big factor in most of the projects I've seen go south.
It also contains truly insightful knowledge regarding building operative systems at IBM in the seventies.
It's one of my favourite books on software development. However, it requires some insight and preexisting knowledge from the reader in order to tell what's what.
The Deadline is a bit more generic (and dramatic): http://tomdemarco.com/Books/deadline.html
I may be suffering from a bout of biased sampling but it certainly seems that the problem of actually defining a product is getting substantially worse these last few years and in the absence of real product strategies there is instead a proliferation of cargo cult product thinking.
I would much prefer to have a small team of very good people figuring out things, develop a clear strategy and then see where more people are needed. I think the main evil is to ramp up team size before the team is ready.
Hiring developers early on with prior experience in the business domain is critical for company success for this reason. Every developer knows how developers work, so tools by developers for developers tend to work fairly often. Most developers probably don't know anything about how your customers work though and will have difficulty prioritizing what's the most fluid parts and what's not regardless of how brilliant and fast they may be.
It's tough to tell whether haphazard selling or haphazard technical design has a bigger impact upon codebases. Add in realities like team turnover and even technical design may matter little compared to how well a codebase is documented for the health of the codebase. I can stare at thousands of unit tests but without some history and organizational know-how it's likely for a successor to repeat the same mistakes by trying to mimic existing patterns.
If you want to get more done, figure out how to shrink the size of your team. The quality will go up, too, since ownership is more keenly felt the fewer ways you slice it.
Projects that require large teams are larger, and thus more complicated and difficult. More difficult projects fail more often.
Artificially shrinking the size of the team without reducing the scope and complexity of the project will rarely improve its chances (unless the team size was only an effect of beaurocracy, and you manage to only kick off the people who aren't contributing).
He doesn't seem to touch on the fact that many times large teams exist because a project is large and complex, which makes it easier for projects to fail.
But the reality is that big teams have problems independently of the capacity of the companies to attract talent. So I don't think that "average ability" is the source of the problem.
As someone else posted, The Mythical Man Month is a good read on this area. And communication overhead is an unsolvable problem as the people involved intro a project grows.
No, it means if you have a large team you will have more average developers - you could have a small team of 3-4 hotshot developers and they turnout a fantastic product - but add 10 normies to the team and it just drags them all down.
Because, yes, that's completely expected. But our perception is biased so it may not be true.
I had an argument where I was explaining that engineering would be more efficient if we hired a researcher - at least just one to start with. Managements answer was to hire 10 more engineers to solve the underlying problem.
It is rare that a project starts with the right idea. You have to make frequent course corrections as you progress and discover things along the way.
Without course correction, the large team builds the original idea, which is often not the right idea.
Most products fail. The vast majority of aspiring entrepreneurs fail. I'd say products built by large teams may well have a better success rate.
The focus on the team is interesting, specifically the "hold them accountable for clearly defined KPIs" point. What about the product owner and manager? What happens to them when the product is delivered feature complete and on schedule but still fails?
As developers we have peer reviews, in my experience, product managers rarely have people review their work. Meaning there are very few people that can push the product manager to improve his/her strategy/tactics/approach. I find it's usually a senior manager or C level that has the most influence over a product manager. The flaw in this is that the person reviewing the work is not an expert practitioner thus cannot provide concrete advice.
The author mixes in methods from LEAN and agile with statements such as:
- "Start with the smallest scope you can deliver that provides value to your users." - "Let them choose their own tools and processes. And then support their decisions."
However, given that is well tread ground, I think the author's main point is to address the issue in large teams whereby "no one is fully committed".
The key piece of advice is "Pick a group of people who are willing and able to shake up the status quo in your organization." Nevertheless, I would say that the act of doing this naturally turns your project team from a large one into a small one.
The advice is not unique and has definitely been tried, with myself witnessing a product manager that effectively created a startup within a startup. One should not discount the manifestation of this phenomenon as a visible sign of organisational disfunction.
In addition to the implied comment that people not on a team were not good enough to meet certain criteria, you also end up with a clique situation as described by a former Valver [1].
All this comes back to the point that no one is fully committed and a toxic environment begins to form. Unfortunately the answer to getting commitment from people is even higher level and ethereal, best described as ensuring the company has mission and purpose. Similar to Dan Pink's focus on mastery, autonomy, and purpose - but executed at a cohesive organisational level.
I would summarise what's needed as real leadership. Meaning someone that can rally everyone around a product and the company. Once you have defeated apathy, commitment can be channeled into extra-ordinary outcomes. What a lot of managers don't seem to understand, or be willing to practice, is that getting and building commitment often means building meaningful professional and personal relationships with the people you work with, which is at odds with the often mercenary nature of businesses.
[1]: https://www.wired.com/2013/07/wireduk-valve-jeri-ellsworth/