The drivers behind software delivery inefficiency
dl.acm.org
dl.acm.org
Taking on another project, or workstream, or idea, or investigation in a team is pretty much entirely downside.
- more context switching, which means people get cranky faster
- lower bus factor for any piece of work, until its 1 dev per thing and that things stops of the dev isn't there
- lower people count on work means everything takes how long 1 person can do it (2 people is twice as fast for parallizable work)
- slower dependency resolution until something as basic as getting a PR reviewed is a big deal
- fewer people seeing the problem means less experience brought to bear, which reduces speed gains from having seen this stuff before
And more besides. For every project the goal should be maximum parallelism.
One person dedicated to a single project / task baseline 100% total output.
One person split between two tasks / projects that require context switching, 70% total output at best, perhaps 50% total output, a loss of 50%.
One person working on a single task that they’re passionate about / intrinsically motivated to complete: 150-500% output.
When you have one task / project to think about and focus on, it engages your brain: sleeping, showering, walking, your subconscious is processing.
Adding just one extra project / task that requires a context switch mostly kills this subconscious processing mechanism.
Say you have a team of 6 (5+1 DevOps Wizard), and a team of 5. so Call it 5x200 +5 x100% output, 1500 output total. Split our wizard in half and you get 5x200 + 5x200 = 2000 output total!
At a certain minimum level the effectiveness inverts because people are waiting on wizard magic and everyone is less effective, but there is benefit to having specialists be split.
You're swimming against the tide with this. AI pilled management is driving teams to deliver more parallel workstreams, not fewer, with fewer, AI-augmented devs taking them on.
Context switching isn't seen as a cost. AI is seen as the solution to the bus factor and experience problem. Same goes with the issue of dev capacity.
IMO it's deeply misguided but it's crystal clear where the winds are blowing.
Now many months in, it doesn’t feel nearly as taxing.
Maybe this is in no small part due to LLMs improving enough you don’t have to spend as much mental energy reviewing their output and them being better at managing worktrees and other tedium, but the idea of doing lots of things at once feels like a mental muscle that was previously weak and has gained in strength in ways I never though possible for me.
I struggle to believe people above me where I work refer to these leaches as “our partners” it seems mad to me.
I'm not talking about multitasking on a moment by moment basis. I'm talking about over the course of a day or week you have multiple reponsibilities. That's the most normal thing ever.
For example, current projects I'm working on include:
- Babysitting a weeks-long data migration
- Designing a better storage system for some parts our data
- Implementing data cleanup for customers who have offboarded
- Code hygiene improvements across the codebase to support static analyzers
- Planning committee work for upcoming hackathon
Admittedly this is a little idealistic, but if it's not the goal then the reality is always far worse.
And what about investigating and fixing bugs? Bug reports don't come in on a schedule, and they're necessarily interrupt-driven unless you just don't care about your customers.
For a bit is ok. I had one month task, that was interrupted daily for over a year, that's how I got demotivated and burned out so much that I needed neurologist.
> two difficult tasks that were supposed to be done in parallel maybe that would be different, but I've never seen anyone asking for that.
Probably that's why you say interruptions are good, no experience of how bad can interruptions be.
The context switch cost is simply moved to the reviewer, who presumably was working on something. It’s net the same result.
There ends up being no shared vision of how it all works. Disparate pieces are developed and some poor sap ends up gluing them together in a painful manner, 2-3 months later. QA finds some edge cases, driving last minute fixes and delaying release.
Ultimately more difficult bugs escaped QA and go on to result in customer escalations.
Eventually, the feature gets cleaned up a year or two later, after another month or two of development effort.
Customers just see a broken feature for 2-3 years and give up on it.
So. Much. Waste.
It’s not about having a Chief Navel-Gazer Architect. Development on a shared code base is just not as parallelizable as one would desire.
One has to thoroughly understand exactly what they are doing and know all the relevant considerations. It isn’t practical for a group to all be capable of such for every feature. And nobody is going to take the time to write a detailed internal API spec that would even allow such to be possible…
What year is it?! Does anyone still blame QA?! If so, please consult the calendar and enter the 21st century!
Over the last 16 years all I've seen is people implying (heavily) that they can replace QA with "good enough unit tests"..
Which then inevitably leads to bugs, and performance regressions and developers doing their own QA.
I honestly think that all of this is a psyop to get devs to do more, that and "devops" which is either rebranded sysadmins, or developers working as yaml monkeys between code reviews and feature development.
Currently working on some legacy stuff with no proper end to end tests and I'm very grateful for competent testers right now.
Ya, in that case, IMHO, self QA can be fine.
Customer facing products (deliverables) need QA/Test. Again, IMHO.
Good enough end to end tests then maybe.
Is this slop, or poor english skills? Is there any peer review barrier to being published here? Because this kind of looks like a low-effort 4-page survey of other people's data with no original empirical evidence. In other words, probably should've been a blog.
Science isn't just cold reports on aimless facts. You're supposed to discuss with your peers.
Don't get me wrong, the subject and abstract is a topic I relate with so I'll probably still read it. But curious if this is normal. I just want to know how much salt I need to use.
Organizations can't even track and improve on sprint performance. They aren't gonna track and improve inefficiencies either, because they have no motivation to improve. Even if they're losing money and customers due to their product sucking, they still won't see those inefficiencies as the cause of their problems. They'll instead imagine that what they really need is to push more sales/increase initial subscriptions, or add a killer new feature they have no time to implement, or pivot to a different business model. But their lack of rigor is only partly due to executive stupidity; the other part is mediocrity. If you have lazy, ignorant devs, you're gonna have lazy, ignorant process, leading to all the waste.
If there is a significant amount of user interaction to test, QA is going to do a better job than devs.
If there's a lot of integration behind the scenes with a ton of edge cases, devs will do a better job.
If you have both (probably), you need both teams to pitch in more than upper management wants to believe and they'll just cut you instead! :D
In theory, I agree with what you're saying, but I think a codebase needs to be in a certain state and a company's leadership needs to be in a certain mindset to make it work. Our codebase was far too large and interconnected for any developer to fully understand all of it because of years of building with unclear boundaries between teams. QA's value wasn't really the testing they did. The value was that we had a dedicated team who's job it was to see the big picture and keep up with the changes 15 different teams were making on the same product. Management wanted the benefits of removing QA without the pain of paying down all the organizational and tech debt that made them valuable.
Again, I agree with your sentiment, but also, I think that most large organizations are not willing to do the necessary work to get a place where QA can be removed.
However, there were too many cases that things were delivered as “ready” when they were not. PR review was not enough as everyone were checking diffs from GitHub only.
I implemented an intermediate step where the responsible engineer must post a screenshot or a short video of the ticket before passing to the PM and that alone resolved a lot of issues before they materialized on customers screens, just because you must take the extra effort to work on a proof of work.
And removing Product Managers makes devs care about the product. Therefore, we should strip away all roles apart from devs..?
These roles exist because the skill set is sufficiently large to justify a dedicated role. If you're a startup, sure - maybe it's better to be lean - but in a larger org, QA folks are worth the investment and can definitely help to spot things that devs may not otherwise catch themselves since they know the product intimately and likely learn of common pitfalls. The few QA folks I've worked with have been worth their weight in gold.
Hold up... We saw it at the same place :D
Writing automated end-to-end tests? Those are very often best done by those writing the software, with assistance from experts who know how to effectively build test infrastructure. I've very rarely seen them done effectively by QA or "test engineering" folks.
Running manual test scripts and torture testing the UI to break it in new and unusual ways? Definitely get dedicated testing / QA folks. It's very easy to be blind to how others might use the software, especially if you wrote it.
This so much - but most of the places I've worked at have insisted that manual testers with zero programming experience should build and maintain the test frameworks. Talk about trying to push a square peg into a round hole. Granted some of the testers have embraced it and not done a terrible job but by and large most of the testers have had no idea and done very little.
When I first joined Amazon, my team had no dedicated QA, Product Manager, Technical Program Manager, or frontline support team. I had come from a startup where none of those roles existed either, so it felt normal. Engineers also carried the pager and dealt directly with the operational consequences of what they built.
Since then, I have worked with teams of very different sizes and levels of product complexity: internal and external products, Tier 1 services and much less critical systems. Some had every specialization—QA, PM, TPM, and a dedicated support organization.
On a few of those teams, I encountered a very different engineering culture. Product understanding was the PM’s job. Quality was QA’s job. Delivery coordination was the TPM’s job. Customer problems were support’s job. Collectively they had created enough distance that engineers no longer felt accountable for the whole outcome.
For me it is less about whether a team has QA, PM, TPM, or support line, and more about whether those roles increase the team’s capability or partition its accountability.
The litmus test I use is the 3 a.m. page: does the engineer have enough situational awareness to understand the customer impact, navigate the system, make a credible mitigation, and help drive the permanent fix? They do not need to be experts in every discipline, but they should understand the product and system well enough to own the outcome.
Specializations are necessary as systems, products, and organizations grow. But it has gone too far when engineers can no longer operate effectively without every specialist standing beside them.
I am also exposing my own bias here: I believe engineers should carry pagers. That is how I grew up professionally, and I think direct exposure to production consequences produces materially better engineers.