Software Development Waste
neverworkintheory.org
neverworkintheory.org
For example, there is a team I know of that is building a web application that serves maybe a couple of thousand people (if they ever acquire all of them as customers). They spent all this year building a humungous infrastructure with all the modern acronyms and have nothing to show the clients but a bunch of K8s this and Docker that.
It was a lot of engineers flexing. And of course, the same engineers that started it had moved on to greener pastures well before the system went live and was tested by real life and real customers.
I think I've finally grown enough of a pair to never tolerate something like that. Prove that you need microservices first. Prove that you - prove that the organization - can manage the overhead and complexity.
Is it even worth it adopting ci/cd for my small craft or am I fine sticking with push, ssh, pull, [vi config.json]? What do they actually achieve with it?
Edit: incorrect idiom
The ideal is deploying every single commit. What could prevent you from this? Primarily two things:
1. You risk screwing the process up, causing expense in terms of damage to customers.
2. You don't like how much work it is to deploy every single commit; the transaction cost, even when everything goes right, is too high for your personal tastes.
Continuous delivery automation tooling helps with these two issues. If they are not problems (in other words, you do things right every time and you don't mind doing it manually for every commit) then you don't have the problems tooling would solve, so no reason to use said tooling.
Continuous integration is different from continuous delivery, and more about coding patterns than automation.
Having "unit" tests is a good habit to get into no matter what, really, but that doesn't mean you need a bit CI/CD pipeline. CD isn't really needed until the creation of your application is complex, or you have actual customers/users that depend on it.
Anyway, would you call putting the entire interaction in a shell script "ci/cd"?
There is then a feedback loop, as companies then have to require developers with CVs that have the technology.
A bit of background: a company I worked at had one primary product with a web interface for the customers, businesses who used the product, to do all their functions. Think AWS console but not so massive.
There was another, minor product brought in via acquisition, which had its own web interface.
A third group came into existence when my employer got acquired, which created a new web interface for its product, which did mostly the same things as the primary web interface but for a different set of customers.
These all had separate logins, so customers had to know multiple credentials. There were also several large customers of the primary product who had multiple logins for reasons which defy my full understanding to this day.
The project I was on was to bring in a system that would eliminate separate logins and provide a single sign-on system for them all. Good goal. very doable.
The VP in charge of the new group and a product manager got into their heads that what we really needed was a single landing page portal, a piece of software that didn't exist and wasn't necessary for the sign on. The team wasted at least three if not six months chasing that landing page goal before the VP decided her product needed to go live and oh by the way we didn't have time to do the single sign-on any more before her product was to go to production.
Ultimately the distraction killed any chance for the company to actually implement the desired sign-on that the customers were clamoring for, and to my knowledge they are still limping along with a every customer having at least three different logins, one for each system.
I think Ward hits more fundamental points (e.g. this page suggests the waste lies in not consulting a domain expert, whereas Ward suggests the waste lies in the hand-off it takes to consult a domain expert in the first place, rather than have the development team become domain experts. Or when this page talks about building the wrong product, Ward talks about wishful thinking and being gut-driven rather than data-driven.)
Here are some of the wastes I remember making notes on from Ward's book. If you want more details on any, I'm happy to answer questions.
- Scatter: disruption in communication, communication barriers, changing priorities, distractions.
- Hand-off: essentially when there's a common goal, but different people are in charge of knowledge, responsibility, action, and feedback. Important information disappears into black holes during hand-offs.
- Useless information: reports, updates, analysis, etc. that is created not because it helps the customer, but because someone wants it to feel good.
- Waiting: over-utilised resources cause queues which lengthen feedback, decrease urgency, increase variability of performance, etc.
- Wishful thinking: making gut-driven rather than data-driven decisions, planning further out than you can reliably predict, reducing uncertainty too soon.
- Discarded knowledge: spending time and effort building knowledge, but then filing it away and never reusing it.
- There are some areas where humans can develop good intuition on their own. For example, I believe that with experience, you'll learn to distinguish a good variable name from a bad one. You can go with your gut when naming variables.
There are some areas where human intuition will pretty much always be very wrong. Figuring out what the customer wants is one of them. IIRC good developers guesses right about what the customer wants about 30 % of the time. This is abysmal. This is where you need to let data drive your decisions.
There's lots of psychological research on this. Meehl has his classic studies on clinical versus actuarial reasoning. Kahneman has had his adversarial collaborations with that other guy who promotes intuitive decision making.
The key is to continuously test your abilities to know your limits and don't get overconfident when your gut tells you you're right.
Gerd Gigerenzer, I believe. I'm with Gigerenzer, as his research tends to replicate a little better ;)
As a manager, you need to manage resources and they are limited. You can't tell your staff to do overtime if it's not an actual emergency that affects paying customers, that's not sustainable.
There's really only solution I've seen that works: don't allow everyone to be a cook. Designate exactly one person to be in charge of making these sorts of decisions.
I started by countering, "Since I'm going to be working late on your project, what % of your commission are you offering?"
This is where everything we do is metered, and our lives become "data-driven."
It does bring results, but I feel there are significant costs; usually difficult to measure.
One of the things that I will do, as I write software, is evaluate the code I'm working on, for reusability.
Often, to make something reusable, I need to avoid optimizing it for a specific application, and "generalize" it. I'll add specializations into an "adapter" or "façade" layer, over the module.
Most of my little SPM modules were developed this way. If I think it deserves it, I'll stop work on my principal project, and break out the subsystem into its own project and repo. Once I have it standalone, tested, and ready, I will re-integrate it, as an external SPM module.
In more than one instance, I have removed the code from the principal project, because it was not something that would contribute to the main goal.
But I now have this neat little SPM module, ready to be used in other projects. Here's an example of a project that never made it into any of final results, but I think is pretty neat[0].
Some of my modules are basically my "baseline." These are used in everything I do.
Many would consider my workstyle "wasteful," but it works for me.
This is my big gripe with the "data-driven cargo cult". It completely disregards the qualitative, and focuses solely on things that are easy to measure. The problem is that some of the most important properties are hard to measure. Optimizing on data feels good, but refusing to make decisions on critical parts of your business that are hard to measure will lead to suboptimal outcomes.
"data-informed" is preferable to data-driven, but requires more thought unfortunately.
You're right, of course. Development by definition requires novelty. If we just built the same thing over and over, we wouldn't be developing. We'd be manufacturing.
But development still does entail repetition on some level. We pull features through the implementation process, over and over again. We review code, over and over again. We hold retrospectives, over and over again. We do status updates, over and over again. We make supposedly backwards-compatible changes, over and over again. We deprecate features, over and over again.
If you look, you can find all sorts of things that repeat.
These things can actually benefit from being standardised. Standardisation does not mean preventing any future changes; it means doing things consistently, and systematically evaluating and implementing proposed improvements.
But why bother? You can't improve anything unless you can do it consistently. You avoid spending creative energy on repetitive processes, and can use it where it matters: on the actual development. You can learn faster what does and doesn't work, because everyone does the same thing on the same schedule.
One person can create, duplicate, and distribute earth-shattering functionality. They can then, iteratively, improve or alter the product, almost in "live" fashion.
Manufactured goods have a lot of constraints that come from rather immutable structures; like the laws of physics. There's usually a number of parallel engineering efforts that converge, and that's before the product even rolls off the assembly line.
Once there, it still needs to be placed in position to be accessed by the end users.
Each of these steps are quite different from the others, and require a confluence of disciplines, skills, and experience. Many of them can be subjected to measurable process. Maybe only the initial engineering step could be fairly compared to software development. Even then, prototyping introduces some considerable structure.
The thing that Kanban software development chooses to focus on, lead time, would be my top candidate. That seems universal. I'm guessing most others would be team-specific things that come out of a DMAIC analysis. If you've got a problem with too much rework due to team miscommunication, the specifics of how you measure that are going to depend a lot on how your team defines rework in the first place.
That's great. It tells you which stock to short.
Creating reusable knowledge is not in itself wasteful, but neither will it in itself deliver value. (The reason we do it is because it makes value delivery more efficient in the future.)
Many people today will emphasise delivering value and sacrifice reusable knowledge. They will call it "wasteful" to create reusable knowledge, but it's not.
The trick is finding the right balance, of course.
I am not a fan of the "MVP" concept, as it is defined in the software startup industry. I feel that it delivers time-delay disasters.
Mostly, because of the deliberate, and premeditated, willingness to ablate Quality, in a drive for deliverable.
We get bad stuff, fast. All too often, that bad stuff forms the nucleus of the product, for the rest of its lifetime. Basically, a foundation of sand.
It's hard to argue against the "dollars and cents" rationale for it. People make money, this way; lots of money.
As long as money and profit are the main measure of success (and it can be argued that they are the only ones that matter), we will have this drive to deprecate Quality, in favor of profit.
Quality is expensive. Real expensive. I've talked about this before.
But I've worked for hardware companies, for most of my life, and it's fairly rare, when hardware companies can get away with it; often, because laws and regulations. Also, tooling up a product line is a major commitment. You really don't want to cheap out on that. Distribution channels are also some heavy-duty overhead.
For whatever reason, software shops believe that we are exempt from the restrictions that have governed all engineering; throughout history.
In a way, this is true. Software is not hardware, and requires a drastically different approach. I always had a devil of a time, convincing hardware people of this.
On the other hand, I don't like the idea that we are also exempt from the requirement to deliver a baseline level of Quality.
Personal gripe. I'm an old curmudgeon.
A bad MVP is a bunch of features thrown together badly in the hopes that parts of it will match market demand and that the market is willing to look past the quality problems. What follows after moderate success with this approach is a long journey to try to build back in some of the quality that was avoided at the start. The end is predictable: sprawling, mediocre software. With some luck it makes money. Often it does not.
A good MVP delivers a single, deliberately chosen feature to a well-researched segment of the market. This single feature has excellent quality characteristics and blows any competition out of the water. On that success follows continued incremental, high-speed development, always with quality as a primary driver.
But it takes discipline. In order to economically deliver quality, you must do the research, and you must be willing to deliver in really small, very well chosen increments.
And that is hard. That takes excellent developers. It's easier to just throw something together and hope to get lucky.
I know what I consider "MVP" to be, ideally, and that pretty much jives with your description.
It's just that I don't think most folks actually do it. It's like "Agile," in that respect.
As someone that practices high-Quality development, I am regularly laughed at and told that I'm "wasting my time," doing Quality coding.
Words come to mean whatever their most common usage implies, so I think that both of these wonderful words now mean something other than what they used to mean.
Functional departments and their managers are responsible for creating reusable knowledge. A project leader (chief engineer, in Toyota speak) is responsible for delivering value in a particular project.
The project leader accomplishes their goals by help from the functional departments. However, the project leader has no formal authority over the engineers in the functional departments.
In other words, there will be a constant negotiation between the project leader who argues for something to be designed for a project, and the functional department which must be convinced it's worth spending time to do that thing.
The way a Toyota executive put it was "A great car is made from lots of conflict".
I'm not doing this approach justice in my description. I suggest just reading Lean Product and Process Development by Allen Ward. Lots of good ideas in it.
Changing that is hard. Tesla has a screen where software decides what lights to flash - and dollars to doughnuts it's because there was not dozens of decades old business departments fighting to keep control.
And the example you have above - a Product manager persuading functional teams is a separation of powers problem - think of the US government as a surprisingly long lived example
One would need a lot of top management support to keep that from falling into one of two more stable systems - either the functional team leader tells the PMs what they are going to ask for, or the PMs get hold of budgetary control
Either way the conflict is resolved.
There’s also the concept of “triaging” for “waste.”
One man’s garbage, is another man’s treasure.
I have learned not to summarily dismiss a lot of practices and artifacts as “waste.”
Put another way: they weren't being data-driven optimizers. They were exercising judgement. That seems to be what you're arguing for.
Not an issue.
Have a great day!
It's frustrating to me, because it seems to be underutilized in the industry, but lean software development (as discussed by Mary Poppendieck) has been about exactly this idea for around 20 years. She mapped out the ideas from Lean to software development equivalents and mapped out ideas about what constituted "waste" in this field.
These are exactly the kinds of things that people should've been looking at in retrospectives, but it's become deemphasized and is instead a shortened ritual with no useful effect.
What if I our team has a concern that isn't mentioned in a checklist?
What works is when people share goals and collaborate.
My current employer just lets me do literally whatever we want. A service is basically owned by a developer. And so far my customers absolutely love my work.
Not OP, but on my past two teams and my present team, this part has never happened. It was/is just a box ticking exercise.
I think the purpose of having such a checklist is not to limit the discussion to other concerns, but rather serve as a reminder what all the different things can be a problem and should be discussed.
In my experience, the value from retros is a function of the skill of the facilitator.
Perhaps you could be a force for change on your team, by speaking bluntly about not spending enough time on that third column. Others might be receptive to it!
> Extraneous cognitive load
> Psychological distress
Are types of stress that introduce "waste" (or rather inefficiency). The root cause? Ignoring the human factor.
Programmers, designers and other tech workers are not robots. We can't be. That's the job of the software we build.
Our well being, happiness and engagement matters. Ultimately we are craftspeople, want to work with good people, take care of our tools and products and we need time and resources to do it right. We want to be proud of what we build or at least confident in how we build it.
Too many programmers are anxious and worried because they are not given enough time and agency. Many lack connection to the users of their products and services, so they don't see the point.
So I cannot fully agree with what you're saying. Sure, inefficiencies happen, waste is unavoidable. But the issues above are concerning.
When we do things with care and consideration, give them both enough breathing room and focused attention, then we make better things and are happier for it. Mix that with a healthy dose of pragmatism and we also avoid doing the wrong or too many things.
For example, a program I recently learned about was “lerna”. If I had know about it a few months ago, it would have been less painful to manage all of our libraries. Another good example is deploying to serverless backends. Without the serverless framework, this was otherwise painful. And once we find out about these tools, there’s a ramp up cost to integrate them into the existing workflows
Let's say I can make a product in a billion different ways. That means I have to make around, say, 30 decisions (log2 of that) how to go about it. Now let's say I am more limited, and I can only make it in a million different ways; that means only 20 decisions have to be made. More decisions mean more overhead - everybody in the team needs to learn them, accept them, etc.
So in the past, people had limited variety of tools, and the way they could build SW products were more limited, but it actually was more productive because lot of the decisions didn't have to be made. That's the paradox, we think that more specialized tooling leads to higher productivity, but in creative professions, where the job is to make decisions, and physical labor is not the main constraint, this is the opposite.
I'm not sure how we'd be able to define productivity in a meaningful way to compare.
I feel like programmers create more value for business today than ever before, even if they're not necessarily writing the code creating that value.
The bar for "creating value" is way to high today and the simplicity is gone in favour of expensive complexity ofent for no good reason.
Excel (from the user perspective) is probably the last man standing of the simple way.
> The bar for "creating value" is way to high today
It's actually really low, and usually just starts with doing work.
This is true, but I don't see what the tooling or the productivity have to do with that. It's just that there is an always increasing demand for new programs.
All the simple programs have already been written, this means we can only go forward writing programs that are ever more complex to maintain the same amount of value to our stakeholders. And this increased complexity requires slightly different skills and activities than those we learned starting our careers. For in order not to break down from this complexity, we need much more powerful abstractions and ways to combine than before - more complex solutions are usually also more brittle.
I think it's rather a feeling of loss of control than loss of productivity, caused by the accelerating pace of innovations. We're all kind of becoming more and more involved with architecting, configuring, installation, managing and operating rather than building (or rather, building software is changing shape), and the 'building' part is where developers naturally locate 'productivity'.
Or profilers were not as useful when the programs were printed on punch cards.
The thing is we're orders of magnitude more productive if you look at what results we must achieve: if IBM wanted to publish news on a worldwide networks in the past they would have needed thousands of people and millions just to setup the infrastructure, let alone convincing people to buy terminals. Now an intern can do it in a few days because of all the successive layers of tools and software.
What you describe doesnt need to happen if you are more dictatorial: get 2 languages allowed for productions (one ui, one logic), 1 more for back office (for statistics and reporting) and force everything in it.
Do the same for libraries and dont do it by commitee, just say that s the stack, you can do go or groovy or rust elsewhere and done.
Then people start discussing solving actual problems rather than war-of-the-stack every meeting.
This includes :
- locking systems to stop people trying to understand how it all works
- providing and actively promoting high level tooling and platforms to allow other business to do their jobs without needing to understand how a computer works.
- hiring at any cost any engineer with the technical capacities to create real alternative technology (and to profit from his knowledge is just a good side effect of removing its free time)
I know few companies (maybe zero ?) that have not any critical dependency on a GAFAM product, be it a framework, a platform, libraries, tools ...
I'm truly fascinated (and worried) at how you can successfully run a software company with programmers who don't even understand (or remember, because, for most, we were taught) how a computer actually works. And by no means I'm criticizing those people, I'm totally in this group. It amazes me that some of my best productive coworkers never tried once in their life or have the curiosity to install some Linux distro. I'm not talking about loving it and keeping it, but just to have the curiosity.
This is somehow nearly universally seen as a bad thing (except at Amazon) yet companies seem completely fine with it surging.
My team is wonderful, even astoundingly so, on most of the communication points. My legacy codebase is so challenging that I probably waste 90% of my time.
This isn’t a typical situation! But I want to make sure it isn’t dismissed in a rush to wholly classify this as people problems.
Also, it's hard to know whether something is a "waste of time" or simply "work". I've dealt with legacy code that's been built over many years and while it's been complex it's often necessarily complex in places. Working with that isn't a waste of time. It's just the job.
At least we all got better CVs as part of it.
I am on a team that basically just imploded after a year of chaos. I mostly got promised a better CV for work we never got to do (and never will at this point).
They fired all the product managers. All of them. At once. We call them Venture Leads, but they were effectively product managers. On Monday I am finalizing a feature for one of them and on Tuesday I have nobody to show it to. I don't even remember if I ended up pushing it.
They apparently intend to resume work on all the projects, but at a later date when they hire new product managers/venture leads.
This was not their first time firing product managers either. They fired the one for the first project I worked on.
For more, read Glassdoor.
https://www.glassdoor.ca/Reviews/AltaML-Reviews-E2534656.htm...
Why did I join this company? A year ago they had 5 stars and were highly recommended, including by the person who referred me to the company. Who was one of the product managers.
And it was all based on difficulty understanding how the internal billing worked.
- why do we do this in the first place? What _is_ the customer value (and therefore is waste?)
- is this complicated reporting process actually needed?
- which strategic knowledge is missing so that the delivery teams can decide things on the spot?
And so on… . The bigger the company, the harder to overcome human issues. At least it helps that the LEAN focus is always to deliver customer value fast, with high quality and still cheap (by eliminating everything else).
in the end, it seems a lot of people just want well defined rules (that work to a reasonable extent) to follow and instead use their thinking for solving the actual problem at hand (programming)
its not unreasonable, especially if someone is hired to be just a "coder", there might be some hiring/expectations issues hidden in there too...
I think as a good manager one should be able to define essential and accidental complexity.
https://youtu.be/_NeJ3Kg6OUo Silicon valley: product and engineering
[1] https://scholar.google.com/scholar?&q=Software%20Development...
You do not need an account, you can just download the PDF.
Is that just alternating working on tasks without ever finishing anything (normal thrashing) more of the time?
"This waste is also related a tension: intransigence versus capricious trashing. Responding to change quickly is a core tenet of agile development and often thought of as the opposite of refusing to change. However, responding to change is more like a middle ground between intransigence (unreasonably refusing to change) and thrashing (changing features too often, especially arbitrarily alternating between equally good alternatives)."