“Blue Light” creating capacity for nothing (2007)
theoryofconstraints.blogspot.com
theoryofconstraints.blogspot.com
In the industry it's fairly well known (ok that's a weasel phrase even though I'm sure I'd be right if I used 99.999999%) in regard to the overall benefits of TAs (Trade Assistants.) Typically a TA around welders, they're grinding and readying the surfaces to be welded and helping position the work. Though there's value in setting up, not just making blue light, welders generally receive a much higher pay packet, since ultimately it's their good welding skills that ensures quality welds leave the floor and a long life of the component.
What's obvious is what would have happened if the blue light was 50 or 60% of the time and the welding machines were at capacity. (Higher capacity welding machines are generally a lot more expensive.) To the clueless a lot more can be squeezed out ... and that's what the hero would have aimed to do, in fact no doubt another 25%. The experienced plant manager would have then kindly thanked them at that point perhaps realising the consultant was out of their depth.
What you are highlighting is the countervailing lesson, kind of a specialization of Chesterton's fence: if highly experienced people are doing a process in what seems to be an inefficient manner, consider that it might actually be the most efficient manner, given constraints that you do not fully understand or are not aware of.
Sometimes, not that I can think of much software or hardware related situations, the fence is just an idea, years of experience have lead to a situation where they've established a comfort zone of sorts, and hesitant to change the way things are done ... often because they don't totally understand complicated processes they might be using ... or because things done differently might be more open (but rare) to a specific problem or being prone to a failure which from their previous experience, might have chewed up half a day to a day if not more to sort it out ... when the fix by someone who is competent (having been shown the correct way from the start) is able to resolve the issue quickly, barely 20 minutes of time and patience and it's in half an hour it's like nothing happened.
As for "do not fully understand" - yes that is an excellent way to describe total inadequacy.
The Goal, by Eliyahu M. Goldratt is highly recommended for understanding Theory of Constraints told from the view of a fictional plant manager.
How much of your time is spent on that thing/those things?
How much is spent on other things and how can you reduce that, especially for any resources that are your constrained (rate-limiting) resources?
ToC tells you that while there will be multiple constraints, your biggest bang for the buck comes from identifying the limiting constraint first and finding workarounds to break / relax that. Focusing on the non-limiting constraint would not give you the best benefit.
This is such a general concept that we can apply it to so many processes whether it is sales, customer service, bug triaging/resolution, logistics etc., Any place we observe a queue (whether visible or hidden), I can guarantee that there is a lurking constraint somewhere.
Of course finding some help moving parts around for the welders would be the first thing the foreman would think of, because thats what foremans do all the time in factories.
Maybe the welders chose to work this way (i.e. inefficiently) because continuous welding produces fatigue which risks H&S and lowers the quality of the product.
So if the story was real, the solution would be extra personnel so the welding is done continuously, by rotating people between welding, moving parts and other jobs.
The critical difference is that they did not need to hire more welders, buy more welding equipment, set up more welding stations.
I don't care if the story came right out of Aesop's fables, because it's a good story and it makes a good point.
It's almost like a story where the genius works out the boss needs a secretary.
When it comes to inventing stuff, important points are often overlooked.
Imagine if the same sort of story recounted a company with a 9 to 5 office where 100 people worked solely at their desktop computer station back in the days of W98 or XP doing whatever it was they did. The guy observes when people arrive for work they're not productive because the first five minutes their system was booting and virus checking. There's over 8 hours of lost production so it's worthwhile to hire an additional low paid casual worker for minimum award, with the minimum required hours for casual (here it's iirc it's been around four hours a day,) to arrive a couple hours beforehand to ensure all systems are running so workers can start working immediately when they walk through the door. Clearly that sounds right, but most here would know, in practice, for many workers, output would not increase substantially. The nuance is in the finer details.
I've seen some pretty terrible inventory management and poor process flow, but this is a whole other level.
I imagine the blue light applied to programming would be a crude tool. I want to hear keys clacking? See issues closed, pull requests? But we can ask what are the things developers are doing which interrupt them from coding and could there be a benefit to someone employed to handle those distractions?
I don't feel like that's a very good conclusion.
If a good chunk of your processes have roughly the same capacity, and they're all running at that capacity, you don't have anything I would call a bottleneck.
A better derivation of Eli's ideas ultimately causes your entire team to work on one feature before deploying another before deploying another. Sprint planning is “what do we want to take on next, how are we separating it across the team, and how will we pass the baton?” The relay-race baton-pass becomes the crucial concept, whoever has the baton is expected to be working 100% on this thing and only this thing, anyone who does not have the baton is either meant to be preparing to receive the baton, or else they are free to work on whatever refactoring or bug-fixing or side-feature or user support or whatever needs to be done. They are the equivalent of this non-welder helping out. And this is also why it's important for the baton to pass from person to person, because it can be exhausting to hold the baton for too long: what do you call someone who sprints for an hour—a jogger.
This also unlocks the true value of daily standup, which is the person with the baton telling everybody else what they need in order to be successful, and the person who is about to receive the baton to be aware about the timescale at which they have to drop everything else they are doing and work 100% on the next part of the sprint, and the rest of the team checking in on the personal health of the person who has the baton, making sure that they are not overworking themselves to death. Are you okay?
What's at stake is covered in Goldratt’s book Critical Chain, that in a “project context” there is a complex network of dependencies, including hidden ones that come from the raw logical fact that one person can't be working on two different parts at the same time, and the constraint, the thing that actually needs to be tracked and preserved is temporal safety buffer. The default state for a project is that if a step is completed late, that delay has passed on to the next step, but if a step is completed early, that advantage is not communicated to the next step because the person working on the next step isn't ready to drop everything and start working on the project yet.
I actually use these principles to manage the cooking of Thanksgiving meals for our family. My wife was very upset the first time I did it because there was a lot of drawing graphs and stuff, a bunch of diagrams, I had never done it before. But now she wants me to do it every year because it is absolutely intoxicating for her to ask me “what should I be doing right now?” and for me to reply “ nothing, you should rest, but in about 30 minutes you are going to have to drop whatever you are doing and put together everything for the corn casserole.”
That this is how it works should not shock us, not those of us working in web development. Everybody here who works in web development should understand that you overprovision your servers and 90% of the time. They don't do anything. That's not for no reason, it's not because we're wasteful, it because it gives us a much better latency on our incoming requests. Making it clear what the Sprint goal is, and whose shoulders that goal is currently falling upon, is part of making a very low latency system which can respond to whatever fires occur on a day-to-day basis while still getting these longer term projects done ahead of schedule. 80% of the team should be resting or dealing with the latest crisis while one person is legit sprinting on the Sprint goal, and progress on a feature should never have to stall just because the sprinter's significant other is suddenly in the hospital: the feature was owned by the team and the whole team needs to be able to work on it, with everybody checking their work in daily and everybody willing to help out to make the group's deadlines.
I've kind of given the sales pitch for it so let me give one major limitation, a lot of our current methodology is structured in order to identify underperforming people by concrete metrics like how many features they complete, how many story points are those features, and the above methodology kind of creates a space where people can hide: they never take the baton so they never are really in the hot seat, but the whole group takes credit for the success of their individuals. Random assignment and pair programming are one mechanism to rectify the concern and then rectify an obvious concern with that solution (“what if we give a frontend task to a backend programmer?”—answer is, they should do the frontend task, but as the person holding the baton they are running the show and thus entitled to say “I want help, can someone who knows more about frontend code pair with me so that I can just ask “how do I do that in the browser?”).
Why not have most of the team coding toward the goals and one or two helpers? Especially because that's how the welding story worked.
My understanding from reading that is the new helper didn't have to get in and out of gear. He wasn't a welder, so wasn't wearing any of the welding gear. His job was to help move things around, position materials so they'd be ready to work on, and generally get the "grunt work" out of the way so the welders could keep their gear on 100% of the time and thus spend most of their time doing their actual specialty: welding.
The new helper was essentially a new floating resource who could quickly move anywhere in the room to clear bottlenecks. Before he was there, the bulk of the time that was wasted was because the welders had to keep context switching, and also because there was some work that the welders were doing sequentially that could have been done in parallel if they had another pair of hands to set things up just right.
Also consider that I'm sure it wasn't really 90% of the welders' time. Even if the author was only seeing blue light 10% of the time, I'm sure these changes didn't bring it up to 100%. There were probably still some non-welding things the welders had to do that couldn't be delegated to others.
Crude but surprisingly accurate. If I am not interacting with my development environment, it usually mean there is a problem somewhere. Sometimes that's me (slacking off, getting lost,...), sometimes it is organisational (useless meetings and processes,...), sometimes technical (slow computer, unresponsive network,...). But it is always a sign of inefficiency.
It was like pulling teeth, but, in fairly short order, the chap I hired became the most popular member of the team.
The official request I made said the server would pay for itself in eight weeks. I think the boss finally understood why I was being so grumpy about it at that point. Or maybe you could just trust the people you claim to trust.
I'm so sorry to inform you, but you have management written on you.
(But in all seriousness, communication is an important management skill, it's good you explained it so directly, but ideally a manager would communicate back-and-forth without needing to be spoonfed all at once.)
And our toolchain... put it this way. We went from no documented procedures, to several. And the several... have no date, no version, no author, no revision control...
Friday is my last day. I am so happy.
This all can be learned from The Goal by Goldgratt which is a story about theory of constraints.
In my mind productivity and value generation are major virtues that benefit the doer and society, big "bigot" has a negative connotation and I don't understand how it goes together.
“Value/productivity bigot” = what they think
“Leeches” = what they are
The person who fills and jigs the welder's pipeline is just as valuable from a systems perspective as the welders.
I'm guessing your issue is not with those that are fixated on productivity, but those that are just poorly educated in the complexity of how to calculate it.
Nearly any able-bodied worker can do those tasks, and they don't require much skill. That doesn't mean they have no value to the overall process; obviously they do. But there's a big difference between finding and hiring a trained welder (or training someone up), and finding any random worker who can lift things and follow some basic instructions.
Absolutely agree with you that focusing only on the "value add" tasks can cause blindness. But that doesn't make the other "non value add" tasks somehow high-skill, high-quality, difficult-to-replace tasks.
Any task requiring or being > 100% better when done with more than one person is now a team. We are now optimizing team output and talking about low-skill is no longer an accurate description of the problem.
It also under attributes how important having a quality fitment/parts prep person could be to the quality and pace of the output. Adding another person can make the work 2x or more productive. If they can spot problems before they get to the welding stage, even more.
When I was young and while I was probably the A-team of the B players in the org, I romanticized about what it would be like to be in a high performing org of all A-players. This thinking is flawed, it devalues the importance of the b-players and puts an outsized importance on the high value heroes.
That prep person may be fungible on day zero, but by the end of the week, they should be a high functioning team member. The value bigot will say that the person that only preps the pipeline should only be paid X/hr because the work is low-skill. But their value to the system is much higher. We should be thinking in systems ant labels about what each component of that system is worth.
But I am not sure the other axes you have to grind here. A surgeon and a janitor are both necessary to the functioning of a hospital, but obviously and not-controversially the surgeon is a lot less fungible. Like, in a pinch, you could ask patient's families to clean the toilet in the hospital room, but you wouldn't ask them to do the surgery.
And more obviously, if people were dying in the ER because the surgeons were busy cleaning the toilets, that would indeed be stupid.
So when the downturn comes, fire the trade assistant.
Or is the point that a good consultant knows to leave before that happens?
In this story, the consultant found a way to remove stupid/low-skilled work from the welders' todo. They no longer have to lift shit, take off their gear, and peel plastic off parts. None of that sounds fun and presumably welders are happy not to be doing that.
I suspect a welder prefers to weld. The consultant found a way to let welders do what they chose/are trained to do, which is weld. I don't see any sort of bullshit metric/target imposed on them.
This is an example of what I mentioned in my other comment.
It is low-skilled work to move a part from here to there as compared to welding.
These are literally correct and practically useful descriptions of the work removed from the welders.
One has to separate the value judgement from the system optimization. The system is the combination of the all the pieces operating in concert. You can't point at a specific part and say, "this is the only thing of importance".
And I never said otherwise. I am responding to your non-sequitur with the like.
> I suspect a welder prefers to weld. The consultant found a way to let welders do what they chose/are trained to do, which is weld.
That's right. An, on a semi-related tangent, it's ironic how a lot of what software industry does is add stupid/low-skilled or mismatched work to specialists' todo.
Consider the "office suite" software - whether by Microsoft or Google or others. The word processor, the spreadsheet, the slideshow creator, the mail client, the calendar and contact manager. All of these tools help us automate and perform faster the work that... we otherwise wouldn't be doing at all, because companies used to have specialists to handle it. But now, instead of having some secretaries and an internal graphics department do the work efficiently, we all have to do a little bit of it on our own. This is particularly impactful in tech itself, where the company "saves" on couple $X/month secretary salaries by having their job distributed to a larger amount of $10X/month engineers.
This applies in B2C and in individual life too. Once I first realized this, I keep seeing this false economy everywhere. And I increasingly hate having this extra work dumped on me. I now feel self-service is a scam - a way to externalize the costs onto users/customers.
I've recently seen it happen at organizational level, too. I once talked with a finance person who complained that (large) companies used to have well-staffed finance departments to handle things like employee expensing. These days, every employee is supposed to handle it by themselves, promising to not make mistakes or misuse company resources, and the much reduced finance departments are called in to occasionally audit this. The finance person I mentioned doesn't like it, because in practice it means that, come end of business year, they have bursts of high-stress, high-importance work for which they're severely understaffed.
My hypothesis as for why this happens is, to borrow the term from Seeing Like a State, legibility. Or rather, lack of it. Salaries of secretaries and other support personnel are legible - clearly visible on the balance sheets. Removing them and spreading their workload equally among everyone else in the company is invisible. It looks like the company is saving money - and the overall loss of productivity is something that "just happens", it surely has nothing to do with everyone being increasingly busy with extra work that isn't what they were hired for.
Like, today a junior dev can be called on to do a demo to another team, and he can easily throw some slides together. It's not like 50 years ago he would have had the in-house design team on hand to do slides, he'd just not be presenting.
I think this is another example of a problem I'm hinting at: the reason said junior dev has to make slides is because they can. If their only option was to ask the in-house design team for help, nobody would expect them to show up with pretty-ish slides. But, because PowerPoint makes it easy for anyone to do slides on their own, everyone now has to - much like Word making it easy to write reports and letters means everyone now has to.
--
On the flip side, I've occasionally been really impressed by the quality of decks I've seen from developers. Not talking about looks-wise (that helps but especially not holding devs to doing that perfectly) but to actually using these materials to support their point and drive results. And I feel like that wouldn't have happened before.
I think our argument is essentially this: in the olden days, cars came with chauffeurs. Now, most people drive themselves. On one hand, that's an artifact of how many more people can afford a car. On the other hand, they can drive themselves to a stupid place. Both are true.
But my point is slightly different. It's not that today, more people have cars, but are much more likely to drive themselves to a stupid place. It's that cars - particularly in the United States - very quickly stopped being a tool that empowers the owner, and became a necessity that's required to interact with the society: with cars being accessible enough, architecture and businesses evolved to compensate. For many, this just means they're forced to endure hours of traffic every day. Is this better than before cars were ubiquitous?
Also, the dream of self-driving cars isn't motivated just by the safety argument, but as much - if not more - because a self-driving car is a chauffeur for the masses. It won't let you skip the traffic, but at least it won't be a complete waste of life.
Another example: every self-service website or app - ticket purchase, food ordering, government form - is replacing what used to be a human interaction. In person, over the phone, or (for a brief period of time) over e-mail. This has some benefits, but one major loss is that you're now limited to a dumb (and often broken) form streamlined for machine processing, where previously you'd be interfacing with a person - so good luck if something breaks, or you have some custom need. Plus, self-service sites very quickly become time-negative: with a person on a phone, you can describe what you need in high-level terms. You don't have to search, explore, read FAQs, and type so much stuff.
What makes this into a problem is that you don't have a choice here - self-service systems are cheaper, they justify cutting human-level support to minimum. Most companies still offer you a phone number to call, but instead of quickly reaching a competent human, you're forced to endure an automated system designed to tire you out, increasingly often featuring a broken "AI" voice assistant, that is only ever good at one thing: keeping you from reaching a human. And then, half the time, the human consultant can't do anything because "the computer says 'no'".
(Note how the same companies that make you suffer through an hour of Muzak and "smart" "AI" "assistant" when you call them, will then have a nice, pleasant, real human cold-call you, to upsell you something that's usually a bad deal for you. It's clear what's driving all those "helpful" technologies: company operating expenses.)
And over the past few years, and particularly in the past few weeks (i.e. since ChatGPT was released), people talk so much about using language model AIs to interface with services and technology - mostly without realizing that what they're asking for is a cheaper, inferior form of having a person on the other end of a phone/chat.
Recall that data science itself is 80-90% data cleaning and basic exploratory analysis, so that means no more than 1-4% of their overall time is spent doing the work that they're actually trained to do, and that you are actually paying them to do.
They look productive because they are always busy. But really they need helpers (data engineers and dev/data ops people) to do all the random software work, so the data scientists can actually build models, meet with senior leadership, and all the other actually value-generating work that justifies their high price tag, and moreover keeps them happy with the work so they are more likely to actually stay at your company.
For example, Newton / Leibnitz's invention of calculus created much more than enough value to pay for every single mathematician ever since, and that wasn't even all they did.
Better to have a more relaxed and achievable target of 100 units with a more efficient way of working than to have a more difficult target of 50 with a less efficient one. Right?
It is about having misconceptions about efficiency and productivity, and that people being busy does not mean they can't be more productive.
Seems like this whole step could be eliminated if they only protected the areas that weren’t going to be welded. You don’t care about cosmetic damage to surfaces you’re going to weld on. Just be smarter about where you put the plastic.
There might be good reasons why they did that though. My experience in factory settings is that there is always a reason. It’s just sometimes a really dumb one.
The basic gist of the story is an assembly line improves efficiency in a factory. Well no duh.
The translation of that lesson to a programming environment seems like it would be really fraught, encouraging people to do stupid things like write more unnessary lines of code.