The Red Beads Experiment (2019)
medium.com
medium.com
Both input and output metrics ostensibly play an important role when attempting to manage a system (especially when leading people), but the most crucial aspect is to start with a good causal model or put yourself in the right frame to discover one with very few iterations. Otherwise you will most likely steer the system into a crash and move on to the next one and repeat the same travesty.
> The contrast between the meaningless fixed nature of the game, and the narratives we tell ourselves as we are playing causes the feeling of the absurd. "We are doing poorly," "We are happy to be ahead," "We are nervous about being passed." None of these stories have any meaning, since the game merely exists, and the outcome is simply a part of that existence.
The workers do a thing, and management creates a narrative based on the outcomes, even though the outcomes themselves are purely a product of the system, and nothing about the workers. But they expect a narrative, so they'll create one out of whole cloth if necessary.
It’s a fun little eye opener that starts conversations. I wish more of those conversations ended up moving decision makers
https://commoncog.com/becoming-data-driven-first-principles/
Human workers have hand skill, learning, intelligence, wisdom, introspection, communication, colour and pattern recognition all of which are not being utilised making the job repetitive and unfulfilling.
Then the threat of firing based on a random metric - this would create a culture of fear and one would expect a high turnover of any ‘good’ employees.
I agree this would be a horrible place to work.
Which is the genius of the allegory - how the wrong metric can make efficiency and progress impossible.
Of course even a good metric becomes corrupted according to Goodhart's law:
“ When a metric becomes a target it ceases to be a good metric because people will seek to game it.”
‘The Wire’ remains a salutary lesson on how all stats when tied to pay and promotion will get ‘juked’.
A metric can measure nothing but random chance ( there is always some randomness that is rarely accounted for ) - this makes management impossible and workers disempowered.
Even with good stats there are too few Baysians vs Frequentists in corporate numerical analysis.
In terms of the company in this analogy a useful exercise is to imagine how to improve things.
An immediate solution could be to allow the workers to make multiple sequential draws and to discuss what percentage of red beads constitutes a good draw and discard and resample if more red beads than that.
Let workers share their best practices and it may be that there is a subtle skill that can be learnt about how one wields the paddle in the box of beads to steer it away from red clusters.
Or divide and conquer and take fewer beads and allow the workers to discard all red beads, then combine the pure samples and batch up to correct size.
In this modified scenario, the manager would have initially known that there is a problem, but then the manager thinks that he fixed the problem and his team is merely underperforming, but in reality the underlying problem is still there, but it no longer turns up in the metrics.
If the company is crazy enough to put people on PIP based on 2 low-precision data points, and fire them based on 4 data points... it's not going to be a good place to work at, and nothing will get done. If you are looking only at the numbers, then at least use proper statistical methods. Or stop treating people like numbers and treat them as humans, this'll give you much more data to work with.
Also, when done well, the teacher acts the part of the manager who keeps pushing people even as this realization spreads throughout the group.
The meta-lesson is something that Deming emphasized over and over: Don't rate your workers based on the outcome of a random number generator.
If those (in-story) managers had a basic statistics knowledge, they would have said: "Wait! The variance is too high, you cannot make any decisions based on 2-4 points". Then they'd do some statistical tests (or at least plot points with error bars and eyeball it), and realize that the numbers given are random noise and are not actually related to individual performance.
But those managers apparently flunked their stat, or they forgot it all, so innocent people got fired, and project got cancelled.
A sad story about why math is important even if you are not in STEM directly.
The point of the thought experiment is to make us think of what the system contributes to team/individual performance. In reality every procedd might have a random component, but it might also have a feedback component — e.g. you give a project with many red beats to a team that did well before only to have them (to us: predictably) underperform, then you give it to another team etc.
The lesson here is to understand this underperformance doesn't necessarily mean they got worse, it just means they got the harder project to pull off and other teams might have underperformed even more.
If you as a manager focus on the totally fictional underperformance of one team vs the other you optimize things that had no bearing on the outcome. In fact you will make things worse as you make yourself believe the (in reality worse) team that only handled red-bead-free projects before will perform better.
A variant of Gell-Manning amnesia arises when real managers see this demonstration, go back to their real jobs, and think "of course, my team's processes aren't as stochastic as that."
But then again, managers are also beads.
A smart worker will eyeball their draw for red and redraw a few times until they get a good draw.
There may be other ways to flick off red beads by tilting and tapping.
None of this is excluded behaviour at least in this article.
That isn't the conclusion, you can have a management team that all technically understand statistics and they still make the same mistakes. It is depressing to watch it play out in real time. The problem is when people don't identify that workers are just doing what they are told and personal effort isn't the major factor determining system outcomes.
Going through the article, the important part about the narrative is the misconception that "performance outcomes are being determined by the worker, so managing the workers will get better performance!". The management is being done by statistics throughout even though the quality of that management is bad.
Actually using statistics would tell you that your numbers are random noise and can’t be used to make sensible decisions. It would tell you that performance outcomes aren’t being determined by the worker. There might be better ways to figure that out (in this case, taking five seconds to look at the process) but statistics will get that job done.
Management was asking the question "given that the workers control the process and we know that there are differences between people, which workers are performing the best?". And they were doing the best they could under pressure with the incomplete information they had. It was a stupid model, but that is why they are going to an Ed Deming talk - the median manager defaults to stupid and needs a bit of help.
Training people in high school stats doesn't necessarily get better results at model selection, some extremely educated and capable people are still stupid about that part.
Yes, actually understanding what the hell is going on is really helpful, but as a backstop, it’s good to have the basic stats knowledge that you can’t come to any conclusion from a handful of samples.
You're working from a model where the performance could be the same and then doing the obvious statistical checks. Typically in the wild management have sufficient knowledge to do those checks if their model allowed for performance to be equal. But they don't have a model where that is a reasonable possibility so there is nothing to check. They're doing the correct statistics for the wrong model. And the model happens to be really stupid but that is managers for you.
You can teach them all that theory and they still won't apply it. The problem isn't usually a gap in statistical knowledge. Take another look at how Deming framed his experiment - if you look closely he is imparting statistical knowledge but that isn't where the real focus is. He's actually attacking the idea that the workers have any influence on outcomes.
If your point is just that management could have that knowledge and fail to apply it, I won’t argue with that. But I definitely wouldn’t call it “doing their best” when that happens.