Lucky Lotto, chaos engineering but for teams
danlebrero.com
danlebrero.com
But even worse, you could also be assigned the "liar" task. In which case you were supposed to reply to emails but give intentionally wrong information some of the time. But in that case you would tell people that you were the liar and that your answers aren't to be trusted.
It seemed like a good way to make sure at least two people could do everything and that documentation was solid enough that you could recognize when someone was wrong.
I refuse to use email at work anymore because of the constant phishing tests. Sick of the gotcha mentality.
I use it now only to see if notifications came in from calendar or such. Even then it’s just looking at subject lines.
I then go straight to the app and check for actual message / event.
I'm 100% certain I've worked with people that did this. I never realized they were DR visionaries
If I could say things knowing that if I made a mistake, someone might friendly correct me with "Ah, you're the liar today, S3 doesn't support that format", I'd be happier to make them.
Another good take on this is the "Wheel of Misfortune"; Take a real incident (or synthesize one in testing) that paged someone inconveniently. They've already solved it - They're running the exercise. Now have everyone else on the team, individually or as a group, figure out how to solve it. Not from the debugging steps or postmortem of the incident, but before that's been shared - Have them all suggest what they'd look for, and how to fix it. Their most important resource is their instructor/adversary for this, so give them all the data, but no map to it. Further, have the instructor figure out how to respond to all the unknowns - What else was broken because of the incident? What would have happend if they tried different debugging steps? Builds team knowledge real fast.
The winner will work on some side project. Still work.
Not a solution to the bus-factor of people but a good solution to the "we never get to work on tech debt / platform work because features" problem. This alone is awesome about this. Everybody, including product managers, gets one ticket every week, even if you don’t want it.
This, if done right, will result in Product Managers providing a vision and consistent answers to similar type questions. Thus the team can learn to anticipate their answers for minor things (which helps even in week where the PM isn't the winner) and in weeks in which he is on vacation (or wins again) the team isn't stuck waiting for a week for an important question that blocks development. Team should avoid delaying the work for a week.
Try to bring one of your colleagues to do the task with you or under your supervision.
This is what to do, when you need to break "rule 3" (which states you have to be completely unavailable). It's a soft rule and I think the point is actually to break it a lot in the beginning. The "under supervision" part means, you are teaching another team member part of what made you the one with the bad bus factor. As time goes on, having to break this rule will become less frequent (which is why they wanted everyone to write down when they have to break rule 3 if you ask me)I like to differentiate between mechanism to make people realize a problem and actual solutions to the problem. Its very easy to say mentor someone to do your job but if that was doable and easy they would be already doing it. The problem is precisely that there is no real good way to do KTs. Forcing someone to solve the problem is one way to KT, but is that the most efficient way? I would rather gather data about what breaks down and come up with a more efficient KT mechanism.
The parts I mentioned are not the 'realization' part. They are the solution parts. They aren't the 'implementation details' of the solution, I would agree, but they are the solution.
If you ask me the problem is not usually that there is no good way to do KT or that it just isn't doable at all. The problem is that in most businesses due to their 'culture' (for lack of a better word - not to start "that" culture discussion) you do not actually get to do it. A good way to mentor and transfer knowledge for example is to do pair programming. Not many places allow for that and will look at you funny for even suggesting it (and I'm not even personally on the extreme end of that like some companies, where everyone literally pair programs 100% of the time - I like a sort of hybrid model, where people pair for as long as it makes sense to them, which could be sitting there "designing" for a couple of hours together, maybe dividing things up after a basic structure is in place and then working on their own for the rest of the day w/ some quick 5 minute sync ups and questions going back and forth from time to time). Another very doable way to do knowledge transfer is to specifically not give let the one guy that wrote that part of the code and knows it inside and out work on the next ticket that will need changes to that part. But many Product Managers/businesses/team leads will not allow that because it would mean that the task will be delivered slower.
The beauty of this approach is that you don't need to actively gather data, make a decision etc. Gathering this data is usually very error prone in that you can fill out forms and skills matrices and such all you want (been there done that), you always forget about something or it doesn't really tell you the whole story (skills matrices are particularly bad)
With this, it just happens! It's the self organizing way of dealing with the problem. I think you might be putting too much emphasis on the "completely unavailable" part, whereas I see the "it's a soft rule" part bigger. Instead of waiting for someone to go on vacation and then you find out that he was the only one that can do X (there's your data point that you missed during collection and analysis) and now you're in deep trouble, because he's touring the Amazon rain forest (i.e. definitely no cell reception there), you get to figure it out because he won the lottery and he's actually at work and can tutor someone through it all.
> But many Product Managers/businesses/team leads will not allow that because it would mean that the task will be delivered slower.
This is precisely the kind of communication breakdown I was talking about. Managers are thinking short term, forgetting all software needs to be maintained.
It may be a team personality thing, but I think there would be way more resistance to this kind of chaos engineering than a survey.
If you can get the chaos engineering "allowed", I think it's easier to actually follow through.
There are multiple scenarios I can think of here.
* This might be something that a team lead or a bunch of team leads want to initiate/try out. You ideally need buy in from at least your boss, potentially a bit higher up. * This could also be thought up/initiated by the development manager/VP of engineering/CTO level who now has to get his team leads to run this and might run into resistance from them actually or from Product. Sort of like with the technology based chaos engineering, where there might be a special SRE person/team responsible for running this, you could have a special team/person for this as well. If we were all in offices, it might be a bunch of tall dudes, physically escorting the chosen person from the team room :)
I'm currently in a high bus factor job, I'm the only developer on the UI - our CTO can do some small jobs here and there, but nobody's touched or even looked at the new UI I'm building (Go + React).
I want to be able to leave, but the way things are going - and the way recruiters are spinning up again - I'm afraid I'll have to bring them the bad news that I got an offer I can't refuse (and that they can't match; we're talking up to 150% pay rise / benefits).
What my company needs is a big bag of money so we can hire contractors to fast forward this project. Which will be a short term solution, but still. But this company isn't eager, it's run by mid-late career veterans who are happy with things bumbling along and a decent 15%/year growth. I can kinda respect that, but for my project that's not good enough. And it's an unsexy company, so they struggle to hire anyone.
That's actually a low bus factor, isn't it?
Perhaps "bus risk" would be a better term for this usage?
Just accept the offer and move on. It is the best thing you can do for a company like this.
> There is a rare alternative definition for the bus factor, namely: the number of people who are indispensable for the project. In other words, it is the minimum number of people who are a single point of failure. If using this definition, then a high bus factor is considered a bad thing (since the loss of any person included destroys the project), and zero is considered the ideal bus factor.
Downside: Takes time (and removes mentor from tasks deemed more important short term), costs money, people might need to get used to becoming mentors (I found most people enjoy this even if they cannot imagine doing it the first time around)
Upsides: Long term thinking and investing in employees is rarely wrong (imo)
[0] https://graphthinking.blogspot.com/2021/06/periodic-rotation...
Your approach is planned, with well defined onboarding and offboarding and with a report in the end. Personally, I hate it just for the idea of having to write a report.
The article approach in unpredictable. At the last moment, you know you are not part of the team anymore, and all communications are cut. People have to take over even if you are in the middle of something. It also involves managers. I prefer this, if anything, just because there is no report to write except if things go wrong (i.e. you break the "no communication" rule).
The chaotic version of your solution would involve periodically drawing two team members, including managers and having them switch teams immediately. For each individual, the period of change would follow an average but be random. And no reports :)
I'm expecting a similar mad scramble when I put in my two weeks at my current company (hopefully within the next two months, interviewing with various companies now).
Could you imagine afterwards being lectured to use common sense. Also, we have secret corporate policies you need to know about and follow. Resolve that catch-22. Repeat for a decade.
You'd be justified in never trusting them with your life or another loved ones again.