Time for a code-yellow?: A blunt instrument that works
nilam.ca
nilam.ca
It's the same with prioritization. I've literally had conversations that go:
Manager: I need you to drop everything and do X right now, it's top priority
Me: Ok, well I'm currently doing Y, which was top priority this morning. Which is more important, X or Y?
Manager: Well, they're both equally important!
Me: OK, sure. I can't work on both, which one would you like me to do first?
Manager: [uncomfortable thinking noises]
Manager: Are you sure you can't do both at once?
Me: Yes
Manager: [Pause] Keep going with Y. I'll see if someone else can do X
sigh
It fundamentally comes from a place where there are too many layers of managers, none of who want to make crucial decisions that can bite them back.
Aka leadership only in org chart but not in setting direction, leading from the front, or accepting tradeoffs.
Ultimately it all boils down to org chart leadership. If managers could be voted out of the org chart by people below them in the chart, they'd have no choice but to show more spine and help the people below.
But this is not going to happen in corporations. So the best thing to do is quiet quitting. There is a reason this exists.
A quick filter of effective management is asking the question: What do you hope to gain out of this job?
If the answer is a promo or ladder-climbing, you know that they will make sub optimal decisions that guide their ladder-climbing journey.
There's a happy medium somewhere.
At sope point the organization comes up with a code freeze, and a manager is afraid of changes near the freeze. Them said freeze keeps expanding as you get down to ICs, and eventually nobody gets to release anything in December, even if something is wrong and is costing the company users and money.
It's bad when people choose to quiet quit on management, but it's scarier when they basically are told to do it, in not so many words.
But they're saying they're doing "CI/CD". Why they do code freezes? Why are people still afraid to deploy on Friday?
Because they're not doing Continuous Delivery.
Do you really think that Amazon is deploying (non-emergency/operational) code changes during BFCM or re:Invent? Or that Fox/CBS/NBC are deploying code changes during the Super Bowl?
Also (unless you have some business need to deploy before the weekend) why deploy on a Friday afternoon? You're just asking for latent defects to page you in over the weekend. Save yourself the trouble and delay that deployment till Monday morning. Your team will thank you in the long run.
Sometimes repetitional and/or sleep damage isn't worth being uncompromising on idyllic CI/CD goals.
When should the CDN undertake updates? Continuously.
(Obviously I do get your main point: the NFL should not upgrade at halftime of the Super Bowl. But there is a time and a place for different models.)
CDNs and cloud providers are a bit of a special case. AWS definitely has soft blocks during tax peak and large sporting events, CDNs (guessing here) would likely be doing follow the moon(?) (i.e. off-peak) rolling releases
My disagreement with the parent comment was driven by lack of nuance
As time passed the papers would get moved from one pile to the next.
I've done this as a team lead/scrum master. You want to add something to the sprint? No probs - you just have to choose something the same size to take out. You want to take a person off the team? Sure, you just have to negotiate with our clients about which work gets taken out of the sprint.
It's amazing how many things are suddenly less urgent when someone is forced to make an actual trade-off rather than push the pressure down onto the delivery team.
I prefer no forcing. Especially when it is say a moronic VP which i don't have enough force to exert upon him. In USSR we had a cartoon where a greedy customer wants a fur hat from a sheepskin, and he asks whether the hat tailor can make 2 hats from the same skin, and the tailor says sure he can ... that goes to 7 hats, and when the customer comes to receive the completed order he receives the 7 hats as ordered https://youtu.be/gSpjDi2BrQk?t=193
Smart people understood what hats they would get given the sheepskin and the number requested. And less smart - well, it is the cartoon fun.
Most teams I've been a part of have the syndrome you describe at every level of the hierarchy. I'm in constant battles to reduce the scope of what I have to build to avoid spending a year before it sees the light of day. I integrated a team a little while ago to help them do better. 7 people, 12 features in progress. I remember an organizational-level presentation we had was presenting the 15 priorities and accompanying 11 focus areas for the next 6 months. In the company wide evals, on top of our "self-defined" personal goals, we also have 3 imposed ones (d&i, security and learning) that were imposed by the corporation over the years as a reaction to something, and no-one will ever have the courage to remove.
This is almost a disease, and I've lost faith in that it will be cured some day
If you can't cut something off, you cannot decide.
decision (n.)
mid-15c., decisioun, "act of deciding," from Old French décision (14c.), from Latin decisionem (nominative decisio) "a decision, settlement, agreement," noun of action from past-participle stem of decidere "to decide, determine," literally "to cut off," from de "off" (see de-) + caedere "to cut" (from PIE root *kae-id- "to strike").
> we leaned on Code Yellow’s so aggressively that they became part of our standard operating cadence.
Code Employees-Prep-Their-Résumés
Which sounds great, until you have two of those.
And in reality, quite often you'd see folders where the "top of the notes" stuff had more entries than the normal contents of the folder itself. So, ironically people would skip the top stuff and jump straight into the folder as normal.
Brilliant! I love it. It's Code Yellow all the time. If it worked once a year at Instacart, and Google made so much progress with it, we'll just turn "the Yellow" on all day, every day.
Then maybe they can have a Code Red for when it's super-duper important to "sweat" the problem.
> you walk away with the confidence that you can handle whatever comes next.
You don't even need a bonus, just enjoy your new found sense of accomplishment. And maybe a subscription to the jelly-of-the-month club [1]
Second place gets a set of steak knives
Third place is you're fired
Always Be (C)losing
It's the manager's job to prioritise. There is always more work than can be done, and always more problems than can be solved with the available resource. Trying to solve that prioritisation problem by "sweating" - ignoring employee's genuine and real need for work-life balance - is not going to work long term.
To the author: please learn how to actually manage. Honestly, you'll do much better. I look forward to reading your article about how you learned to manage problems instead of sweating your team into the ground.
Indeed. Even the words "sweating the problem" just sounds callous and inconsiderate.
It can even work for Google, where can pull in different team members at different times. But in a smaller company that means everyone's work-life balance is permanently stressed.
This manager doesn't get it.
* I promised myself never again. Never again would I call a code-yellow. Code-yellows suck, drain team morale, and they leave a lingering distaste amongst all those involved. Yet, during my 8-years at Instacart, they were our most effective and consistent weapon in ensuring we made meaningful progress on our hairiest problems.*
to be a really long-winded way of saying, "I'm utter shit at management, refuse to prioritize until it's a company-threatening crisis, and I'm happy to make my team suffer for my incompetence."
I'm sure the employee churn at Instacart was in no way related to this fact.
And now he's operating a PE/rollup firm.
Just as many don't care or see it as an inconvenience to be there. They might do less damage to their families by just being honest and skipping.
There's no shame in being honest to yourself and others, as you said.
My Personal Brand Consultant told me a beautiful wife with two kids and a late model luxury minivan are essential components of my fundraising endeavors! Round B investments are 27.3% less likely to succeed for single founders under 26 years old compared to founders with families.
This is much easier to do now compared to 30 years ago, but there's still a lot of pressure to conform to the spouse/kids pipeline.
Rather bourgeois remark. Unsurprising that that developed countries' populations are in question with gatekeeping like that.
The world is by no means a better place today, and there are still "far greater" conveniences people go through every day, young, adult, or old alike. Many are invisible now, but being invisible doesn't make them less valid or less damaging. If you still want to see some visible "far greater inconveniences," look a bit lower on the globe, to the Middle East and Africa. These people will be ancestors of generations to come if they can stay alive.
Being able to grow past a certain age without dying is a very low bar to clear at this age. Humans are much more sophisticated today in both single and collective forms. Our ideas are more complex, our behaviors are more sophisticated, and ethical and personal standards are higher... All in all, we are progressing. Technological progress is a byproduct and enabler, but human (mental) progress is the first trigger.
So, if someone wants a trophy child and never attends to them after being born, it's unjust to the child, and the trophy owner is entitled to reap what they sow during their development.
If a couple in a developed country wants to have a single child and raise them the best they can to surpass their parents and have a better life, I see no gatekeeping here. Yes, sometimes the best intentions bring the worst outcomes, but at least the intention is good to begin with.
It's rather selfish to justify bad living conditions and having a hard life because "our ancestors lived that way" in the past. With that attitude, you're the gatekeeper of progress and a better world that we or our descendants can create. It might be a little less crowded at the end, but it'll be their choice and challenge to overcome.
> "Business!" cried the Ghost, wringing its hands again. "Mankind was my business; charity, mercy, forbearance, and benevolence, were, all, my business. The deals of my trade were but a drop of water in the comprehensive ocean of my business!"
-- A Christmas Carol, by Charles Dickens
I've seen this happen a lot with mediocre leaders. "Code Yellow" equivalents happen because they weren't able to understand that A was really the most important thing, typically because B, C, and D were important for optics or politics, but not genuinely important to the customer or the problem at hand.
A "Code Yellow" is a useful political tool to move an organization to focus a bit more on problem solving by saying "I don't care about your politics, your org charts, whatever, just solve the damn problem." In that sense, it really does work.
* A is clearly important and will have to be "code yellowed" if it's not addressed in the next few months. But you've got to drop one of B, C, or D in order to avoid the code yellow.
* B is one of your best engineers' passion project. She's not a big fan of C or D, and she might run away to Google if you tell her they're more important.
* C is a promise you made to a customer to unblock some big deal. They might not sign if you tell them you changed your mind and would prefer to work on some projects B and D they don't care at all about.
* D is a demand from your VP to address some company priority, and he doesn't think the effort-reward ratio from B or C are very high. If you try to drop D he will probably get mad and tell you that you're not allowed to.
When I read this, I understand that when management fails, and wants something to be get done, because reasons, they can just pull all stops and declare temporary slavery until the problem is solved.
This should be normally a "once in a decade" event. Not "once in a year" instrument. Being proud of removing "life" and "balance" from employees' reality reads like the worst power trip ever.
I worked in similar environments. Never again. We have our "code yellow"s in my current job, but we know when it's going to come, and prepare ourselves and our lives. Go through it, pat ourselves on the back for the good job, learn our lessons for the things we fail along the way, and continue our journey.
Work is there to support other areas of life. Personal life is not something that we "do" in the time which is left from work. It's not the second class citizen when compared to work.
Did he failed at this notion so badly, so he turned to this workaholic, everything else is secondary lifestyle, I don't know.
My read is the management team are not very capable and that this gentleman is not a natural leader. Doesn't sound like he knows how to build systems in a planned and thoughtful manner and has embraced a classic lurching-into-crisis strategy for want of understanding any alternatives.
EDIT Also juxtaposes beautifully with the "‘keeping the lights on’ fallacy" he mentions. Those crazy engineers with their belief that they need to try and keep the lights on. Also, when I forcefully pull them away from that, why do the lights go out on Sunday???
You probably know way better than me, but in my experience, configuring things correctly on healthy hardware gets you 99.99% by default. Adding some surplus capacity adds another 9, at least.
Then you build from there (autoscaling, hardware failovers, etc. etc.).
I mean it fits perfectly well for a lot of "thought leaders" like this. They, after all, have the time to write oh so many blogs about it.
And what is the incentive for the rank and file to do this? Execs get millions of dollars worth of stock every 3 months. The rank and file isn't getting this. Why would the sacrifice their L and B?
To keep their jobs? Naw. For this, they will cut corners and lie.
Management practices have become so terrible in tech. It is no wonder that the best people keep switching jobs every few years. There is zero incentive to work under such poor management practices.
Why would anyone go along with this unless it came with a FAT bonus?
We have the concept of a Major Incident/Major Event (because the term Code Yellow is already co-opted to mean any administrative fault across a hospital which might impact the flow of patients).
These are all-hands-on-deck moments. They may be called by any member of staff who discovers an event which will impact our service, though it will be ratified by a Major Incident Manager (MIM). While the event is underway the MIM is God; except when it affects staff welfare. If a member of staff says that they cannot attend a war room for whatever reason, then the MIM will move onto the next person up the chain, even calling in directors if they feel it is required.
What is being described above is tech-bro shittery. Calling a major event because you haven't hit a sales target should keep the C-Suite up at night, sure, but calling in techs and devs and 'sacrificing the L and the B in Work Life Balance'? The C-Suite should be making the strategic decisions to reverse the decline, not suddenly drag everyone into a meeting to fix their lack of foresight and working towards an end that the average tech/dev cannot influence.
The problem is that, if a major incident is the only process in your organization that results in things getting done in a reasonable amount of time, there is a large temptation to declare them for everything. Somebody wasted 6 months failing to renew our contract with the payment processing vendor because other stuff was being prioritized above it? Open a major incident and we'll get them to call the vendor CEO at home on Sunday. We're a big account, so they'll answer. It's the end of the fiscal year and if we don't decomm this expensive system like was promised, a VP somewhere will lose their bonus? Declare a major incident and ElevenLathe will get it done in a few hours, even if it is Thanksgiving.
I would page on-call people at any hour of the day for real emergencies without hesitation, but always asked for orders in writing to page people (not executives, but real people) who weren't on call, or for nonsense like this. This got annoying so they started assigning other incident managers to the nonsense.
Agreed. We have a very tight description for an Incident and, therefore, a major Incident. We don't call MI's often but anyone can call one.
Not all catastrophes are immediate in nature. Most organizations have incident response protocol which works well enough for an immediate catastrophe. Non-immediate, gradual threats can represent a much greater risk, because there's no inflection point (beyond "way too late") where the threat becomes so imminent as to trigger that response. Code Yellows are a mechanism to artificially force that inflection point before it is too late.
The etymology is not green/yellow/red. It's just not-Yellow or yes-Yellow. See Stephen Levy's In The Plex (2011) pg186:
“A Code Yellow is named after a tank top of that color owned by engineering director Wayne Rosing. During Code Yellow a leader is given the shirt and can tap anyone at Google and force him or her to drop a current project to help out. Often, the Code Yellow leader escalates the emergency into a war room situation and pulls people out of their offices and into a conference room for a more extended struggle.”
Yes, I know. I was making a rhetorical point using a metaphor.
Um, no.
Today, Google has Code Reds, Code Yellows, Code Purples, and Code Greens... and this is after standardizing to remove other made-up terms like Code Mauve.
I wish I were making up these terms.
Purple is/was "developer experience is so bad we need to stop developing new functionality and make the current functionality usable."
- Code red: the situation is actively causing active business harm.
- Code yellow: the situation will cause irreparable business harm if not addressed in the next 3-6 months.
- Code purple: the situation will cause business harm if not addressed in the next year.
- Code green: things are not at risk of causing problems, but we still want to make sure we make progress.
At Google, all of these priority codes need senior VP signoff, which is to say that it is actually an existential threat to one of our main product areas (e.g. Search).
I only remember the RAM crunch (2020 or maybe 2021) being a code yellow, but it's possible it was downgraded after the first month or two.
Code reds don't always have to be met with a total code freeze, but they generally do preempt all work outside of incident response.
The point of a code yellow should not be to punish the team, and an appropriately-declared code yellow should be met with significant introspection from leadership about how we got into this mess and what we need to do to prevent us from getting into this mess in the future. It's a blunt tool that allows the organization to dictate that it's going to drop its existing commitments on the floor because they are simply less important than fixing the systemic problem.
You don't need a code yellow to try multiple things in parallel, or to ship a prototype without worrying about scale. A startup certainly doesn't need a code yellow to empower individuals to wear multiple hats. And if your team is spending 50-75% of its time on keep-the-lights-on work, then your systems are being held together by duct tape, and this is simply not sustainable.
Of course, but if it's not immediate, why does it warrant forcing people into a Zoom meeting on a Saturday while you're at your kid's Tae Kwon Do competition? If it's not immediate, why do you have to "encourage the team to sacrifice the ‘L’ and ‘B’ from Work-Life-Balance"?
Either it's an immediate catastrophe which requires immediate action, or it can be handled by giving it absolute priority during regular business hours.
And some military terms are never used. Reserve: the idle resource you have that you can deploy to stop a breach or to profit from an opportunity.
Nope, we're using Human Resources like people used to use machines in the 80s: you have to get them working 100% of the time. And when there's an emergency? Code Yellow. When there's an unforeseen opportunity opening up? Well, too bad so sad your competitor got it.
Sounds like nobody was sticking around long enough to even find out.
After ChatGPT’s initial release, Google famously sounded the alarm and brought Larry Page back into the mix, showing how this kind of all-hands effort can quickly organize everyone around a single goal.
Valuable lessons from a competent person. Great article!
I think that's what this post is saying, but all the posturing about forcing people to working harder to build "grit" is making it hard to read it as genuine. Clearly this is not a management issue, it's the team's fault for not having enough grit to work hard all the time!
This is true and important. For example, no matter how critical a project may be, it is understood that the first duty of any white collar worker is to show up to every meeting on their Google Calendar. When we are are allocating, prioritizing, and estimating resources for projects, what we are really talking about is the time left over after discussing ~anything that ~anyone may want to pull them into. Only an executive whose calendar is kept by an EA subjects meetings to the kind of cost-benefit tests and prioritization that we use for IC project bandwidth.
In one of those books management gurus right at the end of their workspan, one of them wrote that he did not believe a project was worth doing if it was not worth having a war room in which to focus people on the problem at hand.
Lowering Work In Progress is one of the easy ways to gain success...
What a douche. So it's always code-yellow, and now they'll need to invent a code-red.
But an organization usually has more than one problem that needs to be addressed concurrently, unless it wants to constantly be "fire fighting" problems that could have received partial attention earlier but instead have become emergencies.
So this guy is going to want to sweat more than one problem, which really just means doing a normal job of prioritizing things, but with more sweat. Just remove the L & B from WLB, permanently. Sort of like this boss imagines he does by writing emails at his family's events, but without the extreme upside potential afforded to executives.
Maybe if you work hard enough you won’t need to stay at such a bad job
in many cases deep financial bets have already been made before the code is even written or the idea properly sounded out. culture is culture..
human desire to appease the social order is stronger than rational decision making
Probably because their companies spend so much time deciding which task should be prioritized and having meetings about it that nobody can get actual work done.
> The biggest constraint I have seen teams imposing unconsciously is the ‘keeping the lights on’ fallacy. This manifests as the project team giving the project/goal 25-50% of their attention (although saying it is their main priority) because they feel that they need to continue ‘keeping the lights on’ with other work activities or projects.
It sounds like "keeping the lights on" means ... working on other things as well? It doesn't specify that it's to make sure those other things keep working. Maybe it's implied? I have no idea.