The Shitty Stack System | Hacker News Reader

The Shitty Stack System

tryingtruly.substack.com

124 pointspositivesum50 comments

So recently I was discussing with a colleague how misalignment of incentives has shafted our team in a certain Big Tech.

I'm part of a high performing team, and for the last few years we have been delivering high quality output while hardly increasing our team size.

We have a sister team that is, to put it politely, not as high performing. They have been struggling to deliver even though the have almost twice as many engineers.

Last year we got some headcount to grow our team, but leadership decided to shift it to our sister team, since they had problems reaching their goals (you know, the squeaky wheel gets the grease). This was fine with us since we weren't in a particular hurry, and we would pick up the headcount later.

Fast forward and we have a hiring freeze, meaning that our sister team was able to grow headcount, while we can't grow ours.

The double whammy? since we don't have as many people as we expected, we couldn't pick up projects that were in our roadmap. Guess who got those projects instead? our sister team.

Morale of the story? do a shit job and you will be rewarded with bigger teams and more scope, which in turn will translate into promotions and bonuses.

Decades ago when I was getting started there was a team that fell way behind and everyone on the team worked weekends and many days until midnight and finally delivered some buggy arguable product 6 months late. They were hailed as heroes. No one even noticed me and my team delivering on time. I learned a valuable and entirely inapplicable lesson.

IME this can sometimes be overcome with a bit of self promotion. Don't let accomplishments be forgotten or overshadowed. It is a political game.

Also teach the execs that heroism doesn't scale. We all want to be entertained by the hero stories, but no one actually wants to live in a world that requires high-visibility heroics.

I understand this intellectually but it is just exhausting

> They have been struggling to deliver even though the have almost twice as many engineers.

My experience tells me that they struggled because they had twice as many engineers.

I don't think I've been to a 20+ person project which could honestly be called "successful". Hard to say if it's a symptom or a cause. Personally I lean towards the former, because when someone tries to throw people at a problem they either don't know what they're doing or delivering is not their priority.

I've had the same experience at the big tech company I worked at.

I expressed these frustrations to upper management and the explanation I was given was that for many of these big companies, their goal is to grow their 1,000 sub-businesses to 10 million ARR businesses each. Growing these 10 million ARR units to 100 million ARR is much more difficult than going from 0 to 10 million, and so when they have a successful unit, they disproportionally reinvest that success into the poor performers (the 0-2 million ARR units).

This felt backwards and unfair to me as an engineer on the teams that worked hard to get to the 10 million mark. It really sucked seeing my team lose headcount or engineers to less successful and poorly designed/managed projects. But from a business perspective on the larger scale I see why they do it - I just think it creates a poor engineering culture and, in the long-term, a sub-par customer experience because velocity in product development slows when projects are de-invested or put into KTLO.

I kind of see your point at the macro level, but in our case we are both working in the same product, and contributing to the same bottom line. I see no strategic advantage on shifting resources from a high to a low performing team.

And of course the triple whammy happens when low quality software written by one team is handed over to a better team to fix it. I'm going through that right now. We have an upstream dependency that it's always causing all sorts of trouble. We've been told by leadership that if we notice issues we should go fix them ourselves rather than asking the other team to fix it. Go back to the incentives, what message are we sending the other team?

Frustrating dynamic. Something similar played out for me in a different industry, where a company doled out a bunch of different parts of a large project to different subcontractors. The subcontractor I was working at was apparently the only one that hit the dates and deliverables. So we got put on pause (aka defunded) while the other contractors were paid to catch up.

I never was sure why some of the unfinished work wasn't shifted to us, but the end result was the same as yours -- getting penalized for doing the demonstrably superior work.

is it common to roadmap features your team can’t deliver at their current size? that sounds like a problem.

also, the team that takes on the most work but does it poorly getting the promotions and bonuses also sounds like a problem.

both of those thing aren’t problems with developers though—theyre management problems.

i would interpret the moral as bad management leads to sub-optimal outcomes for developers. not advocating correctly for headcount, and not showcasing the high quality work your team does are literally what your manager should be doing.

>>[...] both of those thing aren’t problems with developers though—theyre management problems.

Sounds like the sister team had a better manager in some aspects of "managing" (e.g. managing the message up the chain) compared to the OP's manager who may have been great at facilitating (or minimizing meddling with) the high performing team but was less skilled in managing up.

agreed, i like your perspective. so hypothetically, what do you do here? ideally these managers should learn from one another.

should we send them to a tropical off-site?! joking

> is it common to roadmap features your team can’t deliver at their current size? that sounds like a problem.

Remember, we had the headcount last year, and we planned our roadmap accordingly (in fact my company tends to plan at least 1 year out to a certain level of detail). When they shifted the headcount, we just pushed the projects back. However this year came and we didn't have the people to deliver on our roadmap.

About your second part, I cannot say since I am not present in those meetings. I have always found my manager pretty good, but there are some things that are just outside his control.

This the type of bs that would make me start thinking about starting my own company.

I'm not brave enough to start my own company, but I'd definitely like to try a much smaller shop where people can't play politics so hard.

Can’t be promoted if you’re too good/irreplaceable.

It could be compensated at least by a pay raise.

Company I worked at had this inflicted on them by McKinsey, untold woe followed. There were suicides, many nervous breakdowns.

A colleague (a great man who is utterly unregarded professionally but is one of the most profound thinkers I ever encountered) had a brainwave which was to calculate how much the stack ranking was costing. We discovered that every year the company was spending nearly 15% of its budget for management grade employees on the process. This estimate came from two sources, HR's official requirements and guidance and surveys of the management group.

It turned out that large management consultancy found that it was doing the same at the same time we did; this information and my colleagues presentations of his teams research resulted in the system getting dropped.

God damn the people at McKinsey that inflicted this so widely. "In search of excellence" must be counted as one of the artifacts and efforts that has been most harmful to democracy since 1945. I am waiting for an apology.

I should add, on the journey to the evidence that finally killed the stack rank we did some interesting empirical work. This was in collaboration with an organisation that actually became part of the dreaded firm.

Still, it was good stuff. In F1 racing you have a unique opportunity to study employee performance empirically. Why? Well - every year there's a new car. The components that get on the new car get on it because of an empirical process. If your team can lighten something, create more drag or reduce drag or increase power then your component gets on the car.

This insight allowed individual performance to be tracked. The results - well it turns out that for almost the whole workforce it's luck; future performance cannot be predicted from past performance. You screw up one year, but then the next you knock it out of the park. There was an exceptional group though. Being in the highest percentiles in successive years was strongly predictive of being in the highest percentiles for future years.

In the follow up work this was investigated and it was found that managers could reliably identify the individuals in their departments that would be in the group of "persistent out performers".

There was no similar group of low performers.

The London Review of Books had a pretty damning article on McKinsey and management consultancy: https://news.ycombinator.com/item?id=33869800

"One gets the sense that Bogdanich and Forsythe think the consultants they write about are rotten apples, but the barrel is sound. Their own material makes clear, however, that all the services often spoken of as merely helping businesses and government departments run more efficiently – management consulting, audit, software development – are in fact focused on enabling capitalists to enrich themselves further without the inconvenient interference of workers, taxpayers or regulation. Thanks to the hegemonic model McKinsey and other management consultants invented, these firms not only make and remake businesses and government in the image of their laissez-faire fantasies, but see homo economicus as the last word in modern selfhood."

(HN comments: https://news.ycombinator.com/item?id=33869800)

https://www.alfiekohn.org/punished-rewards/

"In this groundbreaking book, Alfie Kohn shows that while manipulating people with incentives seems to work in the short run, it is a strategy that ultimately fails and even does lasting harm. Our workplaces and classrooms will continue to decline, he argues, until we begin to question our reliance on a theory of motivation derived from laboratory animals."

That was 1993. The people at Microsoft, as well as the author of this substack, would do well to have a think about Kohn's arguments.

MSFT stock up 100x since then. Edit: I guess this was your point, criticism was either heeded or was off.

MSFT stock stagnated for more than a decade and jumped once they fired Ballmer and got rid of his stack ranking system. https://www.macrotrends.net/stocks/charts/MSFT/microsoft/sto...

I heard it's back now (not sure if for this year only due to the reduced budget, or just ongoing).

Sure, and NFT prices went up times infinity. Doesn't mean that's benefited human society any.

I’d argue Kohn’s follow-up work isn’t too far out of line with the author.

I've only read that one, but had been meaning to read more. It's been bumped up the list. Cheers

This reminds me of Bret Devereaux's timely observations about coup-proofing in authoritarian regimes. Fostering internal competition absolutely wrecks your organization's efficacy in accomplishing goals, but if you're operating from a position of dominance relative to external threats it can help keep power concentrated in the hands of the people who aren't forced to participate.

> For low legitimacy forms of government, like autocracies, the concern is squarely centered on internal stability, and here we see a wave of armies designed primarily for ‘coup proofing.’ Russia’s military is actually a pretty good example of how this is done. An authoritarian government is looking to both maximize the ability of the army to engage in repression while minimize it as a threat to its rule. ‘Coup proofing’ of this sort follows a fairly consistent basic model (which I may elaborate on at a later date). First, command needs to be divided so that no one general or minister of defense can turn the whole defense apparatus against the leader. You can see this with how the Russian armed forces were fragmented, with Rosgvardiya and Wagner Group not reporting to the ministry of defense, but it also extends to the structure of the Russian Ministry of Defense, where the Army, the Navy and the Airborne forces (the VDV) all maintain infantry forces. Setting things up that way means that, in a pinch perhaps elite, well-paid and loyal VDV forces could be used to counter-balance grumbling disloyalty in, say, the army. Of course such fragmented command is really bad if you need to launch a conventional war, as, in the event, it was.

https://acoup.blog/2023/06/09/fireside-friday-june-9-2023/

A company I worked for did worse than stack system:

This company has many buses. And it started to measure every movement of the bus maintenance team (with the help of a consultancy). How long does it take to remove the engine? 1 hour. And the wheels? After that, they made a calculation and fired "unnecessary employees".

The remaining people couldn't do the job, it wasn't possible. An engine with a bad bolt is much harder to remove, easily doubling time.

In the end, a steering rod broke, killing the driver and others on the bus.

Nobody was arrested, nothing happened.

My experience in several companies... led me to create the theory (or have I read it somewhere?) of Entrepreneurial Theater.

All the methodologies I was forced to work on basically didn't work, someone with power wanted to do that because they did. Those who do well in this theater are not good employees, they are the ones who perform best in the current play.

> Those who do well in this theater are not good employees, they are the ones who perform best in the current play.

This so succinctly captures my experiences in the last few decades, both at various corporate exploiters, and with customer/partners.

I spent many years in denial regarding my career/industry being so predicated on fluff.

Now I just look at it as a paycheck that enables what really matters. Good times with my family.

Seems to me the key issue is few people in corporate leadership roles actually want responsibility or care about the mission. It's the classic Peter Thiel line that "competition is for losers". Far easier to change something arbitrarily without considering the consequences, because it makes you look "dynamic". If anyone criticizes your plan, talk about "necessary trade-offs" and just shrug off the criticism.

I see it at my own large corporation which is going through a top-down reorg right now under the new executive administration. For those of us actually creating and delivering products, the conversation is: "So this plan is somewhere between meaningless and stupid and will solve a whole lot of nothing, how can we use it to get more funding?" No one outside of fresh college grads expects leadership from the executives, they're largely clueless assholes engaged in corporate peacocking and backstabbing for their own shallow reasons, but the clusterfuck of high-level backstabbing spits out money; so like kids playing in the sprinkler, we position ourselves to catch the water as best we can.

I wish it operated differently, and at smaller companies where the executives are closer to the ground or have an actual vision it appears that it can. But the norm is mediocre and shallow leadership, like those aristocratic 19th century British officers who could literally buy their commissions, and did things like haul grand pianos into war zones.

Competition drives innovation when its external, if its internal it would need to be on a per team basis, and even that feels risky. Pitting members of a team against one another seems counterintuitive. I think of every company I join as a ship, so if another person from another team or my own is struggling with something and I can throw any advise or look and help, I do it, if I cannot because it is out of the scope of my area of knowledge, there's a small chance I might peek and tell them sorry I don't see anything that sticks out. Of course if I'm super swamped I might not at all, but if I'm reasonably busy or not busy enough, I'll pitch in.

I don't think a company ever really recognizes the value in teamwork that I have seen, but it doesn't matter to me, the way I see it: we're all on the same ship, if you sink, I might sink too. Even if I never get rewarded for my efforts, I would much rather take the gratitude of other developers. As a result, when I apply for jobs I have several people who can vouch for me having worked with me, and I'd gladly do the same for them as well.

“Incentive structures work, so you have to be very careful of what you incent people to do, because various incentive structures create all sorts of consequences that you can’t anticipate.”

- Steve Jobs

My current employer has similar problem. Company hired executives from a certain company about two years ago (hint: nepotism) and shifted towards incentivizing teams who met OKRs and OKRs were only product focused - features.

Though this makes sense on the surface, what it ended up promoting is doing whatever to meet the OKRs. Short-term solution was often acceptable leaving huge tech debts and it is hard to justify doing anything to maintain system complexity in a reasonable bound because they do not directly correlate to an OKR.

Personal developments aren't part of OKR so junior engineers who should be focusing on learning struggled.

Team collaboration became a hostile because dependent teams will call out other teams for blocking a feature releases.

Executives shared during all-hands their intent isn't to neglect personal development or team collaboration but problem was it doesn't matter to employees because those aren't what's incentivized.

Perhaps if the team or personal performance review required feedback from them (360 degree) and takes a non-negligible portion, then things might have been different but it's still all about what I got done

> Though this makes sense on the surface

Stopped reading your comment at this point to not bias my reply, but - how does it at all make sense on the surface?

Would you like to get this feature-rich product? https://t.ly/6KUd ...

Depends on who your customer is. In large enterprise markets it tends to turn into feature bingo

It wasn't just Microsoft - many companies were equally infected by this evil meme coming out of Jack Welch/GE which of course imploded.

Yes it's strange they wrote this whole article without bothering to investigate the origins of the system. Not even a cursory look at Wikipedia apparently: https://en.wikipedia.org/wiki/Vitality_curve

It could actually be even worse as Welch was also keen on something called “Topgrading” which mostly never really caught on.

https://en.wikipedia.org/wiki/Topgrading

Maybe the team should be resilient first and foremost: when high performing members leave voluntarily, it can readily accept and incorporate new members, while maintain a certain work output and high morale.

Such a team has a reasonably amount of voluntary turnover. Too little turnover and too high turnover create different problems. Either teams stuck in their ways, overly rigid in roles, and overly dependent on certain heroic members, the org is terrified to lose. Teams with too much turnover lose valuable experience before it can be passed on.

In my mind good teams are teaching and learning machines: members adopt new/different ideas, different roles, new members, by being given safe space to fail and be curious.

I don't think artificial turnover is the solution. Teams need good documentation and design or else onboarding/offboarding will be painful and a huge distraction. If you've got good documentation and design methodology, then there is no need to churn the team artificially. Keep documentation up to date, design well, and redesign areas that are tech debt or not done well. That's easier said than done though. There is no magic bullet.

> I don't think artificial turnover is the solution.

I didn’t see this being suggested in the parent post.

I actually think there’s optimum voluntary and non artificial turnover where people naturally move to other opportunities. And conversely a calcification where people get stuck in a comfort zone.

I don’t think people should be forced to leave. Instead I paradoxically think people should be hired more interested in growth and this more likely to leave / move around the company / try different roles.

Who defines calcification and comfort zone? Be careful you don't confuse a mature stack with subject matter experts who can execute effectively as "outdated calcification". I would be wary of someone who wants to hop around a lot. Are they adding tangible value to every team or disrupting things before bailing to the next team?

The entire second paragraph is about finding optimum turnover to keep the team fresh and dynamic. I think it's impossible to control organic voluntary turnover, as you don't control people's lives/desires. The alternative would be artificial turnover, moving people in and out of teams as needed to meet this goal. I think people joining and leaving is disruptive regardless of it being organic or artificial.

> “People responsible for features will openly sabotage other people’s efforts. One of the most valuable things I learned was to give the appearance of being courteous while withholding just enough information from colleagues to ensure they didn’t get ahead of me on the rankings.”

Ah academics are well versed in this method! You would think science would be an endeavor to share knowledge, but in the current cut throat scientific world this has become a norm. Research must look reproducible but not actually be so without significant work to fill in the gaps. It doesn't hurt if it covers up actually unreproducible results either as long as it can get in a top journal.

It also squashes genuinely novel research since it'd be too risky. Better to do small iterative tweaks.

Watched this come and go at a previous employer and the toxicity it generated was staggering. It’s especially awful when you work on a small team of bright, functional people who are all doing well: no matter what, management says someone has to get whacked. We used to joke about teams trading their bottom-5% employees every year to keep from having to hit someone useful, or deliberately hiring someone useless every year to make sure we had a bottom candidate in our back pocket. Looking back on it, all that joking was just a way of dealing with being afraid you’d be that guy next quarter…every quarter. Once that system died a well-deserved death, the amount of inter-team cooperation and general morale improved noticeably across the company.

Can confirm, worked for a while back in 2011, never saw more dysfunctional, hypocrite and senseless organisation ever since. Now I consider it hilarious, but I remember how terrified I was of kafkaesque processes I were involved and what CPE was. I still keep 4 pages list of KPIs i have to be ”green” with.

Even better. Managers under a director are stack ranked. The lowest ranked manager is asked to "improve soon or else...".

What does this manager do? Finds a scapegoat in his own team as the fall guy for poor performance. Sets the fall guy up for failure. Tells the director, that the team is suffering because of this guy. Fired the fall guy, buys more time for himself from the director.

Don't ask me how I know.

Steve Ballmer doing of course. He's got to be the absolute worst CEO of any major tech company ever.

What is Microsoft like now? How are people evaluated and compensated?

I assume not everyone gets the same salary/bonus, so what criteria do they use to decide who gets what?

[I'm not doubting stack ranking sucks--I was at Microsoft 2005-2011--I'm just wondering how they fixed it.]

It's not a bad piece, but the writer needs to discover what the difference is between "it's" and "its."

Also, it's "relativistic", not "reletavistic."

[P.S. I'm available for all your editing and proofing needs.]

Reply on news.ycombinator.com