Hacker News Reader

> While it is still a stretch, the title should be "Google's promo committee is killing Kubernetes", since no other FAANG-equivalent is using or contributing to Kubernetes in a meaningful way.

Apple puts Kubernetes on https://opensource.apple.com as a project they "lead and contribute to"…

> Very little work that a single engineer does can be assigned a provable revenue number. How does someone working on an internal build tool get promoted?

While I generally agree, I've made changes in our internal build tools that I can easily put a number on, although it's not a revenue number. Make build process 10s faster, 1000 engineers run this 10x a day, you're saving more than an engineer day per day of time. That's a definite impact right there, assuming your management is on board with accepting it :)

Apple is #32 in contributor commits[1]. Between SAP and Glassdoor, with ~7000 commits. You need to start adding zeroes and then some to get to Google's number of commits.

We shouldn't just measure in commits. I'd be interested to hear the case made. Where has Apple lead in Kubernetes? What have been their major initiatives? That'd be interesting to hear.

[1] https://k8s.devstats.cncf.io/d/9/companies-table?orgId=1

I really have no idea, I just find it amusing that they put it front and center on their open source website.

If you don't want the pointy-haired bosses to start measuring productivity by number of commits, start with not doing so yourself.

Commit count is largely meaningless and a low-effort way to fling around "contribution" numbers. For example, large commit numbers could be a reflection of a specific company's internal conventions around making a larger number of small changes as their own commit.

Now it's certainly plausible to me that Google is a much more major contributor to Kubernetes than Apple, but you need better numbers for showing that. For example, which companies have contributed to designing and implementing the major features in the last 5 releases?

Yeah, I'm thinking of some colleagues that can't remember any syntax that do 20 commits on the same line of code and don't clean it up.

There is currently an Apple employee on the k8s steering committee, so they could mean that.

Or, since the sentence is applying to a list of projects rather than to k8s specfically, it could be just a misparse and they're not claiming to lead each of the listed projects.

To be fair if you look at the last year of contributions they’re #12 which is better.

> Make build process 10s faster, 1000 engineers run this 10x a day, you're saving more than an engineer day per day of time. That's a definite impact right there

Agreed, and you can measure and include similar statements regarding your Kubernetes contributions. However, if what you have to show for it is making the life of a hundred random startups easier, well why exactly should a Google promo committee care? Unless it is the specific goal of your business unit to make this impact, then the work is meaningful again.

> However, if what you have to show for it is making the life of a hundred random startups easier, well why exactly should a Google promo committee care?

He is actually talking about this in his tweets: "fixing a bug in kube-apiserver might retain a GCP customer", now he can put revenue associated with that customer into his perf packet?

There are a whole lot of companies who can look at MTTR in terms of dollars, especially eCommerce. Rollback failures and CPU throttling can translate very directly to dollars when each request is a potential transaction.

The problem is that's a very strenuous argument to make (unless he could find an email that said that a customer was going to leave unless they fixed the bug) vs another engineer who found a new way of compressing emails and save the company X exabytes of storage space.

Keeping the ship running seems to be continuously undervalued at these companies compared to how they value building a new wing.

> he could find an email that said that a customer was going to leave unless they fixed the bug

Yes, otherwise it can totally be that his impact is not that large actually.

Could be could not be, hence the hard to measure part.

I've stopped using many tools because they created just too many issues. And in that case you will not write these emails as a customer, you just leave (as long as there are better options).

FAANG manager here, with experience in other industries. I have definitely seen that promo committees here value maintenance work less than other industries. I think the problem is that the committee is made up of individuals from other functions, so they really have very little idea about the services a candidate supports, they're just looking at the accomplishments written in the packet. In other industries, that's typically but the case - if you run a service that has a reputation for being unreliable, you're not even getting considered.

You still have to do the above and beyond work elsewhere, but in FAANG some people will get into the conversation by ditching maintenance and focusing solely on the stretch work, and they tend to have success getting promoted.

> Make build process 10s faster, 1000 engineers run this 10x a day, you're saving more than an engineer day per day of time.

I upgraded my home internet to one 2x as fast as the old service, so that I would spend only half as much time online, but it didn't work. Speed up the build process and the build will just get more complicated and the test suite slower, until it once again slows to the point of being barely tolerable. ISTM that part of Go's early popularity was from its fast builds.

Make build process 10s faster, 1000 engineers run this 10x a day

Depends where you are starting from. Taking the build from 100 to 90 seconds probably won't make a difference since people will be used to doing something else while waiting for the build to finish in the background. taking it from 10.1 to 0.1 seconds will make a huge difference since you can change your entire development process when builds are 'instant' and you can get constant feedback on any changes you make. Of course the second one is a hell of a lot harder.

or from 12 hours to 30 minutes. changes everything.

I think the tweet is a bit too much. I worked for GCP for several years before. Impact in promo is defined within the same group and aligned with the goals of the group. If improving the stability of k8s is among the goals, then no problem at all.

OSS projects are not built on top of anyone's mercy. For infra projects like k8s, companies just contribute what they need and Google is no exception.

> I worked for GCP for several years before. Impact in promo is defined within the same group

Did you personally go through promo from L6 to 7 or L7 to 8 since that's what the OP thread is about?(or have seen the process from sufficiently close).

I can see how your description of impact & promo would work when there's enough people inside the group to compare from (i.e other L4s, L5s, L6s), but when going L7 & L8 there might not be enough people in the group. Plus the folks at that level would be the one defining the goals, so it seems logical that they can't get promoted on that criteria.

Up through 7 is defined within the same group for Cloud, afaik, which is almost certainly the relevant portion of the company.

> Plus the folks at that level would be the one defining the goals, so it seems logical that they can't get promoted on that criteria.

Fwiw you start defining goals and L6 (or even 5), that's one of the defining characteristic of 6 vs. 5, you can't get to 6 from execution alone, you need to be setting roadmaps that impact beyond your team. And at 7 and 8, the scope of both people and technology get's larger. A strong-L7/just promo'd L8 would probably be expected to exert direct influence over an organization of around 100 fulltime engineers (caveat: I'm saying this from below, but it fits what I've seen).

Which is to say that the positions in k8s that qualify for L7/L8 promo-worthy work are to a first approximation, just the steering committee. And that looks like it's working as intended: the steering committee is full of people who are Principal or Distinguished (or more in some cases) engineers and manager equivalents from a variety of companies.

> Which is to say that the positions in k8s that qualify for L7/L8 promo-worthy work are to a first approximation, just the steering committee.

So you are basically agreeing that an engineer who wants to advance to that level has to go and start a shiny new project (open source or not) so they are, by default, on the steering committee or the equivalent.

The k8s steering committee is a rotating set of 7 people with elections of 3-4 members every year. You need a sustained record of contributions and leadership to k8s, but no you don't need to have been one of the original creators.

What happens when the promoted person is rotated off the committee?

The same thing that happens when any other L7 or 8 completes a major initiative. They find something else to work on. Many of the steering group members also lead multiple (in some cases, a dozen or more) k8s working-groups, but they presumably also have responsibilities related to their company (and possibly how their company uses k8s), so they'd focus more intently on those things, or perhaps they started some initiative while on the steering committee and continue to lead that particular thing even once off of steering.

No. Leadership is not a frozen thing. And leading a tiny nascent project doesn’t have remotely the scope required to hit these levels. It is absolutely not the case that “start greenfield projects” and “grow to 7/8” are connected.

And yeah, growth to these levels is hard. The expectation at Google is that the majority of engineers never hit 6 in their entire career.

Here it’s focused on k8s, but it’s not out of line if we generically apply it to AMZN. How many services look cool, have a great re:Invent kick-off, and then go KTLO a year later? I’m going to lay that at the feet of the AMZN promo process, which prioritizes creating something new over real and needed maintenance.

I’m going to further reduce this to lazy management. It’s easier to point to splashy announcements to show impact than it is to put in the work and help your direct reports quantity unglamorous maintenance work.

Not sure how long you've been at or spent at Amazon, but the promotion process doesn't prioritize building new things at all. It's true that people think complexity = build new things, but promotion criteria specifically encourage people to stick to simple things and build on top of what exists rather than invent something new for promotion's sake.

I'd argue this depends on what org and team and the culture surrounding. It can be vastly different in various nooks of the company.

When selling your work to a promo committee, building a new shiny thing lowers the communication hurdle.

Personal anecdote, at Amazon, my wife led a very challenging refactor of legacy code and didn’t get promoted, I built a shiny thing barely anyone is using yet, and got promoted.

> Personal anecdote, at Amazon, my wife led a very challenging refactor of legacy code and didn’t get promoted, I built a shiny thing barely anyone is using yet, and got promoted.

How much of that is the effect of shiny vs. dull, and how much is male vs. female (apologies if my assuming you are male was mistaken, but it seemed the way to bet)?

There were multiple factors, hard to know what percentage gender contributed, but I do think it played a role.

That's an odd bet to make. At Google promo committees were desperate to promote women. There were entire schemes set up to encourage them to apply more, to give them special help men didn't get etc and that was years ago. Probably it's worse (less fair) now.

It wasn't just process, either. I saw severely unfair decisions go the other way. A woman in my team who everyone hated because she was dishonest, staggeringly unproductive and had a habit of upsetting other teams, sailed through promotion several times into management. On the other hand team members who created entirely new products from scratch or were the backbone of their team got bounced. The patronage of another (female) boss higher up in the management chain seemed the most likely culprit, along with a pervasive "everyone's gotta help women get ahead" attitude.

Laughably false.

Even if you can prove a revenue number, it doesn't matter if they don't want to promote you. I've seen it explicitly stated at more than one company, that revenue impact is irrelevant and that you need to show a "large enough design and execution" to be promoted as a reason to just not give it out.

Not only that, but the "promo packet" and "promo committee" is fairly unique to Google. Other companies do things very differently, e.g. FB's twice-yearly (now yearly) manager-driven PSC+"calibration" cycle, which has fairly different incentives than Google-style committees of unknown engineering peers with arbitrary periodization.

Google's system is well known (at least, in my circle of colleagues!) to not promote healthy long-term maintainership, although it does incentivize point-in-time brilliance and solving deep complexity.

The system you talk about is mostly dead.

Google promos everywhere up to a certain level are in-org (IE not unknown engineering peers). They are done alongside rating calibrations. Google promos everywhere up to an even higher level are basically in-PA.

For cloud, they are in-org to even a higher level. So the stuff talked about in this tweet would have been evaluated not just by cloud engineers, but almost certainly fairly local cloud engineers.

My experiencing having been on promo committees for google for 15 years is that for every case in my org i see of google incentivizing the wrong thing (or disincentivizing the right thing), i see probably 5-6 cases of it doing the right thing.

I see a lot more cases that are like:

1. Person builds new shiny thing for ... questionable reasons.

2. They make a mess of the world by not making it easy to migrate people to it and get rid of the old thing

3. Things are in a bad half-state, where both things have to be supported and the new thing does not provide obvious higher value.

4. Google doesn't promote them, they get pissed off about it and post on twitter about how Google didn't care about product excellence or something.

Than cases like:

1. Person toils away on the right thing forever

2. Person tries to get promoted for doing the right thing and having impact

3. Google turns them down

4. They get pissed off and post on twitter about how Google didn't care about about product excellence or something.

Externally, of course, people can't distinguish these two cases because they look the same (angry people on twitter).

I have been promoted from SWE III to Senior Director at Google, working only on open source until i was an L7.

Agreed. I work in the PP's PA and was promoted on my first try for doing unsexy but worthwhile maintenance work on an important tool. It can happen. In my estimation the best thing to do if you want to get promoted is 0) have a positive relationship with your manager 1) read the SWE ladder for the next level 2) give your manager a document that describes your work using the language of the ladder as closely as possible. Said ladder does not say you need to deliver a shiny new thing, despite what is commonly assumed in this forum.

It's good to know the system I'm talking about is mostly dead; the committee of unknown engineers always felt bonkers to me. That being said, a ratio of 1:5 or 1:6 of bad incentives at promo committees still sounds pretty suboptimal to me — when I was on calibration committees at FB, it was super rare for me to come away with the feeling that the wrong thing was being incentivized or rewarded at an engineering level (oh boy was product management a different story, but that's a different post). There were a couple times where I disagreed with an outcome, but that was like 1:150 or better.

> While working on open-source projects might be tolerated, even on company time, it makes sense that it won't put you on a path to promotion unless there is direct benefit to the company from it.

The thing is - this is all theater. It's mostly made-up storytelling. An individual engineer's self-review doesn't actually speak to their direct impact on the company's success.

It's fallacy to act like you can map commits to dollars. But there's a lot of money to be made by HNers for trying, so I'm sure they'll act like they can do the impossible.

Define "killing kubernetes"? It's still pretty successful and the adoption hasn't slowed in any way I can measure. I promise you that some site you used TODAY is running, at least part of it, on Kubernetes.

Google employees regularly get promoted, at all levels, based on their OSS work - Kubernetes and other projects. We have dozens of people who work on Kubernetes, in one area or another, at varying degrees of depth. Is that "killing" the project?

Of course it is never ENOUGH. I'd happily consume hundreds more people. :)

> (All of these are examples of very senior promos at my own FAANG company).

you have your own FAANG company? remember always that "your" company will drop you in a moment.

Azure and AWS k8s offerings aren't a meaningful usage? Asking earnestly.

> While correct, the same is true for most projects at these companies. Very little work that a single engineer does can be assigned a provable revenue number. How does someone working on an internal build tool get promoted? Or a new training model? A large-scale refactor of a legacy codebase? (All of these are examples of very senior promos at my own FAANG company).

Which FAANG though? Google is usually singled out for this criticism more than the rest of FAANG combined. It’s possible that your experience at a non-Google FAANG might not be relevant for Google specifically.

(Note: I wouldn’t know first hand, I’ve never worked at a FAANG).