Hacker News Reader

Comment by SSchick | Hacker News Reader

Design doc culture at google is why I left the company.

I wrote a very high level doc shortly after joining the company outlining a fairly trivial task (one that has been performed many times across other product areas).

A colleague virtually took me aside and told me "this is not how we do things here". Which was kind of a wake up call as he told me to make sure to 'evaluate many more ways to get this done' despite the way I outlined literally being a slight variation on the recommended way.

When questioned about why this would be needed he said "it shows broad consideration".

"Fake work" is very much a thing at google, I wish I joined a different team.

View on HN

You get the behavior you incentivize. In the “early days”, design docs were a tool to agree on a direction and provide context to your coworkers about what you’re doing. Later, as new people joined at an exponential rate, they were told by well meaning managers to write the docs for perf reasons, and things kind of spiraled from there. Google’s culture became a cargo cult of itself.

Some companies I worked at after Google were reluctant to discuss their promotion process in detail, because they saw what happens when people microoptimize for it.

Microoptimizing for what the cargo cult believes gets you promoted is strictly worse than microoptimizing for what actually gets you promoted.

It’s a meh from me - either is bad. If you are in a highly complex and professional field (in the traditional sense of the word) single axis optimization hollows out the whole craft. Engineering is delicate and constant battling of tradeoffs, and decision consequences are often delayed by longer than average tenure, let alone a 6 month perf cycle. The more you McKinsey the process, the more mediocre and incoherent the results.

You get what you measure. You can't hire smart motivated people and then be surprised when they figure out to optimize for their rewards.

If you want a team that lands stable customer-loved products, make THAT your performance review. Turns out that's hard to do objectively and consistently.

According to your other comment, then, a company that "lands stable customer-loved products" would be a "hotbed of nepotism" that people shouldn't work for.

Nonsense - you can have documented metrics which are known and tied to performance.

For example:

* Require features to be launched for a period of time rather than in development or shoved out the door a week before perf

* Have metrics for customer satisfaction and oncall pages be incorporated

Etc.

Nonsense were a lot of your replies in my experience :), this "measure everything" is very much another cargo cult. Software engineering is complex, people are complex, thinking you can come with a true metric and who doesn't have it doesn't know what they're doing is the nonsense in my world view.

Have excited engineers, pay them well, have them work on something that is aligned with their career & deliver value to their customers, they don't really need much more than that. All the rest is simply trying to "processi-ze" what is a human

> You can't hire smart motivated people and then be surprised when they figure out to optimize for their rewards.

Never said that. I don’t have a solution. Well I have a partial solution, but not complete:

Reward all team members equally for the performance of the team, or even the company. For instance: the pie is divided into 3 pieces: company, team, and individual performance. Candy is given out based on performance of both.

It doesn’t fix the hackable metrics issue, but when people collaborate the side effects are better than when they think of their own promotions, and worse when they’re competing against their direct peers for a fixed bucket of candy per team. Such incentives are directly opposing collaboration between the very people that need to collaborate the most.

At some point, the cargo cult are the promo reviewers and then the two become the same thing.

The reviewers presumable have access to the true metrics.

What if I told you these are their "true metrics?"

If a company doesn't have consistent metrics for performance reviews uniformly across teams, get out of that company yesterday. That's just a hotbed of nepotism.

Please give an example of a metric for performance review for software engineering.

Conceptual examples (not looking to argue about the specific bar):

Senior engineers will:

* Design a significant project which lands with measurable customer impact

* Demonstrate expertise in at least one core skill outside of coding (test infrastructure, SRE, security, accessibility, etc.)

* Demonstrate leadership by either being a TL, owning and leading team pillar efforts (security review, etc.), etc.

* etc.

Mid-level engineers will:

* Own either a small feature end to end or a significant piece of a larger design

* Contribute to at least one non-coding pillar

* etc.

These generally have more areas and can be further granular such as "low-performing midlevel is X, satisfactory is Y, exceeding is Z"

Those are requirements, not metrics.

With sufficient granularity there is no difference. I can't give exact quotes without violating NDAs but variations on these have been the performance review standard at every major company I've worked at or known people who work at.

Agreed.

Just proves the point, this person just cargo culted what makes google so famous in recent years. If you only metric is to launch new projects, you’re only gonna launch new projects. Who’s gonna get promoted for maintenance.

>Conceptual examples (not looking to argue about the specific bar)

Again, not trying to debate what the specific bar should be, just giving examples of the kind of metrics you can use. What's your preferred alternative? Lines of code? Whenever your boss feels like it? Stuff like that is why no one believes a "senior staff superstar" from the hot startup of the week is any good and requires them to do whiteboarding.

What do you mean by "true metrics"?

The metrics which determine if someone gets promoted. Those decisions are being made by some group, presumably with guidelines. If the process for getting promoted is "this group does whatever they feel like" that company is a disaster and you need to get out.

Thus, there are rules, and they are written down. If they're not visible to normal employees, they guess the rules based on who they see get promoted (cargo cult) while the committee uses the true metrics.

It's not like becoming a manager is a secret cult where you're dropped in and you suddenly have carte blanche to do anything. At established companies there are rules and procedures, and while you will likely have access to more information than individual contributors, you won't just be promoting whoever you feel like without having to go through others.

Either way -- 'twould seem that if such a level of microstrategizing is what monopolizes one's attention on a day-to-day basis -- then one is definitely in the wrong place.

When I was at Google I mentored a lot of early career engineers and I always told them to write more docs than they think they need. Sometimes docs will catch problems beforehand but early in the career writing easy docs is a simple way to show independence and competence. It slows down projects but Google is probably gonna cancel that project anyway and my theory was "Google doesn't care about you so caring about it just makes you a chump". It was in most ways a great place to work but it slowly drove me insane.

But if they are going to cancel it anyway I would rather at least code and build just to have fun and at least have a prototype rather than doing the boring work. Also I feel like I can't get into right frame of mind of problem solving by writing a doc compared to if I am coding.

If the project is anyway eventually going to be deprecated and shutdown, maybe it's better to take one of the Google bike and go round the campus drinking one of these tasty granitas from the cafeteria.

Is there alcohol in it? If so, I would consider, but I would still take the alcohol and code the project as well.

The craziest thing to me is that at my location not that many people did the free workout classes. Between those classes and the food I was in the best shape of my life while working there.

I steadily got in better shape as they kept cancelling the projects I worked on and I slowly lost and sort of will to care about my actual job. The quality of my work ended up suffering a lot as I realized just how pointless trying to work there was for me but the pay was too good to want to leave.

Edit: I was unlucky though. I knew some people with more stable projects. I went through 4 cancelled projects in close to 6 years

If they like coding and didn't want to prioritize career advancement I would tell them to put effort into finding the projects (or parts of projects) they find more interesting to code and make sure they get assigned that. My advice was always personalized. The thing is that for most early career engineers they are already coding enough (and it's not like the code is generally particularly interesting) so most of the time the advice is to do more docs since the lack of docs was hurting them for perf/grad but it varied from individual to individual

This sounds like you are championing the very mindset that has made google crumble?

Write docs that slow things down, don't code/build projects, and do it all in the name of greed and the only justification is that it will get cancelled anyways (probably because no real work was being completed).

I was just a cog in a machine. The mindset didn't come from me but how they setup incentives internally and ran projects. To be clear, I ended up really resenting how it works and the advice was based on my experiences with trying to care (in fact I'm trying to get out of tech entirely). I ended up channeling my care and effort into building up coworkers rather than into the company or it's products and that's how I got through it for a while. Different people had different experiences though (it's a huge company after all), anything I say about my experience is very personal.

Edit: also if Google wanted me to really care about my job more than for self interest that relationship needed to be a two way street. The relationship of the devoted employee and disinterested company is common but to me comes across as a bit pathetic. I did a good job there purely because that's how I enjoy doing my job, if the company collapsed I wouldn't have shed a single tear (that's a lie, I miss the free food)

Caveat: Googler here.

One of the things that had been problematic at Google is the creation of many more products than are really needed, leading to the infamous Google Graveyard. One of the reasons I have come to like writing design docs for my ideas is that a lot of the ideas turn out to be crap when they get written out. Quite a few are good enough that they have already been done, or close enough, which my colleagues can then inform me of. This is a very lightweight way of trimming out useless projects.

And no, Reader wasn't useless. Different story.

Docs are very useful. I liked them, especially considering how slow development could be there, It's just that in my experience they incentivized them more than they needed to. For example writing a doc while you are already implementing something only takes like a day but it looks a lot better than a day of coding to people without context. Luckily at least for lower levels they switched to perf evaluators being closer organizationally and the manager being there so at least there was a little context (though often my manager was busy enough that they didn't fully know what I was doing either). There were a lot of specifics essentially check boxes for promo and to a much lesser extent ratings and eventually I decided it was easier to just give them what they want

The problem is with the culture, that the company does not want a "boring" product that can earn a stable amount of per year (say 100M). It only wants products that grow expotentially to billions.

Google could still keep the boring, but profitable products. Just like Microsoft does (although they bundle them).

But it seems that in google promotions are not made for boring things that work.

This is likely due to culture established by the teams working on old, mature products. There, you're going to work with at least 10 people to launch trivial projects. In my case, it's usually 20~30 people with a blast radius of 100~500. You're not going to casually 1:1 with all of them since you and they are all busy. If you don't get a good review from stakeholders, there's a good chance that some angry folks will come for you and make you roll back your launch.

In these contexts, design docs are meant as an asynchronous communication tool for information heavy topics. And this is so asynchronous, you will talk to those people join 10 years later if you product becomes so successful. I've been saved multiple times thanks to some random design doc from 2010 that explains the weird decision that still haunts us. This probably doesn't work very well on lean/small teams or less complex tasks. But engineering culture usually has its own rationale and context, even if it has become a cargo culture.

We do "one-pagers" if there's basically one, simple, straight forward way to do something; however, I'm with google on this one.

If you're designing something, and there's only a single solution under consideration, either there's no design, or you're not being thorough. Choices and tradeoffs are what make design

What if it is something really obvious and has been done so for last 20/20 times it feels almost embarrassing to consider anything else? You could list out the embarrassing methods, but it still feels useless work for show.

Then it is just a change and reviewed through normal code review. Design docs is only for when you do something that isn't obvious.

But then you don't get promo?

You don't get promo in your scenario either. A design doc for something that's trivial will be called out in promo packet review and not given much credence. If the packet is primarily comprised of such artifacts the promo will be denied.

It depends on level. For a junior, these are great design docs because they educate other juniors about engineering, and educate senior bad-doccers about good doc writing, and show developing skill in the art of doc writing, before the engineer has the additional cognitive burden of writing about something much harder.

Do good non-doc-worthy work => Get doc-worthy work => Write doc => Get promo

I don't want to get sucked into defending Google-style design docs (I have.. opinions), but on this particular point a couple of things come to mind:

1. Presumably you've written the doc to be read by other people. What's obvious to you might not be obvious to them.

2. If you have no "alternatives considered", it's an indicator that you didn't consider any alternatives. I can think of times when "this is obvious and it's the way we've always done it" sent me down the wrong path. Spending just a couple of minutes considering whether obvious == correct, and writing down why, is not a bad investment in the long term.

3. I can only speak for myself, but I don't enjoy being criticized and I don't enjoy being wrong. "Alternatives Considered" is often at the end of the doc and I'm tempted to avoid it because there's a very real possibility that I will find the process of explaining why we don't just do option B instead leads to a realization that option B is a better alternative than the plan I just spent all that time on.

9/10 times it's short and easy, but it's still a worthwhile exercise for the reasons above. Explaining the "don't do anything" alternative is a good way to reinforce the cost/benefit of what you're proposing, and it's usually pretty easy to put yourself in someone else's shoes for a second and think of the first "but why don't you just" that will probably pop into their head. Write down "because it won't scale" and you've saved yourself that conversation. (joking. maybe.)

> I can only speak for myself, but I don't enjoy being criticized and I don't enjoy being wrong.

That's actually a huge problem because it can veer onto intellectual dishonesty and being combative for nothing. Instead, one should be trying to look for the right/best path forward, regardless of what they thought. Should be easy to discard erroneous ideas.

The goal is not to be right. It's to find what's right.

>The goal is not to be right. It's to find what's right.

This is why I think design docs need to be lightweight and reviewed early. Design docs shouldn't be a masterpiece perfected in isolation over the course of days or weeks. That guarantees the author has calcified their opinions. When I was on review panels at Amazon, 99% of them were an exercise in futility -- the author had already poured concrete. It is very, very, very hard to avoid the mental trap of "Hrmph! I've thought about this more deeply than anyone else" that comes from living down in the isolated world of "doing design."

The earlier you get other eyes involved, the more likely people will actually listen to feedback and consider alternatives. You still, of course, need that heads down time to put in all the details, but the overall shape of the design shouldn't be a big reveal when you hit the design review.

So much this. Bouncing early design ideas (informally) off other stakeholders or at least more experienced engineers can save SO much pain. The reverse of that is true, too: giving early feedback on other people's designs can give them the insights THEY need not to inflict pain on YOU.

I think there is a risk of going too far in the other direction leading to design by committee. It’s often the case IME that a casual reader actually hasn’t thought through the problem enough to understand why their drive by feedback or alternative doesn’t make sense. A process that results in those things getting accepted may just be introducing more entropy into the design process.

Good comment. I think both of your perspectives are fair and manifest in the size of what is being designed.

Meaning that if someone designs something huge, end-to-end, it's going to be difficult for external people to address potential flaws within the design. A lot of interlinked parts requiring to have the full mental model loaded in the brain. So one should go step by step.

It's also true that not every feedback is equal and it's important to think in advance about how to address rebuttals since most people don't give in-depth feedbacks but surface gut feeling (which can be invaluable too, even if simply in terms of UX), unless they have wondered about how to solve the same exact problem before. That requires more time spent thinking about the design.

I had a wise friend tell me that engineers are always in a state of being wrong. What you build today you will most likely laugh at in five years. The goal for today is to build the least wrong thing you can. I used to argue more than I should have simply because I didn't like being wrong either. After hearing this, it helped me to become more objective.

My problem is that the least wrong thing I know of is very different from all existing processes, and take more time and money than most are willing to stomach.

Time and money are factors into wrongness, too. That's what differentiates software engineering from computer science.

Indeed, but then there is, "using a query language designed for data analysis in event sourced systems to: read, parse, version, and write system and configuration files instead of using VCS like Git" levels of wrong, and sales/marketing professionals in charge of the system wrong.

Time is measured in quarters, money is measured vaguely by proximity to sales.

Thought using excel as a database was bad? Try working in a system where strings are `eval`ed in SQL, and every state change is stored in intermediate or virtual tables, then thrown away, so no actual version control is taking place; just diffing and merging Unix configuration files.

Why not use diff and patch? Takes too much time and money, they only know SQL.

Then it'll be easy and fast and really not embarrassing at all.

Hmm...most of the time when I do a design that's for something at all substantial, I'll usually go through a few ideas that seem reasonable at least through the "napkin stage" before dumping them in favor of what eventually becomes the "real design". I'd just save those napkins, list them in the alternatives section, and explain why they were abandoned. Easy.

That’s the idea. I try set it up so the time I spend on the idea in the doc is proportional to its viability.

I have a reverse problem at the place I work. When I ask people to write a very high level design doc for a fairly trivial task they go - "There many more way to get this done so writing these is useless. And as the task is trivial an engineer should be able to figure out one of these ways and do it."

Many of these people are external consultants who have been working with the company for 15+yrs. So, there is a fair standard in place already because the tasks are done by the same people. But then they go out of their way trying to create a boogeyman of "what happens if people don't follow the standards?".

The end result is that either design docs don't exist or woefully out of date. Hence the company has to keep hiring these same consultants year after year on hugely inflated costs.

> Hence the company has to keep hiring these same consultants year after year on hugely inflated costs.

There it is. There's good scratch to be made in prolonging the problem.

Used to feel this way on other teams. Almost as if you were expected to write them for the sake of writing them (cargo cult engineering).

Now on a team of many OG Googlers (15+ years tenure) and Design Docs only exist when they’re needed (something that touches multiple systems, something that’s obviously complicated with many trade offs, etc) Otherwise it’s just “write the CLS.”

  > Otherwise it’s just “write the CLS.

(apologies for the noob question) but what is cls in this instance?

Change Lists (Pull Requests, Change Sets)

One presumes in those simple CL cases, there is already a doc for the system being edited, and the new code followes the design.

> Now on a team of many OG Googlers (15+ years tenure) and Design Docs only exist when they’re needed

This almost certainly has nothing to do with the team or tenure, but everything to do with what levels they are(n't) trying to get promoted to. Have you controlled for that and still seen a distinction?

This isn't unique to Google. This is encouraged at Uber and Airbnb as well. I would think its a common practice. The idea is to show that you consider alternatives and did due diligence. Following the way its always been done isn't necessarily a good thing. If its a different product area there could be better approaches unique to that area. Its not fake work, it just proves you actually considered other approaches instead of blindly following status quo.

You have a point in that bureaucracy can needlessly get in the way of meaningful change. It frankly always will to some degree. That being said, part of the way you make sure it gets the least amount of in the way is to standardise it, and actually enforce those standards.

From your description you fall into both categories, which means your colleague was correct. Now, your amount of details is very scarce, so it’s extremely likely that I’m getting the situation wrong. But look at it from an enterprise level perspective or change management, if you allow slight deviations here and there for thousands of employees then you’ll very quickly end up with what would be the equivalent of the difference between Italian, Spanish and French. Which all shares the same Latin “standard” and then deviated ever so slightly.

If you think there is a lot of pseudo work involved with managing the bureaucracy of enforcing a design doc standard then you should see just how much pseudo work it saves.

I don't disagree, hence "fake work" in quotes.

Some people may (and clearly do thrive) in such environments but I personally did not.

This particular scenario was a simplified excerpt but very well captures my perceived experience. Having to write a design for for something that was a solved problem (by a team maintaining a framework) felt like pure busy work.

If there was a higher meaning to this process this higher meaning (other than looking good / getting promoted) was not communicated in any effective manner.

Maybe I'm dis-illusioned with the current culture in big-tech but a promotion should be a result of doing good work, not doing work that is designed to look good.

And yes I understand that some of these mechanisms may exist to provide employees a clearer path for ascending the corporate ladder.

Sad to hear your team was like that, mine definitely went the way of "if you know which way you're going to solve a problem anyway, why bother writing a design doc?" and I never felt burdened by them.

Good work needs to look good in order to be seen. And in enterprise, it needs to be able to be compared in order to be fair and afford some level of control from top down.

Standards solve those problems at the cost of some overhead and waste, due to being what they are: a fairly generic reification of best practice (at best).

Even though some overhead is just inherently part of the game, the efficacy of standards (and metrics) does vary. Not all have equal merit. And it does sound like some of the Google standards went off the rails a bit.

Oh I’m not a big fan of it either. Especially because I’ve never really seen a lot of the “fake work” have much value. Usually enterprise architecture or whatever you want to call it ends up being an insane amount of aged documentation that is useless because it’s never updated despite the best intentions. Over the years it’ll typically also erode its own standards as the people leading the efforts change and their standard preferences change with them.

Last but not least, it’s a total waste of time to have actual developers do it. Instead of working on something useful. Of course the challenge then becomes that 90% of the architects and whatever actually don’t know how to write their own design documents from changes, which means they need to waste developer time to do it… meaning that no only are they a completely useless employee they are also wasting the time of useless employees.

This shouldn’t be taken as black and white. As I wrote earlier, it can be done to help work flow. It’s just than in reality these processes usually end up doing far more harm than good. But if you, are, going to do it, at least enforcing standard is the right thing to do.

I personally avoid working at places with too much bureaucracy because I don’t want to waste my time writing documents nobody will ever read after they are approved.

What you said would make sense if they weren’t forcing the author to go find more alternative approaches.

> You have a point in that bureaucracy can needlessly get in the way of meaningful change.

What a good early design doc can do is turn meaningless changes (e.g., doing a project for the sake of promo) into meaningful non- or minimal change. The best code is the one not written.

And at least in my area (Core), the design docs only go through anything like a formal process when it's a larger project that would take years or multiple engineers. Other areas are probably different.

Then include templates from other areas pointing out the repeated work. You can’t just hand off a couple of lines in a brief to a dev, especially if they are not aware it’s been done before.

The mistake was production of a design doc instead of just writing the code. If it's trivial enough that there's nothing to discuss, you generally just change the code. If it's complicated enough, it becomes a negotiation process where at least 5 different people have to be able to put it into their perf if you want to get it done.

the compensation structure in big tech makes people lose their minds optimizing for that next promotion/stock grant. It helps big tech retain talented people but the incentives to look "impactful" drive all the wrong behaviors

The old school way of fixing that problem is to have a standing change order for routine work so you can point to it and say "my problem is just like that" and get the green tick right away. Though the moment you suggests things like change control people get all weird these days.

Right.

> The decision whether to write a design doc comes down to the core trade-off of deciding whether the benefits in organizational consensus around design, documentation, senior review, etc. outweigh the extra work of creating the doc

I'd argue that extra work of creating the doc is an upside because it clarifies your thought process, and design-by-committee is a massive downside.

In fact, I happen to be writing this during a design-doc meeting. I am getting absolutely nothing from it. I was writing code all morning and the background noise now prevents me from continuing productive work.

The problem with stuff like this is that it saps the (finite) energy of your team. Not everyone is a tank. High int lose their hit points and they're gone with all their mana.

I'm dealing with this right now at another large company. Being asked to write a document for an integration we've done 20+ times because its part of someone else's larger promo project's design.

Couldn’t you just reference a recent doc and say this is mostly the same with these x differences leading to y alterations for z reasons?

I interviewed a couple of folks from Google recently and it was mindblowing how they "work", it's very unfortunate. I feel for some of the people who only worked there their whole lives (e.g: from recent grad within the last 5y), it is unlikely that they actually know how to program.

Another word for Design Doc = "Shelfware"

Reply on news.ycombinator.com