Comment by pclmulqdq | Hacker News Reader

Comment by pclmulqdq | Hacker News Reader

Parent Full thread

I worked on one of those infrastructure services (like Bigtable). It was a huge boon to my productivity that the one version rule existed.

If we had to maintain 6 past months of releases, we would never get anything done. Since breakages in our client library were our problem, getting a months-old bug means that we can just tell you to recompile rather than figuring out how to backport a fix for you.

Hyrum's Law considerations got really weird, especially when people take them too far. I think this is based on the kernel idea of "don't break userspace," but its practical implications are nuts. Hyrum's law has killed infrastructure projects that could have made things a lot better, and has resulted in crazy test-only behavior being the norm.

One person on an adjacent team loved taking behaviors as promises (and he also had a reputation as one of the most prolific coders at G). We had to clean up his messes every time he relied on unspecified behavior. I pushed back on his nonsense a few times, particularly when he used an explicitly banned combination of configuration options that we forgot to check-fail on, but always lost. 1.5 SWEs on our team were full-time cleaning up after him.

You are a rare engineer who actually pushes back on nonsense. For that, I say "thank you". I feel that there are too few people at Google who will say "stop, that is going to create a problem in the future". Some people are very enthusiastic and well meaning and "productive" (high change count), but create a burden for their colleagues in their fervor to change things.

To the point in your last sentence, a few years ago I worked at a software company where there was a prolific code writer who similarly tied up about ~2 SWEs cleaning up his messes. He was a Perl programmer who was set loose on C# code. The broken ways the company tracked productivity meant the prolific writer rated as doing great while the three SWEs who had to spend 2/3rds of their time fixing the messes he left behind were rated as unproductive.

How did the company track productivity? Why did the one get rated as doing great?

Features checked off as "working" when they pass a test/demo instead of passing code review?

Probably lines of new code?

Agree. I've maintained some widely used libraries at Google for over a decade (the larger one getting invoked from over 5e9 qps at peak) and I'm very grateful for the build-everything-from-head practice (and short build horizons). Yeah, it introduces a few small problems, but I think they are ~easy to deal with —e.g., protect new functionality with flags to control canary/roll out when needed; file bugs (and, eventually, deliberately break) customers with unreasonable tests.

Just to pick on the last example, having customers with unreasonable tests _is_ a real problem. But letting clients pin down specific/older builds of their dependencies (your code) to deal with this doesn't solve the problem, just pushes it down the road, imo making things worse.

.. and these problems are absolutely worth having in return for the simplicity of ~only having to support head (and, in some cases, like client LB policies, just a relatively short build horizon).

Any ideas on mitigating, kneecapping "prolific coders"?

My observation has been the torrent-of-poo coder also demands quick (pro forma ~~performa~~) code reviews while nitpicking concern trolling other people's PRs. Not sure what to call this? Gatekeeping, control freak, PKI hacking, passive-aggressive, or maybe just being an asshole.

It's a culture thing IMO. I used to be an electrical engineer, which has a strong culture of "everything I don't specify is a black box that I am free to change" (since the core products of EEs literally are black boxes with some wires sticking out the bottom). I was shocked that programmers don't think the same way. Also, the super-coders having bad attitudes on PRs matched my experience, and is incredibly toxic.

I think part of it is that programmers don't like to write a lot of documentation, but infrastructure services have very long lives and naturally build up a lot of documentation. If you write down a lot of promises, you also have documentation about what you don't promise. Unfortunately, the culture of G was that docs don't get updated and you promise all observable behavior.

This is always the result of someone looking at productivity as zero sum, some management practices encourage zero sum behavior. If management doesn't actively incentivize non-zero sum behavior - then zero sum behavior rules. Consider the following list of highly effective zero sum behaviors.

Effective Zero Sum behaviors that management should avoid:

0) I can ship faster than all of my team mates if I nitpick their CRs and get mine quickly approved by our easier reviews.

1) I can make sure my CR reviews are always faster if I am the nitpicky a-hole on every other CR

2) I can ship faster if I don't add as many tests as my peers. I'll promise to add them in a later CR and forget about it.

3) I can ship faster if I write software that is difficult for others to work on.

4) I can seem like a smarter engineer than everyone else by using jargon and writing difficult to understand code.

5) I can convince management to look at my X,Y,Z productivity metrics, while the team remains ignorant. (Code Review iterations is a dangerous example of this)

Big thing is to recognize signs of zero sum behavior, and investigate if it's being done intentionally or unintentionally. Even if its intentional, it may simply be a bad reaction to the surrounding incentive model of the organization.

Agreed.

I think about Eli Goldratt's The Goal all the time.

Paraphrasing: A team (system) is only as fast as its slowest member (task); to speed up the team, focus on making the bottle neck faster.

Said another way, Goldratt also explains how a process that outpaces the others works to slow down the system overall. Local optimization leads to gloal inoptimization.

Pretty basic queue theory stuff. This aspect is aka Theory of Constraints.

Anyhoo. I don't buy most "10x programmer" tales. I just wonder who's on the receiving end of such awesomeness. And at what cost.

Holy shit I just got flashbacks from the dude on my team who used to do that asymmetric code review warfare. Hardly saw his PRs cause he had one person on the team insta-stamp them. At the same time he’d happy-glad you to death with blocking change requests. Said requests would trickle in over the course of a day or two because he didn’t just sit down and do a review, he’d graze on it.

> At the same time he’d happy-glad you to death

Can you explain what “happy-glad” means here please?

I think you meant "pro forma" not "performa".

Reply on news.ycombinator.com