The Friendship That Made Google Huge | Hacker News Reader

The Friendship That Made Google Huge

newyorker.com

532 pointsadrianhon101 comments

What a lovely profile. I worked at Google early enough that I got to know Jeff and Sanjay, although never worked directly with them (to my regret). Every aspect of this article rings true to me, the way they are, the way they work together. I always admired how incredibly collaborative they were. They never competed with each other. Or really with anyone else, either, they just quietly did excellent work and helped others do their work too.

I'm so glad this this article (unlike so many other representations of Jeff and Sanjay's work) does not overlook Sanjay's contributions.

I don't know what it is; maybe Sanjay is a weird (read: non-English) name, maybe it's that computing culture's obsession with lone hackers leaves no room for a partnership like Jeff and Sanjay's. Anyway, kudos to the New Yorker for not falling into that trap.

I think a big part of it is something the article touches on: Jeff is an extrovert, Sanjay is an introvert. Jeff is the much more visible of the two; the one who people are more likely to have heard speak personally or to have dealt with. It sounds like this is how they both prefer it, but it definitely means that Sanjay's contributions get passed over in the common discourse. I was also glad to see the article avoid that trap.

You mean like Steve and Woz? j/k :)

Ditto on the sentiment. I’m not a Googler but I’ve worked closely with a few, and Jeff was revered accordingly (along with the Chuck Norris-esque jokes). I’d honestly never heard Sanjay mentioned (and didn’t know who he was) until I read this article. I’m glad he was given his due and I’m glad to have learned about his contributions.

> (along with the Chuck Norris-esque jokes)

Funny thing, that was started by kentonv [0], who worked on open-sourcing protobufs [1], then CapnProto [2], then (with Jade Wang) Sandstorm [3]. He lurks around here :)

[0] https://plus.google.com/+KentonVarda/posts/TSDhe5CvaFe

[1] https://github.com/protocolbuffers/protobuf

[2] https://capnproto.org

[3] https://sandstorm.io

Hello...

Indeed, I wrote the internal web site (built on a pre-release prototype of AppEngine!) that aggregated these jokes, and released it on April 1, 2007. I was not funny enough to come up with the jokes myself; I enlisted some friends to seed the site before "launch".

In retrospect, I regret that Jeff Dean Facts gave people the impression that Jeff is more notable than Sanjay. The only reason it was "Jeff Dean Facts" and not "Sanjay Ghemawat Facts" is because Jeff seemed "funnier". But I dunno, that might just be some form of subtle racism that I failed to recognize at the time. :(

PS. Your list doesn't include my current project, Cloudflare Workers... :) https://blog.cloudflare.com/cloud-computing-without-containe...

So much!

As a college student, it was so weird learning about Jeff Dean's meme status, and it wasn't until after joining Google and coming across things like the original protobuf design doc that I realized how significant Sanjay's contributions are.

I was so delighted by this piece and then I got to the byline - James Somers strikes again!

You might know his writing from “You’re Probably Using the Wrong Dictionary” or his amazing article about New York’s subway system told through the lens of the countdown clock: http://www.theatlantic.com/technology/archive/2015/11/why-do...

Highly recommend all his work: http://jsomers.net/

seems to be a awesome engineer too.

> Most of the fragile insights that laid the foundation of a new vision emerged not when the whole group was together, and not when members worked alone, but when they collaborated and responded to one another in pairs

The article gives several examples of this dynamic, but one that came to mind for me was how Monty Python members created sketches for Flying Circus:

> Writing started at 9 am and finished at 5 pm. Typically, Cleese and Chapman worked as one pair isolated from the others, as did Jones and Palin, while Idle wrote alone. After a few days, they would join together with Gilliam, critique their scripts, and exchange ideas. Their approach to writing was democratic. If the majority found an idea humorous, it was included in the show.

https://en.m.wikipedia.org/wiki/Monty_Python

Interesting tidbits:

To solve problems at scale, paradoxically, you have to know the smallest details -Alan Eustace

Today, Google’s engineers exist in a Great Chain of Being that begins at Level 1. At the bottom are the I.T. support staff. Level 2s are fresh out of college; Level 3s often have master’s degrees. Getting to Level 4 takes several years, or a Ph.D. Most progression stops at Level 5. Level 6 engineers—the top ten per cent—are so capable that they could be said to be the reason a project succeeds; Level 7s are Level 6s with a long track record. Principal Engineers, the Level 8s, are associated with a major product or piece of infrastructure. Distinguished Engineers, the Level 9s, are spoken of with reverence. To become a Google Fellow, a Level 10, is to win an honor that will follow you for life. Google Fellows are usually the world’s leading experts in their fields. Jeff and Sanjay are Google Senior Fellows—the company’s first and only Level 11s.

> Level 2s are fresh out of college; Level 3s often have master’s degrees. Getting to Level 4 takes several years, or a Ph.D.

I..don't think this is still accurate. New grad SWEs are L3's.

Yeah it's mostly accurate but drop level 1. Lowest I've seen for technical employee of any kind is 2. New grad is 3. 4 is about 2 years later. 5 is the point where you have "no expectation if advancement" and is the most common level for mid-career people.

If you want to see an example of Sanjay's elegant code, take a look at his Go stream package: https://github.com/ghemawat/stream

Sanjay's "experimental" directory in Google's repo -- a place where users can commit unreviewed code that are just ideas, or park some code that isn't ripe yet -- is full of gems.

Good stuff. Every sport has great partnerships (Jordon - Pippen, Brady - Gronk) and I always wondered if such a thing would even be possible in SE as I've felt overly self conscious and inefficient while pair programming.

Though what really makes this my favorite long read of the year is how it follows their story over the course of Google's rise and the subtle juxtapositions between early Google of "doors across sawhorses" and today with "Level 0-11". Given all the recent news with engineer walkouts and protests it makes you realize that Google have probably done the best job of handling the scaling challenges of being a unicorn and making the necessary cultural adjustments as they grew in size.

Don't forget Sam and Frodo

Yeah, teams are good but every once in a while you need that lone crazy Tom Bombadil genius to come along and save your butt from disaster.

And sometimes you need eagles

The really bad times are when your entire universe screaches to a halt until Brandon Sanderson helps it limp across the finish line.

http://cycle-gap.blogspot.com/2007/09/extreme-pair-programmi...

Extreme Pair Programming - Guy Steele and Richard Stallman

> long read of the year

Are books extra-long?

After reading this article, I'm surprised that Google doesn't encourage pair programming / developing developer relationships at the level that Jeff and Sanjay have here. One might even argue that the manner in which interviewing and hiring is currently done in the tech industry in general might actively work against building the sorts of relationships where the output is greater than the sum of the inputs.

Edit: added a missing "the".

Their specific choices seem hard to replicate. Very strong friends, both very strong programmers, both willing to work at the same company. I'm sure that parts of their successes can be applied elsewhere, but from the article a lot of it is about having a strong relationship.

See also Andrew and Brad: https://m.youtube.com/watch?v=yG-UaBJXZ80

In a way, Google does encourage it by requiring all code to be reviewed before check in. If you're pair programming, you already have a reviewer and can check in right away. One-person projects can be more frustrating due to the delay.

It doesn't seem to be enough to encourage widespread pair programming, though.

Every company I've worked at as a software engineer, even my first where we emailed patches to each other like cavepeople (and no we were not using git) has had code reviews before check in. It's not at all the same thing as pair programming though.

Holy moly, a mention of Bogdan! He was a legend of his own. Some of the most consequential changes, like elevating your service to higher QoS levels, which something like e.g. Gmail would require, were gated on his approval and his only. Someone made him a SW Easter egg, but the story already on the Internet gets some basic facts wrong.

Do tell! There have got to be a few good ones. He interviewed me for his current company.

The internal Easter egg was basically a random fortune, which cycled through his greatest "hits" (#1: "No"). You really had to know where to look.

Does anyone have a reference for "In 1966, researchers at the System Development Corporation discovered that the best programmers were more than ten times as effective as the worst." ?

I'm seeing the year 1968 at http://thecomputerboys.com/?tag=black-art#fn-156-1

Is there a 1966 reference out there somewhere? Or perhaps the actual study was done in '66 and didn't make it into press until '68? I didn't pull the original paper.

(The fact checkers at the New Yorker are typically very sharp.)

edit: I pulled the original paper (10.1145/362851.362858). The studies were done in 1966!

Interesting to see the genesis of the myth of the 10x programmer.

It is awesome that you were able to find this. This very well may be the genesis!

For anyone else, that DOI points to: https://dl.acm.org/citation.cfm?doid=362851.362858 "Exploratory experimental studies comparing online and offline programming performance" from CACM 1968.

I'm not so sure it's a "Myth" when Jeff and Sanjay are out and about. Honestly, I'm beginning to think it's inherent and can't be taught.

"Interesting to see the genesis of the myth of the 10x programmer."

How could it be a myth? Ever see a professional sports player compete against a non professional player? Why wouldnt this dynamic exist in programming?

Maybe I'm unimaginative, but in sports where performance is objectively comparable, I can't think of any where professionals 10x amateurs.

Amateur marathons are about 4 hours, professional are about 2 hours. Amateur deadlifts are about 500lb, professional are about 1000lb. Amateur long jump might be 6m, professional almost 9m.

Cristiano Ronaldo and Messi are extreme outliers in soccer/football, even compared to other greats. They're probably 10x. Trouble is amateurs don't play in the big leagues, so we don't have data on how a normal person would peform. There is an amusing Ronaldo vs Ronald video. I think it's called "Tested to the Limit."

Occasionally a 1st league team plays a 3rd league. I saw a video of Napoli vs Nowhere go 26-0. They scored about as fast as the game could be reset to kick-off. That's more than 10x.

You are confusing game scoring with some concept of "x times ability". In many games, someone who is only 10% better than opponents can win close to 100% of the games.

What's your productivity metric? Perhaps having only 10% fewer bugs per hour makes someone 10x faster at writing complete programs.

I emailed the author and he pointed me towards this copy of it: https://apps.dtic.mil/dtic/tr/fulltext/u2/645438.pdf

I haven't read through it yet, but it appears to be a more extensive whitepaper version than the CACM version.

This is a wonderfully told story of two very senior engineers who developed a lifelong friendship and wrote some of the core of the Google search engine I use countless times per day - I really enjoyed this piece.

Wow, this article is pretty amazing to me. I have a joke (err, kind of a joke) that I like to say, "It's never a cosmic ray." What I mean by that is that when you are debugging and getting incredibly frustrated, right when you feel that you want to give up because you can't find the bug, that you shouldn't blame "cosmic rays" because 99.999% of the time it's a logic problem.

Here was a case where it actually was cosmic rays (at least, not a logic problem), and these guys found it. I also agree with the title and intro of the article - I think it's very possible that if these 2 exact people hadn't been at Google at the time than Google's path could have been very different. Makes me also appreciate how much luck is involved in any successful indeavor. Not luck that they found the bug (that was hard work), but luck that these 2 guys found themselves working at Google right when Google needed them.

It’s never a cosmic ray but it’s often bad ram. You don’t have to inspect the binary to diagnose bad ram. The system would be behaving erratically in other ways due to regular memory corruption. It’s an amazing story but by no means the only way it could have played out.

“In terms of sexiness, compilers are pretty much as boring as it gets,”

That bit surprised me. For me it doesn't get much sexier than compilers.

Yeah. Maybe that's the outside view, or view from the surface level folks concerned with the next shiny framework, but I think the best engineers are always at least a little enamored with deeper nuts and bolts.

I think he meant the business side isn't going to give a damn about a compiler.

That makes sense. At least, outside of massive companies like Google, who would see the value in a compiler that optimized code for their use even a fraction of a percent better. Then compilers become the "shiny object" for them.

Writing a modern concurrent garbage collector is as complicated as writing a compiler, to a great degree because of their "real time" constraints and concurrency.

But to call them sexy is another thing ...

“I don’t know why more people don’t do it,” Sanjay said, of programming with a partner.

=======

Maybe because most employers won't pay for that. Two people? For one level of output? You'd have to prove yourself a level 11 googler before most places would give you the luxury of working as a long-term pair.

I don't think that's it. It's typical to have a team of programmers working on a large project, and many managers are willing to try things.

However, pair programming does require a lot of commitment to make it work. I've only seen it work on a team where it was an XP project to start with, so everyone hired was already willing to give it a solid try. Your average team isn't self-selected in this way and won't have the same level of enthusiasm for a new management initiative.

I think from the very beginning it was understood they were producing more than one level of output together.

Pivotal Labs has had across-the-board pair programming for years[1]. Although the folks there are undoubtedly talented, I'm fairly certain most aren't at Jeff and Sanjay's level.

I don't know why companies don't do it, but it's not cost alone. Here are some of my hypotheses:

1. It seems like it would be more expensive - which might deter companies from thinking about it

2. If you're not already doing it, it's hard to convince all or even a large fraction of your engineers to do it. It's a completely alien way of working to most people

3. It's not obvious to employees who haven't done it that it's clearly better. And it might not be - I honestly don't know

4. For 100% pairing, you need 2 programmers' schedules to line up exactly, 5 days/week - so goodbye flextime, work from home, remote working and all the other arrangements that people structure their lives around

5. It must be exhausting to sit with, and talk to, another person constantly your entire working day

1. https://pivotal.io/careers

I’ve paired a bit before, and #5 is definitely true. I can only handle 2-3 hours of it.

Yeah, based on what I've seen/ heard, pairing should last no more than half the day. It's just too demanding to maintain all day long, like a cognitive sprint.

There've been studies on the "more expensive" part, and IIRC it's about a 15% premium in terms of development time - there've been a few studies. Here's one I could turn up quickly: https://collaboration.csc.ncsu.edu/laurie/Papers/XPSardinia....

Simultaneously, you get a bunch of benefits. (Most importantly, there's no bus factor of 1 for anything)

Absolutely. Just the bus factor thing was amazing on its own. It's the very rare startup where anybody can take a vacation whenever they need. But that was definitely my experience. Even though I was the first programmer, in pretty short order there was nothing where my absence a blocker. It was great.

i'd think another benefit might be 'better code' overall. has there been any study about the quality of code from these setups? number of bugs or security issues, for example?

Also covered in that paper

Regarding #5, you get used to it. It's intense at first, but after a while, it's just normal. One secret is to take actual breaks. And another is to not over-work. When pairing, there's none of that stuff that might stretch out your day. E.g., checking Hacker News. It's solid work time, and after 8 hours of that I'm definitely done for the day.

Thats definitely not the only reason. I've been on teams where managers were keen on pair programming but only a couple people ever did it.

> The world’s most robust computer systems... used special hardware that could tolerate single bit-flips. But Google... bought cheaper computers that lacked that feature.

> Together, Jeff and Sanjay wrote code to compensate for the offending machines.

I'm guessing the article is referring to Google buying servers with non-ECC RAM.

How do you write code to compensate for non-ECC RAM?

By pushing this complexity up the stack and also having redundancy in storage. What's not correct can be an error and result in retries from the client that may be routed to good storage.

But how do you even tell something's not correct? For example in the article they had to manually examine the index and find out some words were missing. How do you detect that not on the hardware level but up on the software level?

Checksums

Just had a fun flashback to checksuming new builds of Windows media after manually burning/ before installing the build!

https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_cor...

> When a car goes around a turn, more ground must be covered by the outside wheels; likewise, the outer edge of a spinning hard disk moves faster than the inner one. Google had moved the most frequently accessed data to the outside, so that bits could flow faster under the read-head, but had left the inner half empty; Jeff and Sanjay used the space to store preprocessed data for common search queries

So Google was a company that was too cheap to buy ECC RAM, but willing to let half of their hard disks go to waste. I'm struggling to understand the logic behind this; can anyone explain?

RAM is expensive. Disk is cheap.

these are the kind of articles i like reading, well written, interesting subject and good content length. Too many times i read bold headlines on HN, "Why X is bringing america down", then go to read it and just when they get to the crux of the matter i find the article has ended smh.

The New Yorker does this very consistently every week at an extraordinary level of quality.

They will even mail you a copy in exchange for a subscription. It's magical.

i will look into it

Only $6 for 12 weeks for print + digital. Subscribed!

Site's down for me. Archive link: http://archive.is/w4oWL

> On Sanjay’s monitor, a thick column of 1s and 0s appeared, each row representing an indexed word. Sanjay pointed: a digit that should have been a 0 was a 1.

I was curious how he can see "a digit that should have been a 0 was a 1" in a "thick column of 1s and 0s"? Can you give a specific example. I'm not questioning that he did but I'd like to know a specific example.

For example the high bit in blocks of ASCII text will always be zero. In a binary dump it would appear as a column of zeros.

Ok. So you don't need to actually "read" and understand the binary code like you are reading a programming language. You look for out of place numbers.

I wonder if more couldn't be done to encourage pair programming and collaboration in general? What are some other great partnerships?

Would highly recommend checking out `The Undoing Project` by Michael Lewis. It's a biography (of sorts) about Amos Tversky and Daniel Kahneman that has excellent stories about their fruitful collaboration in behavioral economics.

Was surprised to hear the audio version of the story has a different title - "Binary stars". I think the editorial board went with a more click friendly title.

That was also the title of the print version. The New Yorker tends to re-title online "reprints" with less vague/terse/artsy titles.

Anyone know where folks like Peter Norvig and Urs Hölzle fit on the L1-L11 scale? I assumed that all of the early employees/now legends were at the top level, so I was surprised to see the article state otherwise.

Urs doesn't need a level on the individual contributor scale; he's a Senior Vice President.

Urs however does have one, he's listed as a Google fellow (L10) if you go looking.

As a general comment, to compare, a director is L8 equivalent, and VP L9, (but on the management ladder) although at VP+ it gets tricky because there are subdivisions.

Also worth noting that the management ladder goes up more. But the total number of people who are >L11 is probably <11, so it's not like there's any data on how exactly that's broken up.

Director is L8, Sr Director is L9, VP is L10, SVP is L11.

Not exactly; there are 3 levels of VPs and 3 for SVPs (at least in 2015). It’s just not publicized.

Peter is a director, and Urs is a senior VP. As such, they are now both on the management ladder. I think the article was discussing only people who were on the eng ladder.

I can't remember what Senior VP equates to on the tech ladder, but in terms of his overall impact on the company, Urs is definitely an 11.

It is refreshing to see an article celebrating collaboration.

This article will queue up a slew of startups that pair program.. perhaps a good thing!

There was an earlier thread in HN on them here - https://news.ycombinator.com/item?id=16744353

It is such a beautiful story of friendship. Even married couple don’t this level of match in mind. Only a few a luckily like that. Their partnership has resulted monumental contributions.

Great article! Two powerful minds working on analyzing and solving on same thread of thought. Kudos to the writer, very well presented.

Serious question - is this an ad? For the company, showing they have a solution to new growth problems which will keep them a growth company? (custom chip, architected for AI)

And how they solved the problem before with this team?

Does pair programming only work with friends?

Nah. You can do it with anybody. I've been part of long-lived teams that pair. But there we rotated pairing partners once or twice a day, so that over the course of the week you'd pair with everybody.

I think that's much more sustainable for most. Pairing with the same person every day is sort of like sharing a snowbound cabin with them. There are very few people I get along with so well that they wouldn't grate after a few weeks.

Enemies are the best to pair with.

Awesome article

"Jeff Dean once shifted a bit so hard it ended up on another computer."

"Jeff Dean puts his pants on one leg at a time, but if he had more than two legs, you'd see that his approach is actually O(log n)."

Sounds like he discovered the Rowhammer exploit?

Great article. Glad to see Sanjay get the credit he deserves.

Reply on news.ycombinator.com