Hacker News Reader

Comment by wallflower | Hacker News Reader

If I could give a six word response for the technical side: "Jeff Dean, Sanjay Ghemawat and team".

To paraphrase some Cal/UC Berkeley professor that I cannot remember, he said that the problem with scaling is that when you go up a magnitude or more, you may need a qualitatively different solution for the same problem. Jeff Dean and Sanjay Ghemawat and their team have time and time again managed to design and build state-of-the-art, qualitatively different solutions (MapReduce, BigTable, Spanner). And now their team's attention is on AI...

2016 (might be more recent ones) update on AI at Google http://highscalability.com/blog/2016/3/16/jeff-dean-on-large...

Video of full lecture: https://www.youtube.com/watch?v=QSaZGT4-6EY

EDIT: Sanjay Ghemawat and Jeff are like the dynamic duo of large scale systems engineering.

View on HN

> the problem with scaling is that when you go up a magnitude or more, you may need a qualitatively different solution for the same problem.

Never been to Berkeley but I've heard something similar enough times that I've adopted it as one of my own aphorisms.

When the order of magnitude changes, so do the good solutions.

(This is also my thesis for why we keep repeating software history. We stop using a solution because it doesn't work with the current ratio of speed or costs for network, memory, storage and processing. Then someone introduces 10GB networking or SSD disks and the old solution works for 5 more years until someone introduces a new CPU that scales better horizontally or vertically and we start over.)

I think it's important to mention Sanjay Ghemawat is very likely just as important to google as Jeff Dean, just Jeff has been made the public figure for a variety of reasons.

Thank you. I increased the answer to six words because you are very right.

Urs Hölzle (engineering VP) and Eric Veach (AdWords) also don't get nearly as much recognition as they deserve.

Huh, I had no idea the guy who wrote the light transport thesis also built Adwords.

He's also the chief contributor to the S2 Geometry Library.

Thanks for this, anyone know what he's up to now?

I can't believe he left CG behind :(

I disagree with passion. The vision of Larry Page and Sergey Brin is responsible for Google's success. Jeff Dean, Sanjay Ghemawat executed Page and Brin's ideas, but they were not the key decision makers. Did Dean and Ghemawat decide to buy and then build Android? Did they decide to build Google Maps? Did they decide to buy YouTube?. The answer is no. Page and Brin would have hired other smart people if Dean and Ghemawat weren't available or had they left Google.

I agree, as I was not talking about the overall success of Google, just the success of Google's secret sauce (their backend infrastructure) that enabled it to sell ads and deliver search results with a high level of quality. If you read some of the books about early Google, they (Jeff, Sanjay et al) were really, really struggling to do stuff like index the web without the crawler collapsing mid-way. That's how MapReduce became to be developed.

Yes, Danger/HipTop, YouTube, and Keyhole are arguably the three smartest acquisitions that Google made.

Also, the Google SRE book makes the point that they couldn't have achieved their scale w/out approaching ops from an engineering perspective. Linearly hiring ops people as services scale would have been too expensive.

Approaching operations from a software development perspective wasn’t unique to Google.

Also ops is engineering and has been since before Google existed.

Ops is probably more reasonably called engineering than most so-called “software engineering”.

The book : https://landing.google.com/sre/book.html

Someday Google's going to have to list the risk of Jeff Dean leaving on their 10-K ... (not entirely joking)

Companies actually do this behind the scenes. They list individual employees as financial assets.

Hmm, I hadn't heard of this practice before, and so I went searching and found this article to be a decent summary (posting here if others are curious): https://smallbusiness.chron.com/employees-intangible-assets-...

They certainly must have him insured...

https://en.wikipedia.org/wiki/Corporate-owned_life_insurance

No, there's no point for Google to pay a third party to insure any risk of $1B or less.

Are there any risks at all worth insuring for Google? What’s the point of insurance if you have $100B in the bank?

Even some small companies (100+ employee) carefully track attribution of their top N% of non-VP employees and tie it into executive bonus.

That doesn't provide a perfect answer. Sanjay and Jeff (and a lot of other senior Googlers) were all at DEC. Why didn't DEC succeed to the degree that Google has?

Because they didn’t have Craig Silverstein and many many others. Lets not pretend Jeff and Sanjay, as much talented and prolific as they are, are singlehandedly responsible for Googles success.

I don't know if Sanjay and Jeff worked directly on DEC's AltaVista search engine but that project was treated as a low-priority "demo showcase" for DEC Alpha servers instead of being seen by DEC management as a new multibillion dollar internet business to pursue. DEC could have had a "Bezos-hey-we-can-sell-computing-as-AWS" moment with AltaVista but they missed it and let Google surpass it.

They didn't - AltaVista was done by Louis Monier and Mike Burrows, both of whom also worked at Google and both of whom also made significant contributions to the codebase.

I do agree that management was the critical ingredient - my former VP at Google (employee #48) credited Urs Hoelzle (employee #9, and Google's first VP) as a key ingredient to Google's success. Ironically, though, Google hired Western Digital's management as well - SVP Alan Eustace came from there - and yet the team under Eustace at Google was much more effective than the team under Eustace at Western Digital.

I suspect that management has sort of a "gatekeeping" effect - if all of your management is good, the whole department does great things, but if a single link in the org chart is a dufus, the whole organization will fail. When I joined Google in 2009, my whole management chain (6 levels, up to Eric Schmidt) were all engineers, and it showed.

> I suspect that management has sort of a "gatekeeping" effect - if all of your management is good, the whole department does great things, but if a single link in the org chart is a dufus, the whole organization will fail. When I joined Google in 2009, my whole management chain (6 levels, up to Eric Schmidt) were all engineers, and it showed.

So true. As an engineer, the most direct impact on my productivity has been the team and manager. But the manager is a SPOF: if he/she is bad, then my productivity suffers.

I have yet to work in an organization where there was a good chain as you mention though. Maybe that's what sets Google apart. Smaller companies tend to have shorter management chains and I prefer them for that reason.

Mike Burrows was a coauthor on the algorithm behind bz2 [1]. It's the most mindblowing algorithm I ever encountered and so I have utmost respect for that couple of guys :)

[1] https://en.wikipedia.org/wiki/Burrows%E2%80%93Wheeler_transf...

thanks for sharing, it seriously is amazing

They also had Dave Cutler and loads of other high profile people over the years.

Dozens have companies have benefited from the diaspora of DEC, and now Sun.

And Softdisk had John Carmack and basically the entire original Id software team on their payroll. And Fairchild Semiconductors had the entire executive founding team of Intel, and Xeroc PARC had not only several key people but essentially the entire IP portfolio of modern computers, including some working prototypes...

"Success" is the result of a very large equation with many variables, and having (some of) the right people is only a few of them.

> To paraphrase some Cal/UC Berkeley professor that I cannot remember, he said that the problem with scaling is that when you go up a magnitude or more, you may need a qualitatively different solution for the same problem.

You might actually be thinking of Jeff's Stanford talk from a few years ago (2011). Slide 21 of:

https://static.googleusercontent.com/media/research.google.c...

  Ensure your design works if scale changes by 10X or 20X
    but the right solution for X often not optimal for 100X

There's no attribution, but that is good advice I've kept with me throughout my career. Props to the original author (perhaps apocryphal at this point?).

"And now their team's attention is on AI..." is quite the hype thing to say. Just because you can build a bigger bridge doesn't mean you're a good car designer.

They have proven they are a good 'car designer' too over the last 3 years.

> "And now their team's attention is on AI..." is quite the hype thing to say.

Googles AI blows every other companies out of the water.

> Just because you can build a bigger bridge doesn't mean you're a good car designer.

What? How is that analogous?

> Googles AI blows every other companies out of the water.

That is certainly true, but that could also mean that Google will be the first company to hit certain unsurpassable road-blocks. Just look at Google Translate, which IMO hasn’t qualitatively improved for the better for at least 2 years now, even though most probably the data on which it is based is at least an order of magnitude “bigger”. The same might also happen to autonomous driving.

Dean's PhD was in neural networks.

Also TPU2 is the only decent alternative to NVidia hardware in production, so that's a decent sign.

I think that Google's dominance has almost nothing to do with technology and almost everything to do with customer lock-in.

For me, duckduckgo.com is as good as Google but I still use Google because it's the default search engine for Chrome's address bar - And that's where I do all my searches from these days. If Duckduckgo made their own browser which was of the same quality as Chrome then I would definitely consider using it and switching to Duckduckgo for search as well.

Out of curiosity, what prevents you from using Firefox? It allows you to specify a default search engine -- which you can set directly to DDG.

> Out of curiosity, what prevents you from using Firefox? It allows you to specify a default search engine -- which you can set directly to DDG.

Chrome allows you as well.

I use FF and DuckDuckGo. It's super easy.

I also heavily use the cli ddg search. Might try that out for anyone else who lives in the terminal.

I used to wonder why search engine companies care about which engine is the default in a browser because it only takes a few seconds to change it but now I know why. People are lazy :/

There's more to it than laziness:

https://en.wikipedia.org/wiki/Status_quo_bias

https://en.wikipedia.org/wiki/Default_effect

It gets worse. I remember I had a friend that had a piece of spyware that set his default search engine to something else. At first glance I didn't realize it was a piece of spyware because all his searches looked like normal searches, albeit on a generic looking page.

I wondered why he didn't switch to Google or something else at the time and it used to annoy me whenever I was using his desktop for anything. Then I took a look into it and found out that he actually couldn't set his default search engine through his browser anymore, in Firefox or Chrome actually.

It was a browser extension. For that entire time there was a piece of spyware on his computer probably watching all of his browsing that he probably could have figured out about himself, if he wasn't too lazy to just switch his search engine. Once I looked into it, it took all of five minutes (if that) to figure out what was happening and remove the problem, and he's not exactly tech illiterate. He literally just didn't care or want to take the time to switch his search engine.

That's a data point. Just one, anecdotal data point, but take it how you will.

Could also just use Tor Browser - it's Firefox modified for privacy, all running through Tor, with search defaulted to DDG. Actually works pretty well as a drop-in system, and you can't beat the privacy.

Ironically, my biggest complaint is the search - a number of queries that I'd expect to "just work" on Google, DDG really struggles with. There actually is a noticeable difference in quality between them.

You know you can change where the browser bar searches; In chrome, in Safari, and in Firefox, and Edge.

There is a huge benefit to being the default, if that wasn't the case google wouldn't pay all the companies they do billions of dollars a year to be the default.

That wasn't true until recently on Chrome/Android, and seems to be a direct response to EU actions.

For me it's because with Google I don't need to go to page 5 to find interesting results.

I've tried to use DuckDuckGo several times, each time I've quickly found myself trawling page after page of results, reverted to Google and found the stuff I was after on page one.

I have exactly the opposite experience, time after time after time.

The one area where Google search provides me with a notably better experience is in image search.

100% same I wanted to love ddg but it's just not as good

If only they implemented "!g" as a simple button, I'd use DDG more often. Typing "!g" requires 6 touchscreen actions on mobile :/

DDG doesn't have stock prices, movie listings, or any of the useful "onebox" type information that Google has. It's just not as useful IMHO.

That used to be true for me but I've used ddg as my main search engine for a couple of years and I find myself using "!g" less and less. Actually you made me realize that I haven't used it in a while, probably weeks (and I've had a very busy couple of weeks with tons of opportunities to look for stuff online).

It's not because duckduckgo finds everything I want, it's because I've noticed that when ddg doesn't find something usually google doesn't either. Also whenever I end up on google now (generally because I forgot to change the search engine after a fresh install) I find the website cluttered which is a shame because it used to be in Google's DNA to keep the search page simple.

For instance I just made a completely random search on Google just to see what it would look like: https://svkt.org/~simias/up/20180727-235325_tou.png

What the hell is all of that stuff? If I wanted to browse the stock exchange or google maps I would. "People also search for" well good for them I guess.

If that happened Google would probably ban Android phones that use Google apps from using third party browsers (like they already did for third party search providers). Then your tabs wouldn't be able to sync with mobile.

DuckDuckGo is terrible for non English speakers though.

Or you can use Firefox

Reply on news.ycombinator.com