Hacker News Reader

The success of 20% projects is a quasi-myth. Gmail is often touted, but you would do well to look up the employee # at Google of the inventor. Point being, you need (very) early hire clout or your 20% project isn’t going anywhere either.

Gmail wasn't a 20%, I don't know why anyone thinks it is. Buchheit was asked to work on it.

I thought Gmail or its predecessor was a third party/acquihire situation?

No, Buchheit was employee #23 at Google. He also developed AdSense as part of Gmail.

I must be thinking of Google Maps then. Or maybe Docs?

YouTube was a huge early acquisition for them around 2005 or 2006.

True, although I thought there was another big one whose name escapes me. Any ideas?

Thanks for mentioning that for the rest of HN’s sake.

Looks like Wikipedia actually has the full list.

DoubleClick and Waze are two other big ones that I remember seeming important at the time.

https://en.wikipedia.org/wiki/List_of_mergers_and_acquisitio...

Thanks for that list! One of my favorite Wikipedia-isms are these very “list of” pages.

I’m pretty sure it was Keyhole, for what it’s worth. Curiously, both Google and Keyhole are In-Q-Tel investments, although I’m not sure if that impacted/steered/influenced the acquisition, however.

https://en.wikipedia.org/wiki/Google_Earth#History

https://web.archive.org/web/20130605121646/http://www.busine...

https://en.wikipedia.org/wiki/In-Q-Tel

From that page:

> As of 2016, In-Q-Tel listed 325 investments, but more than 100 were kept secret, according to the Washington Post. The absence of disclosure can be due to national security concerns or simply because a startup company doesn’t want its financial ties to intelligence publicized.

> […]

> While In-Q-Tel is a nonprofit corporation, it differs from IARPA and other models in that its employees and trustees can profit from its investment. A Wall Street Journal investigation found that in 2016, nearly half of In-Q-Tel's trustees had a financial connection with a company the corporation had funded.

https://web.archive.org/web/20160831020609/https://www.wsj.c...

Which is to say, apparently insider trading isn’t just “illegal except when Congress/the President does it, except when it is illegal, actually,” if I’m parsing all this correctly; apparently In-Q-Tel gets special treatment too, in some cases.

https://en.wikipedia.org/wiki/2020_congressional_insider_tra...

https://en.wikipedia.org/wiki/STOCK_Act

https://en.wikipedia.org/wiki/Insider_trading#By_members_of_...

Chad Hurley is still one of my fave tech success stories.

Be like Chad Hurley.

Google Earth was an acquisition (formerly Keyhole), maybe that's what you were thinking of.

I think that was it. I replied to this point downthread:

https://news.ycombinator.com/item?id=39054947

I’d be glad to hear your perspective or thoughts in this thread or as a reply to my post re: Google Earth/Keyhole.

Maps came from the acquisition of Where 2. Here are a few links:

- https://en.wikipedia.org/wiki/Google_Maps#Acquisitions

- https://medium.com/@lewgus/the-untold-story-about-the-foundi...

>Gmail wasn't a 20%, I don't know why anyone thinks it is.

It's because some media stories used GMail as an example of a 20% project -- and -- those stories also end up being cited in places like Wikipedia:

- >Google is credited for popularizing the practice that 20 percent of an employee's time may be used for side projects. At Google, this led to the development of products such as Gmail and AdSense. : https://en.wikipedia.org/wiki/Side_project_time#cite_note-:1....

The Time Magazine article is one of the places where they say GMail was not a 20% project:

- >Gmail is often given as a shining example of the fruits of Google’s 20 percent time, its legendary policy of allowing engineers to divvy off part of their work hours for personal projects. Paul Buchheit, Gmail’s creator, disabused me of this notion. From the very beginning, “it was an official charge,” he says. “I was supposed to build an email thing.” : https://time.com/43263/gmail-10th-anniversary/#:~:text=Gmail...

The product they often presented as started in 20% time is Google news. I don't know the actual details, just this is what I remember from my time at Google (2006-2012).

It's not a myth. I had a successful 20% project when I was there and I was nobody special. I started an anti-bot system in 2010 and by 2014 it had become integrated into most of the top products. Last I heard there is a whole team working on it still, and there were also spinoffs.

That wasn't only a 20% project but a secret one! For several months I told my manager in daily standups that today was my 20% day, and that it was related to what the team did, but not what the project actually was. Everyone was cool with it.

It's possible that this was achievable only because I worked in an operational role. We had firefights that sometimes disrupted my ability to take the 20%, but you could partially bank the time and there weren't top-down imposed project deadlines.

I also used and experienced many other people's 20% projects whilst I was there. Many were small internal tools and systems that made working life easier but weren't the right size/shape to be staffed up as a full project with management attention, but some were products. At one point someone did a project that let you bid against yourself in the ad auction, to experiment with how to pay to remove ads. IIRC that was a 20% project. Google's early reputation for ML skill was established on the outside by Google Sets, a (for the time) astonishing demo that revealed Google was no mere fancy keyword matcher but had semantic understanding of words too. Pretty sure that was also a 20% project.

I felt pretty strongly at the time that the 20% policy was one of Google's secret weapons. The disdain and disinterest with which it's so often greeted outside of Google's walls is puzzling and sad, as is the quasi-myth that it didn't exist at all. I can believe that it existed to varying degrees depending on where and when you were, but, it's definitely not a myth.

I appreciate your comment and anecdata, and I’m glad you commented here.

That being said, I’m not sure whether it is you or I who does not know what “quasi” means.

I think we both have a quasi-understanding of the word :)

I had understood that the main reason products were killed off so easily were the promotion incentives: you get big awards for launching a product, less so for maintaining it and growing it and doing the hard iteration work required to continue improving it.

I don't understand why they never fixed the incentives.

They could keep starting products and instead of killing them, release them as open source. Or one click deploy on GCP.

Actually, that last one makes a lot of sense for pushing for more GCP adoption.

Open source is very hard for a bunch of reasons:

* Code at Google3 is one giant monolith that has dependencies on things that aren’t open sourced. Even if they could be, they don’t make sense outside of Google

* most services internally run on internal services like Borg and GFS rather than GCP for historical reasons. That maybe has changed now but I suspect a lot of stuff depends on internal infra not available on GCP

* A product can often have data dependencies. Just being able to spin up a separate copy may not mean much in terms of keeping a product alive past Google’s interest if the data is locked away behind Google’s private data stores. Then you want Google to start adding easy export options so that data can be exfiltrated from Google’s onto 3p versions of the product which is a legal, business, PR and technical risk

* Google has invested some amount of money and resources into a product. If a competitor takes that concept and is successful it’s embarrassing to execs at Google who missed the opportunity. Google Wave was dropped even though in many ways it’s a precursor to slack (it was open sourced though and went nowhere).

It’s a nice wish but I can’t imagine any realistic scenario where any business would go down this road until their shown a successful roadmap by a more enterprising business first.

Google Wave was my fave sunsetted project or Google graveyard project. First time I used it I could really see the potential. But as you pointed out, it's often not that simple.

I'm hard pressed to remember a feature in Wave that does not exist in Docs now. What is it that you miss? I just remember it being dog slow with a lot of latency.

Wave was supposed to be "equal parts conversation and document" (their description). It seems like they separated out the conversation part to Google Talk and made the document part Docs. Wave was kind of the precursor to the state of affairs we have now with Slack and Discord.

So, Google’s choice of architecture prevents them from open-sourcing or maintaining projects a tad longer than absolute necessary. I’d say they dug their own grave, if it has such a large impact on the trust of customers. Maybe Google3’s architecture was excellent for worldwide scalability and costs, but let’s see which one matters in the long term.

Google and their codebase architecture have been around for a while now in the scale of software lifecycle. How far out do we need to wait before its "long term", and how will we know if failure was from decades of business decisions vs repo architecture?

Let’s say their repo architecture didn’t help soften the blow of various business decisions of abandoning products.

There was a time when Google was also Guava and a myriad of Java libraries helping the rest of the world. It’s a design choice, they chose to stop helping, because of their repo. Perhaps it also helped avoid leaks, since copied code can’t be used outside of Google.

If anything Google releases more open source now than ever. Because of TensorFlow (if I recall correctly - my memory is fuzzy on timeline of what product it was) they released Bazel and then Abseil. Chrome didn’t have the clout to accomplish that but those things are open sourced and very high quality libraries. Tcmalloc is open sourced even though Google reaps no rewards from that. There’s a bunch of stuff released by Googlers as open source.

I listed many reasons why open sourcing wouldn’t make sense and you seem to have hyperfixated on just one.

Yes - Google chose a monorepo because of the engineering and business advantages it gave them. For example, all binaries had a common dependency that implemented all sorts of common functionality they found binaries on their servers needed out of the box - that’s why a simple c++ hello world on Google3 results in a many many mib binary that includes an http server for performance monitoring. It’s an opinionated codebase but the opinions are largely documented and explained and aren’t crazy (just because you may make other decisions or prioritize things differently doesn’t mean Google’s choice are wrong). And again, please don’t hyper focus on what I wrote as “oh it only has this one advantage”. There’s lots of advantages and I’m just giving very basic examples. It’s been a fairly long time since I worked at Google and someone who works there is going to have a more authoritative explanation of how things work now as things will inevitably changed some.

You may say they dug their own grave while the internal teams find it helps them run services and maintain things because there’s 1 way of interacting with the artifacts of lots of different teams. And they weren’t going to open source EOL projects anyway and maintenance decisions have nothing to do with google3 and are product and engineering led decisions.

Also, they have a bunch of tooling to manage open source projects that mirror google3 projects. It’s doable for projects that care / who it’s important to but it is also a lot of work (eg can write rules that bidirectionally rename files, functions, c++ namespaces, comments that can strip out code before outgoing synchronization, etc etc). if I recall correctly the tool itself was open sourced too.

As for customer trust, again I point out that not only is the Google3 aspect a minor detail, open sourcing wouldn’t even impact customer trust in a large way. It might with a certain segment of the technical population, but that’s not the people Google cares about in terms of public trust. And any open sourcing also has to find motivated maintainers who are not going to be motivated by 0 pay and a huge amount of bug reports and feature requests from users who started with maintenance support of paid engineers and focused product vision. Make sure not to extrapolate your own predilections as a tech engineer onto the general population.

I didn’t mean to focus entirely on Google’s approach to developers. However, one has to notice that Google has lost the upper hand over the web since ChatGPT came about. It can still be summarized in your sentence:

> but that’s not the people Google cares about in terms of public trust

Exactly. At the beginning, Google cared a lot about perception from technology-minded people. It was necessary to rise. Now, this original audience doesn’t matter to them, as you very well explain it, with your explanation that they weighed nothing in the balance of choices for all of the (perceived) upsides of their architecture. Because it’s a megalith with well-established business streams, which puts them outside of danger from the whims of public perception.

The business streams are:

- Workplace: Google is an email provider,

- GCP: Infrastructure company (who doesn’t care about developers, remember),

- Ads: Basically a leech. Everyone associates Google’s services to spying people by any means necessary (hi Chrome, hi Analytics, hi Google Search).

- Youtube: Who likes Youtubers? It’s the most degrading title in society, no-one would like their son-in-law to be a youtuber.

Morally, it has lost its stance. Technologically, it’s behind Microsoft. Socially, it has incentivized verbosity on the web so much that nothing is searchable anymore, shooting their own foot, and shooting the entire web with it.

The glorious USSR lasted 80 years on the remains of their former glory (and on the remains of food that was still produced). I just hope Google’s descent will be less painful, hopefully overnight.

> At the beginning, Google cared a lot about perception from technology-minded people. It was necessary to rise.

Sorry no. You’re overestimating the importance of techies here. Techies were the first on the web but in no way were techies critical for Google’s growth. Heck I was in junior high in Canada when I first came across Google and our librarian thought Yahoo was the better approach because manual expert indexing was surely better - we may have been nerds but none of us cared about open source. Google was just a novel technology that was better than its predecessors. And in terms of what made Google a success was figuring out AdSense. Without that Google would have died.

> However, one has to notice that Google has lost the upper hand over the web since ChatGPT came about.

That’s a nice claim but there’s no actual objective indication that’s actually true yet. It may well be true that the inflection point happened but please point to concrete data suggesting this rather than a blind assertion.

> The glorious USSR lasted 80 years on the remains of their former glory (and on the remains of food that was still produced). I just hope Google’s descent will be less painful, hopefully overnight.

The glorious USSR arguably came back around under Putin’s regime (call it whatever but it’s a very Soviet style political and social culture again).

Google is a massive player in the search space - bigger than MS was with operating systems in the 90s and MS still maintains its healthy dominance in that space for laptops/desktops even though Apple is the “cool” one and ships a lot of iOS devices and Google ships a lot of Android devices and only marginally beats out Windows market share (38% vs 31% vs 17% for ios [1]).

Anyway, you don’t like Google. We get it. But none of this has any relevance about whether open sourcing sunsetted products would in any way alter their trust in a broader sense - their customer base hasn’t been tech heavy since GMail took over from Hotmail (maybe even before then) and I think you’re over thinking how much influence tech nerds have especially as the broader community has gotten more technically adept than they were at the beginning. People now ask their “techie” friends for advice where techie now means I buy and use a lot of electronics gadgets and software tools and not I’m an engineer working in the field.

Indeed in the critiques you outlined of why you don’t like their income streams, nowhere do you even bother mentioning any sunsetted products you thought would be good. If anything, the lesson business leaders probably take away from what happened at Google is that letting engineers make product decisions is a bad idea and results in products being launched without strategy that then hurts the company image because you can’t keep running unsustainable products indefinitely if you’re trying to maintain some semblance of fiduciary responsibility. This is because they look at Facebook and Apple and Microsoft who rarely if ever cut products and only do so if they’re changing strategy drastically or there’s no significant user base (for Apple since they primarily do hardware this is easier and just means they don’t do refreshes of a failed product line or postpone that refresh for a few years).

[1] https://en.m.wikipedia.org/wiki/Usage_share_of_operating_sys...

Are you Gavin Belson defending Google on HN? You seem to be defending Google so much that you might be blindsided about how unpopular and irrelevant it’s becoming.

Just look here regarding the supposed popularity of Google, and supposedly only worldclass developers in small circles noticed:

https://9gag.com/gag/a04BOgq

Yes, 9Gag is a decent source of what random kiddos around the world think. Look at the comments. Google’s loss of monopoly will be slow at the beginning, then very sudden.

And when a company collapses, suddenly the 50-year readiness of its architecture doesn’t matter as much as immediate features, such as being built by geeks, for geeks.

I love how out of date you are in your opinions or in a bubble your opinions are.

YouTuber might be degrading to you but it’s one of the most desired position among teens.

> Also, they have a bunch of tooling to manage open source projects that mirror google3 projects. ... if I recall correctly the tool itself was open sourced too.

https://github.com/google/copybara

> Code at Google3 is one giant monolith

I’ve heard this stated before, and something about it feels off. There’s no way all of Google’s code is in one repo.

Like, does that monorepo contain the full Android operating system with Google Services and the firmware for all the Pixel devices? I feel pretty sure multiple teams at Google must have forks of the Linux kernel for one reason or another, would those be in there too? Is Chromium in there too? What about the whole golang project?

Even if I’m off about some of this, it feels like the statement “Code at Google is one giant monolith” needs some qualifiers.

Ye, like, is there a google.exe at 3912GB with 10k command line options like '--search-server --port=80' or '--Gsolitaire'?

no, there are thousands of Blaze build files that each have their own binaries, tests, libraries, etc.

To be fair though, the baseline hello world C++ binary is quite hefty (10s or 100s of MiB - can't recall) & comes with a bunch of command-line options & an HTTP server out of the box. Or at least did when I was there & I doubt it's changed much. All that functionality is to integrate the baseline set of SRE ops tooling that are common to all services that are deployed on Google servers (eg. HTTP server is to support scraping for performance monitoring integration similar to Prometheus/OpenTracing). It's mildly annoying because any command line tools you write pick up all of that useless functionality but it is easy to teach so trade offs.

There are certainly a few notable exceptions (Chrome and Android), but google3 really is a massive monorepo.

It does contain forks of tons of open source code in the third party folder. I don't recall if the Linux kernel is in /third_party but I would not be surprisied if it is.

A fair amount of the operations are actually documented publicly, ex. https://opensource.google/documentation/reference/thirdparty...

Android, golang, chromium and a few others were separate but there were 2B+ loc of other stuff in that repository

Note that google3 is a subset of the entire Google codebase. But it still contain more than half of its code (>2B LoC), mostly those running over their cloud. In contains all the infrastructure code as well, including Linux kernel, compiler, tool chain... literally everything needed. When you build some big binaries, you may see more than 500k~1M build targets analyzed and built (or pulled from cache).

I like that idea

Even in the early days, I don't believe Google was all that fast about launching new products? The "fail fast" stuff was about people working on blue-sky ideas that failed to get traction before they were launched, or maybe just launched in Google Labs [1].

[1] https://en.wikipedia.org/wiki/Google_Labs

The mistake Google has done here is that it deploys and launches and uses to garner attention projects under it's name.

There is little to no, this is an experimental project by x team. It is this is a Google product.

If everything is a product and you don't support most or expect most to die, then you damage the collective product that is the Google brand. And as a result how your employees feel and are treated about experimenting.

> There is little to no “this is an experimental project by x team”

They communicated quite clearly when products were in beta. For example, they communicated clearly that Gmail was in Beta until 2009 (launched in 2004). What’s less clear is what makes a beta, when a tremendously successful product remains in beta for so long.

Another product that came out of beta in 2009 was… drumrolls… Google Reader[1].

[1] https://www.alphr.com/news/home-and-leisure/125603/google-re...

Google Domains was born in 2015, exited beta in 2022, and was killed in 2023.

They do that, this is what their various Labs and subsidiaries and internal accelerators are there for. The issue is that people figure out it's associated with Google and then expect it to be maintained as such.

I do wonder how people figured out that Google was associated with Google+, Google Inbox, Google Reader, Google Talk, Google Video, Google Wave, ...

(I'm just kidding; I don't actually hold a grudge against Google for terminating projects. Although I do miss Inbox. And it does looks like Google Labs was killed too?)

Those were official products, right?

That's fair enough, I suppose, though I think those are also primarily where the "Google kills everything" reputation comes from - not the Labs products.

> So then what? The answer probably should be that Google should buy up startups that have figured out product-market fit and just need to scale. They can’t do that though because the FTC is already breathing down their neck with anti-trust suits.

And for damn good reason. That kind of behavior quashes competition and encourages that ZIRP era thinking of growth at all costs so you can get acquired before everyone realizes your business model sucks. Maybe they should just focus on their core business so they don’t get buried in the shift to AI.

https://gcemetery.co/ Google’s reputation is not only in innovation

FWIW, the only person I've ever actually met with a 20% project was working on quantum

I had several 20% projects from 2012 to 2016, all of which were on the Cloud Business side (ie not engineering). It was very much a thing up until Larry Page stepped down as CEO