Mistakes Google made in scaling its organization | Hacker News Reader

Mistakes Google made in scaling its organization

quora.com

73 pointsplinkplonk26 comments

As an ex-Googler, I disagree with this second-hand account. Yeah, political infighting sometimes happens and things get shook up, but this person paints the picture of no project ever launching. Addressing some points:

1. See item 3.

2. VP changes rarely happen compared to a PM change. And a PM who comes may change some of the goals, but the focus area of your product doesn't change, e.g. your optimization tool for display advertising will continue to be an optimization tool for display advertising. It would be hard to find your project suddenly disagreeing with some "existing strategy."

3. New products that launch should have a UX designer contributing or at least advising. There should be mocks for the frontend engineers to follow, and the backend engineers are just told to make it work.

4. Yeah, there's a checklist, and it is long. SREs want to minimize the number of pages there, but pages happen because something's going bad, and they want you to have proper monitoring in place to address the pages quickly. The launch checklist also focuses a lot on addressing potential security vulnerabilities. Google is tough on security.

5. Google's all about the software compensating for commodity hardware. I don't think too many teams have specialized hardware. You can ask for what resources you need, but would have to fight for what form you get them in.

7. Can't speak for this. Why reorganize marketing?

8. While I agree that Google likes to fail fast, in my opinion some Google products probably didn't get the love or time they needed to take off. I wasn't really privy to how those decisions are made.

I think calling Google processes "mistakes" is a little strong. For example, I recall at one point at Google where they were trying to improve latency on the products, so you weren't allowed to launch a new feature if it hurt latency.

I could imagine somebody who just made an awesome feature that made an application 5% slower complain about how stupid that policy is. However, they don't realize that however awesome their feature is, speed of applications is a hidden cost that everybody bears, and sometimes the priority for a company may differ with a specific developers vision. Google wants their products to be fast, wants them to be available, and wants them to be secure. I'm not saying that everything they do is great and their processes and checklists are optimal, but when they have processes designed to do a certain thing, it isn't fair to say their processes are a mistake because they don't do something else.

Very much agreed. When an organization becomes large it needs to make tough choices. In an organization built around product-centric teams, it may be easier to release products more quickly and the average employee may perceive a smoother course from concept to execution, but the organization will quickly discover that horizontal changes become extremely difficult. The frustrated product developer who can't get products out is replaced by the frustrated operations guy who can't patch unsecure systems fast enough. Which one is worse? For Google, I would say the latter. While it always seems like a tragedy to hear about "great" products getting killed, pre-birth, in the corporate mill, I much prefer my stable, high uptime, browser-performant Google to a Google that releases way more product than it already does.

Not to mention that most products (like most startups) fail. As a product-centric company, do you break teams apart after a product fails? If a product succeeds, do you take that successful team and move it to the other product? That may work if you're lucky enough to have a team with the proper skills balance (not to mention passion) to move across different products. What about career goals? People in a particular field tend to want to work with the best people in that field--a huge draw of working at Google is working with the best people in Field X. In a functional organization, the functions (almost) never go away--so the teams are stable.

Of course, being on the outside I can't speak for the problems that are mentioned, such as in-fighting, lack of executive discipline ("You will not launch your product because I don't like your hair style, FUUUUUUUUU!"), etc., but those problems are bad regardless of organization type.

to me presence of in-fighting is a manifestation that whatever executive is responsible for both parties and the issue at hands is just not able or avoiding doing his/her job - taking decisions. Usually it is a combination of the inability and being afraid to take the decision and responsibility for it. "Fish starts to rot from the head" as they say in my old country.

Honestly that sounds typical of a large organization, everyone wants to put their fingerprint on a project when it's way up. Once everyone gets their hands on it, every vp packs the team with their buddies, burdens you with no-show contractors from a firm they have a "relationship" with. Then everyone starts scuttling away to the next big thing. Of course someone gets bored one day, and decides to redefine the entire scope of the project, pushes it as the next big thing, then everyone come scuttling back.

Exactly. It's a scramble for symbolic ownership that obscures the fact that everybody is increasingly reluctant to take real ownership. Accountability doesn't scale well.

It's always good to have an open discussion, but I don't find it convincing when an anonymous guy on Quora based on second-hand unverified information claims that he knows better what is good of bad for Google.

I've never seen success when an organization organizes around business function. Invariably, the function takes precedence over the product.

I'd enter Oracle as a counter example.

Personally, however, this is not something I would want to achieve, but to each his own I guess.

Maybe a corollary for businesses whose product is based on lock-in. Poor software design is sort of the point as long as you keep getting your hooks into people.

Not sure if this is off-topic, but have any company ever grown as fast to be as big as Google have? Successfully?

In the light of this article, Google should have a play area with only limited network and processing power for publishing betas of new products.

For example, anything that's Google Search or GMail or similar must meet some performance, stability, and realiability criteria but those applications would also get to run on the big clusters. Conversely, it wouldn't matter if a beta-toy is a bit buggy or hogs down the servers on the playground since it's just playground.

Some users would bear with the unfinished tone and start using the apps there, and at times it would happen that one application would gain lots of momentum, and Google could consider fixing and lifting that one to the full service level.

That is not a mistake. It's a corporation.

That is not a mistake. It's a large corporation. But I repeat myself.

So this basically reads as someone who's taken a couple of Org Design courses in college arguing that it's actually significant whether the top level of an organization is "product" or "function" ("geography" is another alternative), and as a result he basically comes across as someone who could use a little "experience" so that he understands that scaling an organization isn't just about some analytical framework that he learned in school, but about people, dynamics, history, politics, egos, a lot of stuff that is really "hard."

And that almost every fast growing company gets it, in some form or another.. "wrong." But they also find success despite their failure.

Whereas the guy who took the couple of Org Design courses probably ends up working for a VC running innumerable other companies straight into the ground.

I would encourage you to talk to some people who you trust who have worked on a few different projects at google (especially anything not related to search or ads) and see if they think this answer is theoretical.

If you don't have people there you can ask, take a close look at what has happened to the products and teams that google has acquired.

The problem is that people who say it's a "functional organization" problem are both correct and incorrect at the same time.

The kind of problems described certainly seem to have been exacerbated by the way they are structured. But all organization structures have their own problems, and the issues caused by (say) a project based organization* might have been even more detrimental to success.

The most professionally managed organizations I have worked for have generally been matrix orgs (sort of functional, with projects 'borrowing' resource for a year or two at a time) and that worked pretty well, but only because they burned a lot of time and money managing the process. In many cases it may be cheaper just to live with doing it imperfectly.

*Lack of product integration, break up of technical pool of talent, even bigger resource fights, duplication of effort, lack of communication, and zombie projects amongst other things.

What can Google do to fix those underlying issues?

Allow autonomy to product groups. They should be run like little startups: choose whatever tech stack they want, hire their own PM's, do their own hiring (Google recruiting is a complete wasteland), and live or die by their scucess/failure.

It'll never happen, because it's diametrically opposed to the founding ideology, which is that every problem has a rational solution. Without getting too Ayn Rand on this, it's the classic mistake of central planning vs personal initiative.

Are you saying they should be able to go off into the wilderness, decide to run on Heroku or something and come back with an app? Companies invest huge sums of money on infrastructure and the expertise to manage it. Google has too big an infrastructure to let each project roll it's own.

Let's say google reengineered to provide something like ec2,each team can run it's own images. does each group hire security experts? Does each group have to solve scaling problems google already faced?

I think maybe it could work like this.Google proper hires domain experts, security, engineering etc. Provides each mini startup with a budget, mini startups can spend the cash on the Google proper experts, or go outside for areas where Google is lacking. Mini startup pays hosting fees to run on Google's EC2 like infrastructure, or use whatever Google has in place now.

What if you let teams select from internal security teams, internal scaling teams, etc?

You can run a wide range of things on EC2, but I'd be willing to bet certain sets of things dominate.

If the solution google has internally really is the best solution available, let it compete. It'll win until it's no longer the best.

You'll get an internal 'Microsoft' who focuses on things like lock-in instead of quality, and other zero or negative-sum games like that. You'll need an internal patent and copyright system to keep one team from freeriding on another, etc. etc.

Right, I don't have to implement a queuing system because a guy down the hall did it. But he sunk his budget into developing it and I get to swoop in and benefit. That is not a bad thing, that is how corporations are supposed to work, it would be worse if we burned two budgets implementing a queuing system, but it is a bit unfair that he burns his budget for my benefit. Maybe if I paid him "licensing" fees it would be more equitable, but then we have another porcupine to hug, managing all the internal licensing.

I remember reading at the beginning of the year that Google wants to change the way they are organized, in order to stop their employees who were frustrated with their bureaucracy from defecting to Facebook and other start-ups. A possible form would be similar to the advertising group/holdings, where each agency has a lot of autonomy.

I think it's feasible to do that at Google if the infrastructure group would be spun off as a stand-alone entity (let's call it Google Ops) that can offer resources and consulting to any product group. When a new product is started, it would be similar to a start-up: 3 people in a room in the Googleplex who could choose to work with Google Ops, Amazon EC2, or something else. And Google would own a majority stake in this "start-up" and it would give them a certain budget (up to $1M - $10M).

I've worked at places that worked a bit like this. The problem is that it leads to fierce competition between the groups to the detriment of the company as a whole. Groups will think nothing of costing the company $10 million if they can make $1 million, and stopping your nearest competitor group from making money is almost as good as making money yourself.

From my experience in working at another large Internet company, my advice to Google would be: get rid of middle management. Get rid of the people who are not actually touching the product (in terms of code, design, testing, etc.). The impact of PMs is exponential in slowing down the processes in a company like Google (i.e. in the Internet space). Others may disagree (for example, established companies like Oracle may not fit this model); but I think if you're competing in the fast-paced "Web" world, you can't be slowed down by PMs.

Good engineers are fairly self-driven. Hire people who work well in a team; make sure they get along well and just let them loose with minimal supervision. Have a decent set of guidelines in place; make an "architect" available to them who they can call on for advice and "best practices".

I am reminded of this old article: http://www.fastcompany.com/magazine/28/ge.html It's definitely worth a read.

Reply on news.ycombinator.com