What have you learned at Google, as a Software Engineer, that you think was interesting and helped you up your game as an engineer? Anything you can share with people outside of Google that can help them improve as engineers?
Visibility is very important to getting a promotion at a large company. Selling your work is important.
To move up, you must be playing the "choose a good project or team" game for at least 6 months before you try to get promoted. Preferably for a year or more to hit the right checkboxes for multiple cycles.
If you fail to do so, you can do absolutely amazing work but rigid processes and evaluation criteria will conspire to defeat you in a promotion committee setting.
At least, that's true in my company. From my ex-Google peers it seems to be true there as well.
Being in a smaller office means you get fewer of the best projects available to you. Reorgs sometimes steal them. Cancelling projects makes the last half a waste of time from a promotion standpoint.
As for what constitutes a good project. It will:
* Let you lead it
* Have peers at your level + one or two
* Work with multiple other teams
* Ideally work with multiple teams outside of your group, e.g. you're in, say, a chat app team and get to solve issues for some other app
* Good product manager and designers with a well thought out product; it's not fair, but if the project is successful business-wise people will often incorrectly attribute that to your own skills
* Have large engineering scope. Committees get confused and sometimes think simple designs to solve complex problems are bad. You want to solve problems that are complex even in terms of the solution (or can be made to sound complex) to check more boxes.
* Allows you to solve problems for other teams
I haven't optimized for promotion and thats personally hurt me in the quest to get to L6. I very, very close but didn't quite make the leap.
Don't ignore the flaws in this committee process. Exploit them.
Don't be me.
Edit: pretty sure I was off by one originally. But you get the idea: it’s a triangular organization with a lot of people at the base.
You can hide your level such that other people don’t see it, which is another difference between Google and my current company where people introduce each other by their titles, which makes me want to quit every time. Nobody at Google has ever said “And now I’d like to turn the meeting over to Principal Engineer Jane Doe”. Most people don’t mention it.
Pure engineering skills (even the thought leadership aspect of it) doesn't directly translate in management skills and often people get "rewarded" by promoting them to a position where they can no longer use the very skills they have been promoted for.
Related concept / good read: https://en.m.wikipedia.org/wiki/Peter_principle
This obviously goes hand in hand with higher pay and increased role and input. In general you might lead others from principal engineer upwards.
I am sad that top tier software engineering environments have lost sight of the value of simplicity. Perhaps it is doomed to become the dismal engineering, like economics is the dismal science, that never quite seems to have the answers for economics, or never supplies software that is finished and bug free for software engineering.
I worked in a top tier engineering environment (not SW). The problem was the same there. They valued the complexity of the solution, not the impact it had on the business for solving the problem.
I think this is generally a problem in large companies. In a tiny company the impact is much more apparent to the bottom line. Whereas in a big company, it's hard to see how much difference your solution made in the big picture because there are plenty of other bottlenecks in the pipeline (and out of your control).
I agree, but a more charitable reading is that they want you to demonstrate that you can deliver a complex solution if that's the only way to get things done.
Complexity is highly subjective, of course.
Suppose I need a distributed atomic counter service. After doing a lot of research, I come to the conclusion that the best solution is to write a wrapper around something in AWS or GCP but structure things in a way that other teams can use it.
Other teams are now less likely to need to do that research and can just get stuff done.
But the actual implementation is very simple.
Is that complex?
Maybe, maybe not.
This does provide a perverse incentive to continuously reinvent the wheel, though, which is the worst part of it.
Why not re-implement the low level goodies in DynamoDB if that'll increase the odds of promotion? Many engineers think along those lines.
I am much happier working in boring government where working hard for family and with mature/secure with themselves (older) adults is paramount & prevalent. Thankfully not an enticing environment for douchebags.
For those under 40 everyone makes it harder on themselves then it ever needs to be. Not saying people over 40 aren't backstabbers, but its way less and going to work is more enjoyable then you worrying always about looking better then the next, you not believing in yourself and or dealing with those that fall under those categories and are a horror to work with and for.
The only real differences are that leadership people generally won’t fuck you for personal financial gain, because everyone makes about the same at senior levels, and employees rarely get fired, just shuffled around.
If you’re a contractor and are smart, you stay friendly with everyone and always listen. Events that will get you purged are either procurement problems or bad political decisions with respect to the budget for whatever you do.
With employees, you need to do the dance of getting shit done while avoiding accountability. You need to get attention for performance without pisiing people off. It’s critical to find and attach yourself to a good leader.
The government was worse IMO in that work was very slow and bureaucratic even beyond the big company overhead.
Politics couldn't be ignored; you need to keep the contract, outmaneuver the other guys going golfing with the agency head every weekend, deal with having 10 half-backed solutions you contractually cannot touch but need to in order to get things done.
Promotions were almost always about tenure, certification, and education. You could never get far ahead if you're more skilled. Worse: you couldn't really fall behind, either.
In the right spot, though, it would be lower stress. But much lower pay
It is and it's not.
I got mediocre plus results without too much politics, and I guess that for the long run I could advance without back stabbing anyone (the site was closed before I had a chance to test my assumption) but I agree that most of the moving up had to include politics, stress, insecurities, backstabbing and stress.
On the other hand you can relatively easy get to positions where you influence tens or more engineers while in smaller places those positions are rare and super senior
In terms of what I learned? I think the most important lesson was: Don't get bogged down in details that are irrelevant to you & count on others to do their jobs. I came from a small (~15 engineer) company where we all knew the entire product from top to bottom. At Google, that was impossible.
Congrats on making the leap. It's tough.
> The key (I think) was working across teams with people who were already L6 or higher, and who were willing to write for my promo packet.
I agree; having L+1 reviewers is key. In my case I had the equivalent of a Google L7 or L8, but it didn't work out.
Having a good enough project, even though it's often out of your control, is very important.
Also, can you share more advice on the technical side, like what do you are the abstractions, mind models and technologies that you think are applicable outside of Google and would be worthy of studying?
The technical side was probably not terribly relevant to you. I worked mostly on the Linux kernel in C, on drivers and firmware, and never did anything in the main Google repo.
For L6, for example, you might be able to convince the recruiter to set up the right interview process, but your level is 100% tied to your interview performance.
Salary is a different story--negotiate, always, but be aware level and salary are related.
I wonder if your advice can be applied in non-google setting? For example, in some general enterprise company.
If I ever find myself working at a FAANG, I do hope that instead of pursuing promotion, I'll stick to stuff that matters to me as an engineer.
While having a FAANG on my resume matters too, I'd much rather use that time to develop my technical skills than to improve my PR skills. In my experience (at a medium sized company with 200+ engineers) these two goals often conflict.
Having feedback from L+1 or +2 peers is very important, otherwise it can be very hard for the committee to judge performance.
Actually "leading" a project less so - often a single "project" can be viewed as multiple components / projects for promo purposes, what counts (depending on level) is initiative, leadership, and technical contributions. It's not necessary to be the formal lead of a big project for this purpose, but rather for your senior peers to attest how well you did in those areas and how it helped the project launch X months faster, improved QPS by X or whatever metric.
As pointed out in other comments, it can be hard for the committee to see impact, so the more hard evidence you have (before/after resource usage, bugs fixed, etc) the better. Same for improving the codebase, document and show the evidence!
For me, the only reason I will never go back to GOOG is blind allocation. IMO that's a bug, not a feature.
Google et al often do general hiring. This means that they hire you and then later decide where to assign you afterwards.
I had thought at Google hiring managers could bid on you and some kind of choice could be made. But maybe that's only in big enough offices with good enough resumes.
It's bad because you don't know which team you're joining and the quality of the manager. It's good because it lets the company assign you to the most important areas first.
Contrast this with a normal process where the hiring manager for the team you're joining is in the interview loop. You know ahead of time which team you're joining.
That hugely slants the purpose of the interview toward Google, as the candidate really has no idea who they will be working with, or how they think. The interview should go both ways and provide information to both sides on if it's a good fit, but at Google, they go only one way.
This might be a small point for most, but for minority candidates, knowing who you will be working with and having confidence they won't actively cause you grief is a big sticking point. At least it was for me (I turned them down).
That's why I mentioned that I've been asked to do lunch or a phone call with prospective hires. I'm not a manager, just a grunt, and these candidates wanted to speak with people from the team.
I mean, thats the problem with vertical schemes and bureocracy.. the guy who is micro-managing is also micro-managed by some sort of results that actually ends not being very pragmatic, end give him good points in this blind objective system who ranks him and the people he manage according to some parameters that in the end they 'game' to look good.
Of course we want to fight the impermanence, but we should not create a bubble to feel safe inside of it as a virtual environment with rules that doesnt change, just because we miss the peace in the maternal woomb.
The impermanence and uncertainty is a reality, we should not fight in order to avoid it by being alienated of its nature, but to use our science, math and tools after all to understand it, and to navigate better through it.
The XXI century requires a deep change in our philosophical disposition and culture.
Its not just about Google, the corporate culture of Google is a side effect of our culture as anything else that grows inside of it.
But the rigid schemes are a clear sign that it will stop innovating more and more with time as the old startup status-quo defying spirit is completely gone.
But yea, things are not always rosy and maybe Google indeed isn't the right fit.
The biggest thing I grew to appreciate was that for iterating large scale production systems, rollout plans are as important as anything else. A very large change may be cost or risk prohibitive to release at once, but with thought you can subdivide the release into easier to rollout and verify subcomponents, you can usually perform the same work as a series of lower risk well understood rollouts. That's critical enough where it's worth planning for this from the design phases: much as how you may write software differently to allow for good unit tests, you may want to develop systems differently to allow for good release strategies.
Google is a large company, and I think there are software engineers learning very different things from what I focused on. I worked on large scale machine learning in ads; someone working on chrome or Android likely learned very different things.
Yeah Google has a lot to teach about building and maintaining reliable systems.
I noticed that a lot of that, including the name SRE, gets cargo-culted by other companies without really understanding what it is.
I also guess that there might be people within Google that don't end up working in projects that invest in reliability (for whatever reason) and so might have had different experience.
What I haven't learned: how to thrive in a huge organization with people more bureaucratically ambitious/motivated than you. How to turn a blind eye to broken waterfall method dysfunction. Starting to get to me. Probably looking for the door soon; p.s. entertaining offers for remote (Canada) work.
want to add an email or something to your HN profile :-)
You’ll find his LinkedIn profile searching his name online.
They don't cancel each other out exactly though, so some things are way easier (spinning up a huge computation), while some things are way harder (making a small change to shared components).
Working on shared code is so easy at Google because they have a fantastic cross-referenced code browser and search engine, ways to inspect production jobs and configs that anyone can access, and everything is open and visible.
By contrast the other large publicly-traded cloud company where I now work has the opposite culture: no code search worth using, “why are you touching my project?” culture, not enough tests to give anyone confidence in a change. The polar opposite of Google.
I believe when he said "organizational efficiency" he was not referring to technical inefficiencies, but rather chain of command, how organizations are structured, etc.
Not at Google, but that is the case in my company as well. In most of the company, doing extremely great technical work is not going to get you far.
I am guessing that your changes did not change the semantic behavior of other people's code. e.g. you were replacing one hash-map implementation with a more efficient one. That's something where obviously Google's tools do cancel out the inherent inefficiency of large organizations/code bases.
My point was that Google has the level of efficiency of a small organization, despite being a large organization. That the default for big organizations is to "just suck" as you described, makes this remarkable, but I never felt like this resulted in a superhuman level of productivity that you might expect considering how advanced the tools are.
Ultimately no decrease in dependencies have been made. Projects are barely bundled together any more than before. The only thing that’s made that possible has been Docker and Docker Compose, and our monorepo strategy makes it more difficult to build images due to its size and our topical file systen layout making it almost impossible to check out only the parts of the repo you need.
The monorepo itself may be a good idea but it’s no silver bullet. I’d say it’s one of the worst choices we’ve made due to the cultural impact it’s had. In our case I think we’d be better off with small monolithic repos due to a lack of proper training around how to manage either many small repos or a large monorepo.
Often it's better to introduce a new function and migrate usages to it across multiple commits.
Either way, it's on you to fix everyone else's code, which is a different mindset from releasing a library and letting downstream users do the migration themselves when they feel like it.
* Really practice open/active listening and critical thinking: there are very smart people and it's really good to assume good intent when they talk. Critical thinking and question is more around 'where is the disconnect? How do I try and actively move towards and see their point of view'.
* Questioning authority and being able to say no.
* Don't use code reviews as a debate forum but instead as a learning forum. Granted time is of essence, so a timely resolution may be necessary but learning and actively listening through reviews helps your skills as a coder.
* Cut through heirarchy and organizational politics to achieve the larger goals for the team, the project and ultimately for Google.
This is less true now because a lot of their tooling has been exported either directly or by rumors leading to it getting rebuilt outside. I'm talking containerization / orchestration and protobufs specifically.
quality of the tool has also significant impact on the end result. Looking at the number of unsuccessful external projects, I bet they also have enough mess inside.
Note: this is not something I'm particularly proud of or anything. Just an observation.
2) Humans are easily fooled by signaling and vanity metrics, exploit this quirk
3) Depends on the team, but in general working at a big company is 10X easier and more relaxing than working at a startup. You’ll probably never go back to working at a small company after realizing this.
4) Buddying up to the right people will dramatically alter your experience in a big company—meritocracy can never exist when human emotions are involved
5) the difficulty of the interview process is just plain bizarre given how simple the daily jobs of 95% of the engineers are
6) you will have the nagging feeling a high school kid could do most of the (non engineering) jobs in the company with a few months of on the job training
7) due to 5 and 6, unless you are a sociopath, you will feel guilty on a daily basis for building generational wealth while an immigrant contractor struggling to get by dishes out your free food
8) you will learn that at most big companies, the innovation is in the past
9) on the bright side, google does have really cool internal tools and perks
10) after leaving, people will grossly overestimate your intelligence and skill level due to the name google being on your resume (see #2)
https://talksat.withgoogle.com/
Not restricted to just software engineering, though.
For the engineering side: https://www.youtube.com/user/GoogleDevelopers
The biggest thing I've learned is how software changes over time. Previously I thought of a good program as like a crystal. Write it correctly the first time, and it doesn't need to change much, if at all. Sure you might want to add features or fix bugs, but otherwise you write it once and it's good forever, right?
Most programs are closer to organisms than crystals. They're adapted to specific niche, and when the world changes around them they need to either adapt or die. This is the biggest gap I notice between senior devs and strong junior devs. Junior devs have mostly built things, released them or turned them in for credit, and then moved on. It's rare to find a junior dev that has maintained an app for the long haul, and it accounts for so much of the advice that I found strange when I was junior:
* code is read many more times than it is written
* be very mindful of your dependencies, if push comes to shove you could end up owning them
* commenting why is often more important than commenting how. it's good to comment about both, but it's much more important to comment why. the source code explains how, but it has nothing about why
* it's ok to love your code, but delete it the first chance you get. it will save you and the people around you a lot of time
Do you think I'm overselling things? Maybe that's a problem for people who write code in $cursed_language on $bad_platform but your tools favor writing True and Long Lasting Solutions That Stand the Test of Time!
Let me give a handful of real world examples:
* A major security vulnerability is found in an API that previously seemed safe. Refactor your code not to use it.
* Turning on a certain compiler optimization breaks a pattern common in your code. After some experimentation, you discover an ugly fix that you then need to apply to everywhere in the code that uses that pattern.
* A compiler/browser/hardware update adds support for a feature that your code has been using a bunch of ugly workarounds to cope without. Do you use the new feature? Do you still support running without the feature? Is it a breaking change to users downstream of your code? How much does it improve things, how hard will it be for your users to update, and how many users do you have?
* You need to migrate to the new version of one of your dependencies. Maybe this is just a version bump, maybe you end up needing to review every single line of your app.
* Your app for phones and desktops is working great! We want to run it on TVs as well! This means a brand new (and super weird) matrix of features that are and aren't supported, from hardware to OS to UI.
In short, the only constant is change. Write your code so that you can go away for six months and forget just about everything about it and still be able to maintain it. Lots of tests, document why decisions were made, and it's like Bruce Lee said, be like water.
It's such an uphill battle to instill these values in young organizations when leadership has no strong engineering background.
The way I've often describe software to clients is likening it to Formula 1 cars. With a normal car, you turn the key and drive, and it should definitely work for anyone. With a Formula 1 car, the tyres need to be warm, the driver needs to be experienced, and there are so many reasons why a working car can stall that careful monitoring, maintenance, and revision needs to be done long-term by an experienced team.
Sure, the software on your computer might run like the former, but in the background that huge pit stop team has been plugging away for years, and will continue to plug away with optimisations and fixes.
It’s a very miserable monoculture.
How so?
That’s why I changed my view on who was the key person for google success. I used to believe Larry and Sergei, then urs, now I put on Laszlo Bock.
It's not even touching the techniques and approaches yet. I was too lazy.
Here a summary of my understanding: * Relentlessly hire the best, train them and give space for them to grow and shine. This is the basis. It's not that a less quality engineer cannot grow, it's just too costly to do that at large scale. * Build the infrastructure to support large scale engineering. The best example is the idea of "warehouse scale computer" that is the data center, embodied in systems like Borg Spanner etc. * Relentless consistency at global scale. Example would be the Google C++ style guide. It's opinionated at its time, but is crucial to ensure a large C++ code base to grow to big size. * Engineering oriented front line management. L6-L8 managers are mostly engineering focused. That's necessary. * Give the super star the super star treatment. Google's engineers are rewarded as strategic asset of the company. Numerous example are made public.
You can keep the link, in case it's updated in the future.
What is internal is more featureful. And it covers the entire code base.
- Delivery was incredibly badly managed, I have to shut one one eye when I sneeze due to sinus damage caused by working when ill on a project with global visibility and no cover.
- 'Decisions based on data' isn't a thing: if you want an outcome, you research to that outcome or don't depending on how you feel.
As others have said having Google on your resume is great.
- GAE was missing basic functionality that we (as their biggest customer) couldn't get them to work on. The engineer in charge of GAE had other tasks and didn't care for this role. We ended up doing this work ourselves and I'm relatively sure this is now inside the App Engine SDK.
- Delivery was incredibly badly managed, I have to shut one one eye when I sneeze due to sinus damage caused by working when ill on a project with global visibility and no cover.
- 'Decisions based on data' isn't a thing: if you want an outcome, you research to that outcome or don't depending on how you feel.
As others have said having Google on your resume is great.
What I learned, design-wise, came from a coworker. They planned everything out in advance and would draw out components on a whiteboard and define what they'd do. A few weeks would pass before they'd write any code. In contrast, I would start with writing skeleton classes and intuit the breakpoints for building a new class. We ended up with similar code structure at the end of the day, since we were both designing the architecture (even if our methods were different). But in code reviews, I would focus on if the new code made sense within the scope of the change; are there any clear bugs, code duplication, is there a more simple approach, etc. My coworker always went back to the whiteboard and drew up the components again, making sure the new code fit within the originally defined scope or if a larger change would make more sense. Because my skeleton classes were quickly replaced with implementations, I lost my original frame of reference and only focused on what was in front of me. I could still tell when something was obviously wrong (stop sending me CLs with global state to cross talk between components!), but there was creep in the CLs I approved and the code would slowly get more complicated until I realized it needed to be refactored.
There's a tradeoff (like in anything). You can't design your own solution for every code review, as it takes too much time and removes autonomy from your peers. But remembering to step back and look at the big picture was something I needed.
There were other things I learned, like structuring code in ways that it's hard to write bugs. For example, you could write a method like...
void foo(Callback callback) {
if (error1) {
callback.onError();
return;
}
if (error2) {
callback.onError();
return;
}
if (error3) {
callback.onError();
return;
}
doWork();
callback.onSuccess();
}
void foo(Callback callback) {
if (bar()) {
callback.onSuccess();
} else {
callback.onError();
}
}
boolean bar() {
if (error1) {
return false;
}
if (error2) {
return false;
}
if (error3) {
return false;
}
doWork();
return true;
}
Oh, and here's one last tip that I only realized needs to be reiterated because most of my coworkers forget it. Validate incoming data! Every API needs a wrapper around the entry point that...
* Verifies the caller is allowed to call that method
* Verifies the method can be called at this point in time (eg. hackerNewsApi.postComment(threadId, msg) only works if the threadId is valid)
* Verifies that the arguments make sense (eg. 'msg' is not empty/null/above the max comment size).
Glossary: CL = changelist ~= pull request
Design docs and comments by great engineers like Jeff Dean are openly accessible. Treasure cove of knowledge that I try to dip into everyday.
Most companies that interview people outside of FAANG tend to have easier interview questions in my experience. Some even forego algorithms all together.
For googlers who worked at companies outside of FAANG or startups. Is there a noticeable difference when working these people? Will you learn more from a google level engineer then you will from someone who can't pass the google interview? What are your thoughts about it?
If you think that FAANG engineers are genuinely better, say it. There's no need to be politically correct here, honesty is appreciated.
I will say, those problems do filter out a lot of bad people. But it only ensures a minimum level of competence. It doesn’t mean everybody is insanely smart.
Thus by statistics and logic, the majority of the worlds engineers cannot get past the google interview. This is not something I am thinking, this is a statistical reality.
Literally, in order for those engineers to work anywhere else the algorithm questions must be significantly easier in other companies outside of the elite FAANG. This is inline with logic and my observations.
My question is... for googlers who have spent time outside of google at a more scrapy startup. Does the interview filter actually create a better class of engineers? Do you learn much more working at google then you would at a company who's engineers typically can't get into google? What's the experience like if there is any?
Don't get me wrong. I'm not worshipping googlers or asking for tips. A real dichotomy exists in the interview stage for the few engineers who pass and the majority who don't... I'm just curious if this dichotomy becomes obvious outside of the interview... like on the job? Do you notice a difference when working with a google engineer vs. someone who probably can't ever get into google. It's a complicated question with a complicated answer. I'm not really looking for some humble answer telling me that google engineers aren't that much different from other engineers when the interview essentially disproves this statement by filtering out everyone except for a certain type of person.
I want an honest and candid answer.
As a result you will on average learn more than at most other companies. Of course there are trading firms, quant hedge funds, and a few other big tech companies with even higher hiring bars where you will learn more. And some of the things you learn, anywhere, while very useful to getting stuff done at where you work, will not be readily transferable elsewhere.
But large companies as I’ve said before are basically all the same. The people you interact with are a vanishingly small subset of the entire company so it doesn’t even really make sense to speak in terms of what it’s like at the entire company.
The interviewing filter mostly does one thing which is that it biases the engineers at google to be generally good at solving data structures and algorithms. This, by the way, is a skill that can be taught — and even though it might roughly approximate engineering ability it doesn’t map directly.
Your answer is great, btw, I just noted that you used the word "average" over a personal example so I wanted to get an idea of where you're coming from.
In ten years of writing software, I've had a handful of emails/messages from FAANG recruiters, and they've never ended up in an actual interview. I won't name names, but I was offered an interview and we agreed a three week timeframe to prepare for the phone screen, but that screen never came.
In another 6-12 months I might be lucky enough to get an email from someone else, but unless I spend my entire life on Leetcode to prepare for an email that "might" come it's likely that I'd fail it anyway.
Let's also not pretend that these interviews are easy. If we assume that FAANG interviews are around Leetcode medium to hard level, it still takes a significant amount of ability to do well, even if you can write a Max Heap from scratch and can traverse a BST. Even if you're had time to warm up for these interviews, you can come across a hard problem that rules you out for that company.
I guess it depends on how well you do on leetcode already. I think most people could probably just do 1-2 leetcode problems a week for a few months to get good at it. I’ve only been asked a few interview questions ever that I thought would be leetcode hard, most are mediums with some easies thrown in. I don’t think you would ever be asked to reimplement a heap but definitely could be asked a question where you need to use one. And usually if you are asked >3 questions you can not entirely solve one and still advance