Comment by compiler-guy | Hacker News Reader

Comment by compiler-guy | Hacker News Reader

Parent Full thread

Of course it does. Pretty much every FAANG, and many smaller ones, have teams for every programming language they consider important. Google has C++, Rust, Python, Java, JavaScript, and others (although I've heard that several of these teams were hard it in this round.)

To take a basic example: Google has a team that works inside LLVM and Clang to improve the code it produces for Google's specific workloads. (Often these help all of LLVM's clients.) But if this team is able to make a search query faster by even 0.01% a year, at Google scale that saves literally millions of hours in compute across Google's fleet. And millions of hours of compute is millions of dollars in power, space, and need to expand.

This is true of all the other language teams at Google too.

The Python team builds a better python by fixing upstream bugs, reducing the memory it consumes and so on. It upgrades the internal installation of Python to the newest versions, ensuring that the upgrade is smooth and doesn' break the using teams.

It contributes proposals for language features to upstream, ensuring that Google's use cases are at least considered.

in addition to contributing to upstream python, we

* maintained a stable version of python within google, and made sure that everything in the monorepo worked with it. in my time on the team we moved from 2.7 to 3.6, then incrementally to 3.11, each update taking months to over a year because the rule at google is if you check any code in, you are responsible for every single breakage it causes

* maintained tools to keep thousands of third party packages constantly updated from their open source versions, with patch queues for the ones that needed google-specific changes

* had highly customised versions of tools like pylint and black, targeted to google's style guide and overall codebase

* contributed to pybind11, and maintained tools for c++ integration

* developed and maintained build system rules for python, including a large effort to move python rules to pure starlark code rather than having them entangled in the blaze/bazel core engine

* developed and maintained a typechecker (pytype) that would do inference on code without type annotations, and work over very large projects with a one-file-at-a-time architecture (this was my primary job at google, ama)

* performed automated refactorings across hundreds of millions of lines of code

and that was just the dev portion of our jobs. we also acted as a help desk of sorts for python users at google, helping troubleshoot tricky issues, and point newcomers in the right direction. plus we worked with a lot of other teams, including the machine learning and AI teams, the colaboratory and IDE teams, teams like protobuf that integrated with and generated python bindings, teams like google cloud who wanted to offer python runtimes to their customers, teams like youtube who had an unusually large system built in python and needed to do extraordinary things to keep it performant and maintainable.

and we did all this for years with fewer than 10 people, most of whom loved the work and the team so much that we just stayed on it for years. also, despite the understaffing, we had managers who were extremely good about maintaining work/life balance and the "marathon, not sprint" approach to work. as i said in another comment, it's the best job i've ever had, and i'll miss it deeply.

That is a lot of great work. Sorry to hear that it is gone. Take some time to rest and hope find somewhere with better vision!

When was pytype invoked? Runtime? CI/CD? Did it output types to be added to files or did it infer and then enable downstream tasks based on those inferences?

it was invoked manually at build time, and then at CI/CD time (there was no runtime type checking). we did output types to pyi files (the python equivalent of C header files), but mainly we used the inferred types to check the code for type correctness - one of the unique selling points of pytype, and why google was developing its own typechecker in the first place, was that the other available python type checkers would check annotated code for consistency, but would not do anything about type errors in unannotated code. if pytype did detect type errors, the code failed CI and had to be fixed before it was checked in.

we did also have a separate tool that could merge the inferred types back into the source code as annotations, but it never got much uptake.

> the rule at google is if you check any code in, you are responsible for every single breakage it causes

i can no longer edit my post to clarify this, but by "responsible for breakages" i meant that if your new check-in caused any CI tests anywhere within the codebase to fail, even if due to bugs in the other code, you had to stop and fix it, or get the owners to fix it, or find some principled way to temporarily disable those tests, before you could check your code in.

this was a very real issue for things like the python runtime or widely used code checkers like pytype or pylint, because if e.g. the new version of python, or some improved check in pytype, started raising failures in a code pattern that was technically wrong but which the existing toolchain did not complain about, you could not release the new version until you had fixed all those new breakages.

contrast this with non-monorepo codebases, where the other code would have had to deal with the fact that "oh, our code works under 3.10 but 3.11 broke it, guess we have to pin our own repo to 3.10 until we fix it", but as the python team we could just say "we support 3.11 now, you need to catch up"

> i can no longer edit my post to clarify this, but by "responsible for breakages" i meant that if your new check-in caused any CI tests anywhere within the codebase to fail, even if due to bugs in the other code, you had to stop and fix it, or get the owners to fix it, or find some principled way to temporarily disable those tests, before you could check your code

Google engineers hate this one weird hack: just send a mail saying "we're deleting/deprecating [everything your product depends on] in [14-180] days, good luck!"

this reminds me of kenton v's complaint about working on infra being so hard

I'm guessing these are related. If every team is potentially responsible for breakages in every product, there's no such thing as a "just chugging along" product and there will be a constant demand to delete products that are not popular within Google.

It has nothing to do with popularity, it has everything to do with teams doing what get them promoted and not having to deal with cross org fallout.

At Microsoft, if there were 12 teams depending on service foo, the team who owned foo had to have a deprecation plan that included sufficient time and support for all 12 teams to get off foo, and directors/VPs would meet and come to terms before the deprecation started.

Google is the opposite. Despite an increasing trend towards top-down direction there's no requirement or support for involving stakeholders. There can be internal products (let's say a data storage solution) and the team who owns it can say "we don't care about this anymore, it's being shut down in 6 months" even if dozens of other teams and millions of end users rely on it, and it's up to those teams to scramble, put other projects on hold, and try to migrate as fast as possible to avoid an outage.

It's frankly demotivating. For the last 18 months, a huge percentage of eng time on my team has gone to either compliance or mandatory migration work i.e. stuff that ABSOLUTE BEST CASE customers don't notice.

Don't forget to include the standard detailed changelog: "bugfixes and performance improvements".

Can you say more about the automated refactoring? How did it work? How was it triggered? How much human involvement was there? Are any parts of it public?

can't say too much about it because it mostly used a lot of internal tools, but it was a combination of writing per-file utilities to do specific code transforms (think something like libcst), and then a framework to scan the google codebase, find affected files, and apply the transforms to them.

Code mods are fun. Were they gated by human PR reviews, perhaps in batches, or applied without review?

yes, gated by human reviews based on who owned the code, unless they were completely trivial and then were gated by a single global approver. there were always humans in the loop though.

Ah makes sense. At another MAANG, there was a code mod template job with a half dozen subvariants. I recall using a variant that supported ripgrep piping to sed -i for a simple replacement mod. There were jobs properties controlling how many mods per diff and whether it needed human approval or not. I think I did one batched by 1000 running once every 2 weeks to keep noise down.

  > very large projects with a one-file-at-a-time architecture (this was my primary job at google, ama)

Is there any place where we could see such tools? A lot of the scripts that I write are contained in a single file, and I'm sure that there would be much to learn from seeing such project scripts. Thank you.

it's open source! check out https://github.com/google/pytype and https://github.com/google/pytype/blob/main/docs/developers/t... for more on the multi-file runner

Terrific! Thank you.

I’m so sorry.

When Wall St. concerns lead to imposing cannibalistic capitalism plans that are pennywise but pound foolish.

Just wait a few years. When they come calling for consulting services, charge them triple.

Frankly, it sounds like a dream job for me too, to be responsible for a particular technology and make sure that it works for all your internal (and external) users.

agreed! it takes a certain kind of person to find that fun, but the people who enjoy it really enjoy it.

I was your opposite number at AWS for many years. Like you say, it was one of the best jobs I've ever had, but over there it was a career dead end. I wasn't laid off, but I had to leave it to have any hope of growing myself.

I feel for ya, zem; if you ever turn up at a PyCon in person, lemme buy you a drink.

I guess it was in some ways a career dead end at google too, I joined as a senior engineer and never bothered applying for promotion to staff, and most of my teammates were similar. On the other hand, I never felt I wasn't growing myself because I learnt a lot, took on some very ambitious projects, and was trusted with a lot of ownership and input into "what do you feel the most effective thing you could be working on right now is?"

Adding to the wonderful writeup by my now-ex teammate (thanks!):

Several of us were/are/TBD also involved in both long term strategic leadership and maintenance of the open source CPython project itself. That direct feedback line from a major diverse needs user into the project and ecosystem was valuable for the world.

The reason I stayed on this team for 12+ years is as zem said. It was an ideal impactful alignment of people, abilities, priorities, and work life balance. My prior teams at Google... were often not.

For the first half of our Python teams existence, there were only ~5 of us. Many early years were spent paying down internal tech debt accumulated from prior years of neglecting to have a strong Python strategy and letting too many do their own thing. Python was one of the very first languages used widely at Google. It was the last major backend language to get a language team.

Signed, -- the now-ex runtimes TL

yes, i definitely should have said more about the cpython leadership work! i was responding to a comment that had already mentioned upstream contributions so i didn't think to highlight it more, but it was a huge contribution to both google and to cpython at large.

well, why don't you guys stay together and continue work? start your own thing! google fired you, yes, but you're still alive and able to communicate. maybe you won't get paid as much as you did at google. but prove to them that you have something of value to provide to society.

Any guess how many SWE-yrs are now going to be spent migrating existing Python projects away from Python, now that Python seems to be unsupported at Google going forward?

I feel your pain. All the best!

How can someone hire the whole team? :)

sorry to hear about the situation at google, would love to help out. we connect laid-off engineers to startups. if anyone from the team is interested here is our discord server https://discord.com/invite/Bnp5zyc8nM

My guess: the people in your group have very high salaries and they want to dump them. I would not be surprised if they hire people to make a new team at some point in the future.

I run HR at a small software company reports and my HR person has a lot of friends in HR at FAANGs. For 10+ years we've been talking about hiring and salaries at Google (and others). It's just crazy to me how high salaries have gotten. I know someone that switched from Google to FB when they made $750k at Google and got $1M at FB.

I said all this to say: I've talked to a bunch of people over the last 6 months that know someone laid off and: 1) they had a very high salary, and 2) the group they were in was hiring.

We are hiring here at Cruise and I believe your team could be a good fit for one of the roles that we are hiring for.

Here is the job description: https://www.getcruise.com/careers/jobs/2831153/

If any of you are interested, feel free to directly reach out to the recruiter Connor Anderson (Connor.Anderson@getcruise.com) or feel free to apply directly.

Only issue I can immediately see is that the salary is about a quarter of the FANG rate.

I mean, that probably at least partly explains why they were laid off..

Please keep in mind that the listed salary range on the job description is only the median base salary range. Not the full total compensation range that includes bonus and other factors to it. We can pay much more than what is listed there from a total compensation standpoint.

We're hiring! https://boards.greenhouse.io/blackcanyonconsulting/jobs/5754... Good life work balance

> maintained tools to keep thousands of third party packages constantly updated from their open source versions.

Hi Zem could you say more about how you maintained those thousands of third party packages? We are facing the same issue and would love to hear your insight!

sure! it used a collection of tools, the main ones were

* copybara [https://github.com/google/copybara] to mirror between github and our internal repo and transform the source tree layout to our internal conventions

* mercurial patch queues [https://hgbook.red-bean.com/read/managing-change-with-mercur...] to apply internal changes

there are internal tools to automate the copybara configuration for python packages specifically, and to map between mercurial and google's version control system, but that's the main idea, and a specific review process for adding new packages.

the process is actually documented here though not all the referenced tools are available (or even applicable) externally: https://opensource.google/documentation/reference/thirdparty...

> needed to do extraordinary things to keep it performant and maintainable.

This is very interesting. Can you share more about this?

i can't, unfortunately, it was youtube internal code. but by the same token a lot of the performance stuff was specific optimisations for that code and would not really generalise.

one of the current "extremely large python codebase" projects is instagram, and they do have some public repos, notably monkeytype (https://github.com/Instagram/MonkeyType) which youtube did have its own analogue of, and cinder (https://engineering.fb.com/2022/05/02/open-source/cinder-jit...). in general facebook's engineering blog is a great place to read about this sort of thing.

Reply on news.ycombinator.com