Building Uber’s Go Monorepo with Bazel | Hacker News Reader

Building Uber’s Go Monorepo with Bazel

eng.uber.com

158 pointsdayanruben138 comments

Might make sense if you use Go or any other language with a straightforward tooling landscape (compilers, package management etc.).

That changes drastically if your codebase contains a lot of C++ and your SW model doesn't quite match the way Bazel tries to push you into. For instance doing any of the following things will quickly turn into a nightmare when using the Bazel C++ rules:

* dynamically linking libraries

* using a containerized approach for library includes (so transitive relative include paths)

* using different toolchains and cross compiling

* interfacing with thirdparty libraries

We are talking seasoned build engineers ending up frustrated after literally months of trying to achieve something that is easy as pie in CMake.

In addition there is still no real IDE support. The CLion plugin is permanently broken and lags behind versions. No real VSCode support. Using custom rules makes this even worse, to the point where CLion will refuse to sync and not having a way to produce a compilation database json.

There are so many bugs open on those projects and no progress or answers. I can not recommend Bazel if C++ or C is what you care about.

FWIW, Uber ATG uses the same monorepo culture with Bazel. It has for some time now. And most of the stuff is written in C++.

> We are talking seasoned build engineers ending up frustrated after literally months of trying to achieve something that is easy as pie in CMake.

But it's not the same as CMake. A properly set up Bazel project gives you so much more: organization-wide incremental builds, build cache and build farm. Also, actual full hermeticity of builds (no, taking ambient deps from a Docker container doesn't replace that).

Comparing CMake to Bazel is like comparing some barely-working bash scripts on a single box to a kubernetes deployment. Maybe you're okay with just bash scripts, but some of us aren't, and that's where Bazel comes in.

Completely agree that you can't compare Bazel and CMake directly, they are not on the same level. But for compiling C++ Bazel will have to compare itself to CMake, which is plain and simple the defacto standard.

All the complex things that Bazel brings into the game (distributed builds and caches primarily) simply don't have any value when the main task that you're trying to make faster is not done well.

I would much rather like to see more tooling that does these things but builds upon layers of battle-tested existing software.

I've been confused for a while about the common claim that Bazel gives "full hermeticity" of builds -- it doesn't seem to be true in practice (at least for packages with system dependencies). Maybe you can help me clear it up.

E.g. Google's protobuf libraries [1] can be built with Bazel, and they depend heavily on system headers outside the repository, e.g. <iostream> and <stdio.h> and lots more. If those headers subsequently change, Bazel will not pick up on this and will not know to rebuild the parts of the build that depend on them.

To reproduce: run `bazel build -c opt //:protoc_lib` and then put random garbage in your /usr/include/stdio.h and /usr/include/c++/<version>/iostream and then rerun the bazel command -- it will not know to invalidate the build cache. If you `bazel clean` and then build again, you'll get different results.

Bazel does a lot of really nice things and I can believe that within a google3-like environment (where the source code never references a system header?), it effectively provides hermetic builds, but in practice as used outside Google (or even in Google's public OSS releases) it doesn't seem to really match this description or enforce a hermetic seal. What am I missing?

[1] https://github.com/protocolbuffers/protobuf

I wish Bazel was more explicit about things leaking in from outside the workspace, but you can always vendor the toolchain to be more hermetic. For example see https://github.com/grailbio/bazel-toolchain which sets up LLVM and Clang for you.

https://github.com/bazelbuild/bazel/issues/4558

Ok, but that decision would have to be made by the project maintainer, in this case Google, not the person using Bazel to compile protobuf. (And not particular to Bazel -- a developer can make any build system effectively hermetic by vendoring everything.)

In my view the challenge here is that a dependency changes (e.g. /usr/include/stdio.h is upgraded by the system package manager, or two users sharing a cache have different versions of a system library) and Bazel doesn't realize that it needs to rebuild. It would be a pretty heavy hammer if the way to fix that requires every OSS project to include the whole user-space OS (C++ compiler, system headers, libraries) in the repo or via submodule and then be careful that no include path or any part of the build system accidentally references any header or library outside the repository.

And maybe this issue just doesn't need to be fixed (it's not like automake produces build rules that explicitly depend on system headers either!) -- my quibble was with the notion that Bazel, unlike CMake or whatever, provides fully hermetic builds, or tracks dependencies carefully enough to provide an organization-wide build cache across diversely configured/upgraded systems.

> Ok, but that decision would have to be made by the project maintainer, in this case Google, not the person using Bazel to compile protobuf.

No, that's up to the user. If you include protobuf in your Bazel workspace and have a toolchain configured, protobuf will be built with that toolchain (this is also how you would cross-compile). Bazel workspaces are still a little rough around the edges and interactions between them can be very confusing, as Google doesn't need them internally (everything is vendored as part of the same tree).

Under the hood there's a default auto-configured toolchain that finds whatever is installed locally in the system. Since it has no way of knowing what files an arbitrary "cc" might depend on, you lose hermeticity by using it.

Of course, the whole model falls apart if you want to depend on things outside of Bazel's control. However in practice I've found that writing Bazel build files for third-party dependencies isn't as hard as it seems, and provides tons of benefits as parent mentioned.

For example, last week I was tracking down memory corruption in an external dependency. With a simple `bazel run --config asan -c dbg //foo`, I had foo and all of its transitive dependencies re-compiled and running under AddressSanitizer within minutes. How long would that take with other build systems? Probably a solid week.

The best way to get your hermetic builds in order is to _always_ use remote execution and ensure that your executor nodes don't have compilers and headers just laying around. Force yourself to get the toolchain under bazel's control.

I think the reason the public finds this so mysterious is the documentation for CROSSTOOL is terrible, and virtually all of the people who learned how to use blaze at Google go out in the industry with no understanding at all of CROSSTOOL, because there's a small dedicated team who maintain it.

Does your company use remote execution across the board with Bazel? How much effort was it?

I work for no man, as a character in a movie once said. But I recently went through this exercise just to make sure I could do it, and it's easy enough to put a bare-bones executor node in the cloud and use it temporarily. You can even get ARM nodes, to make sure you aren't implying the target architecture.

Doing it on a company scale is probably harder. Last place I worked used bazel but did not bother with remote.

I think the biggest thing to remember is that changing your build tool for a large project is going to be a massive effort, you're gonna need a LOT of hours to throw at it, and you might not find many benefits for a long time.

Starting a new project with it, so it can help you do things the "right way" is a much more attractive proposition.

That said, the IDE/tooling situation for Bazel was disappointing to me in the Java world too. It definitely lags and is incomplete.

I was keeping tabs on the Python plugin for a couple of years (but stopped checking in about a year ago) and Python3 was advertised as "supported" the whole time, but no one was able to get it working. There were a number of open issues, none with any action on them.

And while Nix has an insane learning curve, figuring out how to extend Bazel (and consequently build one's own Python plugin) was a nightmare. The documentation was unhelpful/incomplete and the codebase was incomprehensible and poorly organized, with inheritance sprinkled on everything for its own sake (which is to say, par for the course for Java apps).

I know... "Don't look a gift horse in the mouth! / Don't complain about open source software!" Fair enough, but I don't have to use it, and I'm free to caution others not to as well.

One of my side projects was setting up a K8s cluster that ran Hydra in containers, used Hydra to build packages, and Kubenix to build and deploy services to the cluster. It was a little closed loop.. my goal was to have it building and deploying itself fully.

Unfortunately Kubenix was too immature, my knowledge of Nix insufficient, and Hydra required privileged containers due to Nix build sandboxing.

Have you used nix as a primary build system rather than a meta build system? If it works, I might have the energy to revive my project.

I'm not sure what you mean by "a meta build system" vs "a primary build system", but for many years Nix was what our company used to build and package our software. Unfortunately, it was a very bad experience; Nix doesn't strive to be a build system, it strives to be a package manager. The learning curve for writing packages is too steep to expect every engineer to learn it, and when things broke it was often hours and hours debugging into some obscure C library way down the dependency tree that no one in your organization has ever heard of. Worse, very little is documented well or accurately. Trying to figure out, for example, how Nix's python packaging libraries work is frustrating--partly because there is no obvious way to find the defining file for a given package dependency and partly because the package definitions in nixpkgs are rarely documented. These are all tractable problems, but Nix doesn't (seem to) aspire to solve them.

CMake is a natural fit for C/C++ (even though I strongly dislike the syntax). It has a good IDE project generation, and you can use ninja to build super fast. But it isn’t pre-configured for hermetic builds out of the box, remote, team-wide builds and caching are much further than a command line option away.

You can use sccache with Cmake easily. On mobile so I can only give you breadcrumbs.

1. Set up sccache backing server (I use docker redis) 2. Export sccache config. Set SCCACHE_REDIS to a Redis url in format redis://[:<passwd>@]<hostname>[:port][/<db>]

Set CMAKE_<LANG>_COMPILER_LAUNCHER (cc, cxx) to sccache.

> There are so many bugs open on those projects and no progress or answers. I can not recommend Bazel if C++ or C is what you care about.

I think this is something that's slowly, but surely, changing. There's a lot of adoption of Bazel from some big players out there (Uber, Bloomberg, etc). The Blaze team @ Google has also started incorporating outside OSS maintainers into their OSS software (specifically rules_python).

Things are improving. Many of the things you've addressed (third party libraries, cross compiling, dynamically linking things) have been talked about by many people in Bazel's slack. It might not be flawless currently but Bazel's abstractions of how builds work are cool, RBE/caching is amazing for large projects, and a single build system with the promise of gracefully handling "every" language is something our field has been needing for some time.

In 3 to 10 years I can imagine being able to start writing code and never think about which language some library I want to use is written in, which package manager and toolchain I need to use to build everything, how to package my code to be deployable in production, how to setup unit tests, how to get autocompltions and docs and everything in my IDE, etc. The answer could just be "bazel, a language server, an IDE that speaks lsp".

Bazel has extensive support for cross compilation. You can find pre-made toolchains. Intefacing with third party libs is as easy as writing a BUILD file.

I've made a similar post about building a Go monorepo using Bazel two years ago, check it out - https://filipnikolovski.com/posts/managing-go-monorepo-with-....

We've been using Bazel not just to build and test our Go apps, but to build docker images, compile proto files, even deploy to k8s. It's really versatile and the developers only need to know one tool to build and test the whole environment.

We have been Go with Bazel for the last 2 1/2 years to build most of our backends, but also on mobile (gomobile) as well. It's not an easy tool, but it delivers amazingly and it's very, very reliable.

Haven't run a bazel clean in... 6 months maybe?

The multi-language really is a killer feature when the project inevitably becomes polyglot, such as when doing protobuf/gRPC or using CGo.

I wish they had spent more time describing how/why Go's built-in build tool stopped working. Or perhaps that's part of the basics of Bazel. Anyone able to share more on that?

The Go tool builds Go on a package by package basis. It won't help you with anything else. As soon as you need to build anything that isn't Go, or you need to do some pre/post-flight work, or you need to push artifacts etc etc etc the Go tool can't help.

As the blog post notes, Makefiles are the usual weapon of choice for Go projects of any sufficient complexity, but these are quickly outgrown.

Honestly the main reason Bazel is useful in a monorepo configuration is simply because it’s designed for it. It supports cross-language dependencies and generated files, and is designed for large repositories with many nested targets. Go’s build system is fine, but things like Go generate vs genrule probably come into play. Bazel also offers target visibility, to protect targets from being depended on in unintended ways, and build isolation, to allow builds to be more predictable (and help enforce target visibility.) It also has a few convenience features for vendoring 3rd party libraries and handling their licenses.

Of course Bazel is far from perfect, in fact sometimes you may be better off running your own rules instead of the official ones in some cases, but IMO it makes a pretty decent build system for putting all of your code in one place.

On the other hand, it’s one of those Google things where if you haven’t seen it in action in a functional configuration it’s hard to explain why it’s actually nice. And sadly I feel unsatisfied with the Node.JS rules for example, which is probably how a lot of people will first experience Bazel (since I believe Angular supports Bazel this way.)

As bazel is not limited to Go it can be nicer when the Go programs or libraries are part of a more diverse system. For example, you can build a Go library, along with the rules to generate a protocol buffer package, along with the C library that the Go library interfaces with, along with some static assets that you want to compile into your program all from the same BUILD file, and are built from a single command. I think to do at least some of these things, if they are possible with the go tool at all, you would have to run 'go generate' first.

Bazel also come with a query tool that makes it easy to tell what other targets your package depends on. Go also has such a tool, but again it's limited to Go dependencies.

I believe Uber has been using bazel for awhile now, and probably were invested in it well before the Go module cache was a thing, so Bazel's build cache is an important optimization.

The problem with Go's builtin toolchain that I've run into is dealing with anything that's not Go. For example, compiling protobuffers into go code. Go won't help you version the proto compiler, the proto standard library, etc. The result is that if left to its own devices everyone on the project ends up generating a completely different proto. If you check in the generated files, this is only a problem when someone wants to update the protos, but when certain files are harder to edit than others, people start coming up with workarounds that involve not editing them -- and the workarounds are scary. Any project where I use protos, I use Bazel because while it's much harder to get going, you are pretty much guaranteed that anyone who checks out the project and makes a change will get a binary with that change.

That said, I do agree with other posters that Bazel + go feels like nobody's priority. I use gopls and love it very much, but it doesn't understand how to ask Bazel where files are to analyze. A bug has been open for years, lots of people saying "I'll put it on my OKRs this quarter", and it's just not getting done. I am close to just fixing it myself.

But I have a feeling that this happens to everyone. I have worked on other Bazel projects and they all have detailed setup instructions; for example, Envoy uses Bazel but somehow shells out to cmake, so you have to have the right version of cmake, the right version of clang, etc. to produce a build. To me that defeats the purpose of using Bazel; Bazel should build the toolchain and all dependencies for you, otherwise what's the point? All you should need is Bazel and the source code. But that is not how people are using it. I don't know what the canonical successful Bazel project is, but I haven't found one yet.

I am currently working on a Typescript/Go project with gRPC/Web and will see if Bazel gets me that "one command full build" dream. I am somewhat motivated to make it work, but I have a feeling that there will be warts that makes me regret it. I used Bazel (well, the internal version) when I was at Google and every time I used it it was a joy. Incremental builds where you edited one file and wanted to run the tests were very fast, and builds where you modified some core library and wanted to see the impact across all the affected projects were also fast (easy to use 100s of CPU hours in just a few minutes of wall clock time).

Ultimately I think Bazel is the right approach, but I think to be successful you have to commit to having one engineer work on tooling full time. At your startup, probably not feasible. At Uber, probably very feasible.

(Going off topic, I would love to see what other people are doing for polyglot projects, because I feel like everything is a polyglot project these days. I have never seen a setup I liked -- install one program, check out the code, have a guaranteed working build that is bit-identical to the production image. The fact that nobody does this scares me deeply. But that's where we seem to be as a field.

I am also interested in seeing people's templates for projects. For example, I would like to do a C++ microcontroller project, and want to see how people are building those. I need clangd support, I need to use libraries, and everything I've seen is "hack together a makefile and cross your fingers". It just scares me.)

I wish my company was at the scale of Uber to work on all these kinds of tech. Makes me want to leave my current job just to go to Uber.

Coordinating this sort of change in a large org like Uber is not entirely "hacking on cool tech". It's a lot of communication and writing documents and going to meetings

This. It's very easy to forget about the political processes involved in making projects like this happen. Also, getting a job at a company as big as Uber in no way guarantees you'll be able to work on tech like this. You'd need to make sure you're joining a team close to this project, and if the culture of team turns out to be toxic, the fun of working on "cool" tech fades really, really fast.

> Coordinating this sort of change in a large org like Uber is not entirely "hacking on cool tech". It's a lot of communication and writing documents and going to meetings

And a lot of tearing your hair out when trying to use tools in the ways that they weren't intended to be used.

Of all languages, Go is not the language where you want to be doing things against the grain (that is, against the way the language designers intend it to be used). Some people like that about Go, and some people don't. Either way, Go is a very opinionated language with a very opinionated ecosystem, and ignoring those is a recipe for frustration.

Last time I checked, Bazel was not recommended by the Go developers - and with good reason: there are a lot of gaps in rules_go/Gazelle, which this blog post alludes to but glosses over. While Google uses Blaze internally, Blaze is not Bazel, and the differences are very apparent to anyone who has used both to build Go specifically. Furthermore, rules_go was developed entirely independently of the Blaze ruleset that Google uses internally, so it's a tool that's not actually used internally at Google, but also not used by the majority of the non-Google Go developer community either, in addition to not being recommended by the Go team at Google[0].

[0] There is exactly one mention of Bazel on the entire golang.org domain - in a changelog from over two years ago, in an /x/ package, where Bazel is mentioned as one of two build systems in a "such as" clause that the new package could potentially enable support for (/x/ packages are considered experimental and not subject to the same backwards compatibility or maintenance guarantees as the rest of the project).

From the issues I've read on github, anecdotally, is that the approach of the gazelle and rules_go teams is to leverage the standard go tooling and not make changes to go itself.

Also anecdotally, bazel and golang work really well together IME. The community seems pretty active, and the upsides of using gazelle/bazel with golang seem to outweigh any downsides (though I'd be hard pressed to name a downside, that isn't inherit to golang itself).

> Also anecdotally, bazel and golang work really well together IME... and the upsides of using gazelle/bazel with golang seem to outweigh any downsides

This is really not my experience from having used Bazel with Go for the last four years. But I'm happy you are are apparently not running into issues.

I've been using about the same amount of time. Have you tried using python and bazel? Now there's some real weeping and gnashing of teeth ;) Lot's of bad code depending on python == python2 type of non-sense.

Javascript is another target that is just about as gnarly. The node module resolution algorithm is reimplemented a million similar-but-not-quite-the-same-ways by popular packages, most of which struggle dealing w/ symlink-heavy codebases (like bazel sandboxes).

Having used both for years I honestly have not perceived the difference between blaze and bazel go_binary/go_library/go_test. What's the big difference?

> Either way, Go is a very opinionated language with a very opinionated ecosystem

except when it comes to dependency management

We're not at the scale of Uber yet, but at Jupiter (YC S19) we're using tech like Bazel, protos, and Kotlin to prepare us for when we are: https://starship.jupiter.co/jupiter-stack/

We're actively hiring for a senior SWE right now, so feel free to shoot me a note if you're looking.

I'm not familiar with the Kotlin build chain, but what role does Bazel play? Is it needed for integration with the protocol buffers?

Yes - Bazel has excellent protobuf integration compared to other build tools (we were originally using Gradle). We're compiling Kotlin to the JVM, so the build chain is similar to Java's.

Gradle has a great protobuf plugin, what is so not excellent compared to Bazel?

Protos is more of a faster & smaller network serialization tradeoff vs. ease of development thing than a company scale thing. And a thing to do if you want to integrate with gRPC.

I worked at a company that used protos from day one for example.

Many large companies are still using JSON w/ schemas as their network serialization layer just fine.

I wish they weren't morally bankrupt as I would have loved to work on their tech stack.

> morally bankrupt

How so, in 2020?

I think once you're stealing critics' medical records, your morally bankruptcy looks more like a chapter 7 than an 11. There's no expiry date; raze the whole thing.

Not just large companies - plenty of small startups building cool developer tooling. And much more fun when you get to start on a green field, rather than slowly migrating the existing code over the course of a decade.

More politics, then tech.

Not personal experience, but I’ve had several coworkers who worked at Uber, and they all gave me the same story. This is the story:

Uber reinvented a ton of technology because of the idea that off-the-shelf solutions wouldn’t work at “Uber scale”. However, there was very little accountability for whether the in-house solutions were necessary, and working on these tools would get you promoted. So for every “Uber scale” problem that a team actually solved, there were a couple other projects that were just half-baked alternatives to the off-the-shelf software that they should be using.

It turns out that “Uber scale” is not really that large, despite the name. But engineers kept repeating “Uber scale” and building infrastructure.

The same problem occurs at the larger tech companies like Google, Facebook, Microsoft, Amazon, and Apple, but in different ways and to different degrees. And to a large extent, engineers are copying what other companies do, and bringing ideas from one company to another when they hang out after work or switch jobs. For example, you can bet that these companies mostly have their own containerization and scheduling systems, many of which are undoubtedly not competitive with Docker or K8s in 2020, but K8s only goes back to 2014 and all these companies are older. I’m sure Borg and Tupperware are great if you work at Google or Facebook but I’m also sure that they’re missing a bunch of tooling that you’re used to. Same thing with build systems. Bazel, Buck, Pants, Please, and that Frankensteined system that Chrome uses are all copies of each other but Bazel is the only one with a decent size community and ecosystem, as far as I can tell.

Uber is absolutely not on the scale of those companies, and most of the time they should probably be using off-the-shelf solutions when they become available.

- nothing wrong with NIH syndrome if you have the means and resources to refine existing ideas and rebuild them with scale in mind

- uber is a 50 billion dollar company and they have to de-risk themselves by owning entire stacks, top to bottom. if it means re-creating something from scratch... who cares, they have billions.

Or, at that point, you use industry standard solutions and hire the best subject matter experts on that solution. Like, for example, using Cockroach DB and hire one of the core committers cause you're big enough. It's what MSFT did with things like the Python extension for VSCode and DBOX did by hiring Guido since they're a Python shop.

I still shake my head whenever I hear about how Uber built an entire Slack clone for "Uber-scale".

While I understand the point that they have money to burn, there are better ways to spend that money. Some items like “databases” are core to their SLA that it makes sense to invest. Build, source code management, etc are not “critical to de-risk”. It is not like Microsoft will one day refuse to sell them GitHub enterprise or GitHub actions because some division of Microsoft is competing with them.

That’s just apologia for NIH syndrome. It’s fine to reinvent software if that serves your goals. It’s short-sighted to reinvent software without due consideration for whether off-the-shelf solutions work. That’s not de-risking, that’s adding risk and slowing development. You see it happen at companies like that and usually it’s a sign that there’s something wrong with the culture, incentives, and the way that promotions work.

My take on it is that the engineers want to make complicated solutions to hard problems to justify their salaries and get promoted, and that managers encourage that behavior so they can defend their headcount. I’m not accusing any of these people of acting in bad faith here—nobody’s reinventing tech to sabotage the company, it’s just that the system encourages this kind of behavior.

This problem is not unique to Uber, you’ll see similar things happen across the industry to different degrees.

Well, the "due consideration" needs to duly consider the actual costs and benefits of the off-the-shelf software, not make the kind of unsupported blanket claims about its obvious superiority, like you just did. Every off-the-shelf program has maintenance costs, flaws, and weak fitness for purpose. You just made a sweeping and italicized claim that all off-the-shelf software is both less risky and faster to develop than house-made software. That's a dangerous philosophy and probably isn't true for every organization and application.

> …not make the kind of unsupported blanket claims about its obvious superiority, like you just did…

Not what I said. Let’s move on.

> That's a dangerous philosophy…

Go pick a fight with someone else.

I work at a huge non-tech company (bank), and we are guilty of the same. Tons of stuff built in-house for stupid reasons, including an entire buggy IDE that people curse every day.

I have not had the privilege of working for a "tech company" where tech and tech employees are first class citizens. I'm pretty certain the in-house tooling would be much better at a company like Uber or FAANG than a company like mine.

I interned at a FAANG last summer (Google) and I was blown away at how good the in-house tooling was (including the in-house IDE).

Everything was seamless - I'd log in, tap my security key, and instantly have access to pretty much the entire monorepo and the rest of the production + deploy systems. All of the internal tools integrated with each other -- for instance, I could create a CL (i.e. a pull request) and then fix issues raised by the CI system, entirely from the IDE.

"Owning the stack" completely in-house also extended to hardware -- all of my builds happened inside Google datacenters too (not on my machine), and the development box they provided me was a Chromebook.

I’ve heard similar stories about working at Sun back in the day. It was magical, supposedly.

Sun didn't have a monorepo -- at least not 1998 through the acquisition. Each org of a given size (Solaris os/net, install, etc) had its own separate island of source with its own build and test systems.

I guess my comment wasn’t clear. Talking about how you could just grab a random terminal with your smart card at Sun, and get work done. True at Google, true at Sun back in the day, not true at most places.

Yeah, stuff like this is one of my chief reasons I want to start working at a "top tech company". Just an overall higher level of tech competency.

You don't want to?

Typo, lol. Should have been "start". Or I may have been typing "stop working at a non-tech company" before my train of thought ran away.

This is in contrast to Apple's internal software systems, at least if recent reports seem to be accurate; aside from (I assume Xcode), it seems the business-ey enterprise stuff is outsourced to contractors and software consulting firms and is a giant mess...

Granted, their DNA is still in hardware/devices/OS level stuff versus "services" despite all they claim to be, but how much more quickly could they move if they "owned more" of their stack?

Ah, I remember when the in-house IDE first came out, and it was considered a toy compared to vim/emacs. It has come a long way since then.

That said, I'm also looking forward to "VSCode front-end in the browser" becoming ubiquitous. I remember using Visual Studio back in my first job in high school, and Intellisense made coding so much more... explorable and approachable.

Nice to see a comment like this, I feel the same way. Google is the exception to all these other comments. The tools available to Google engineers make working outside Google feel painful and inefficient by comparison even with their unfettered access to all the third-party tooling.

It’s so good that it would actually be a factor if I were ever considering working somewhere else. Getting to work with Google’s in-house dev tooling is probably worth 15-20k to me. A lot more than 20k if the other company has a reputation for horrible tooling.

A company isn't going to pay you more to be less productive.

To fix the tooling? Maybe..

Which bank if I may ask?

Also worked at one that developed a ton of tools internally, but I wouldn't call many of them terrible.

I'd rather not mention specifics, but suffice it to say it's one of the big US bulge bracket investment banks.

One of the most satisfying moves of my career was moving from a FAANG to a "cloud native" series C startup. The "not built here" syndrome is very real at the FAANG's.

I will defend the FAANGs just a little bit-- at least Amazon, Google and Microsoft-- in that they are now large SAAS providers and so "building it here" is essentially the same as eating your own dogfood, which is very much a good thing.

I would also add that everyone is evaluating the tech stack of these companies in today's context. Google was founded in 1998. Facebook was founded in 2004. Uber was founded in 2009. Microsoft was founded in 1975.

Amazon EC2 was first offered in 2006 and was super new and immature for a number of years. Kubernetes was first released in 2014. Mesos was a research idea for a number of years starting in 2009 and didn't reach version 1 until mid 2016. Etc. Etc.

These companies generally invent their own solutions because the solutions everyone here thinks they should be using didn't exist or were not stable at the time they had a problem they needed to solve.

By the time those new solutions exist and are stable enough, it requires quite a bit of investment to migrate to the newer solution as these companies already have much of their tech stack stable and bringing in revenue on their in-house solutions.

Rebuild $infra from large company and provide it as SaaS was a pretty good recipe for a decent software business for a while. It might still be. For example, PagerDuty came out of monitoring systems at Amazon.

I feel like what's pointed out in this comment is so obvious that it shouldn't be needed ... but evidently it is.

Sadly, it does need to be pointed out, pretty much every single time. Same with the "Why does company X need so many employees?" question, which you can just reply to with this link: https://danluu.com/sounds-easy/

It's also a risk mitigation strategy. As your company's size grows, so does the liability of your dependencies that you don't control. If you can throw a few engineers at a problem and save yourself tens of millions of dollars in risk, it's worth it.

It’s not risk mitigation, more a risk swap between “unknown risk”—we don’t know if this third-party product is going to meet our future needs—and “known risk”—we have full control while we are willing to write checks to fund it.

In the latter case there’s still plenty of risk: you might be spending valuable time building something that doesn’t work, is obsoleted by future change or future products; and the one that most people dismiss, you will be at the mercy of its creators. The people who build your critical technologies will have power over the organization you might regret later, and if they do leave, finding replacements can be hard and/or expensive.

Spot on! Never thought about it from this perspective. :-)

The net risk reduction of course depends on specific situations, it is not always better and it is not always a "swap". In some cases, eliminating unknowns reduces more risk than is added by building in-house, which is when it makes sense to build in-house.

In the case of those larger orgs though, were the off the shelf solution was mature enough for them to use at the time?

> Same thing with build systems. Bazel, Buck, Pants, Please,

It's worth noting that none of these were (really) NIH. Blaze wasn't open source originally, and Buck, Pants, and Please were all essentially reimplementations of the closed source Blaze rebuilt by xooglers who went on to work at FB, Twitter, and presumably thought machine, though IDK if it's the same.

Then/concurrently, Google open sourced Bazel, which is mostly-blaze.

That’s absolutely right, but I think it illustrates one of the problems with inventing your own tech stack, is that it might make perfect sense when you start the project, but several years down the road it might just be an inferior version of an open-source project.

My personal sense is that this even applies to stuff like Borg, but who knows?

You think Borg might be an inferior version of Kubernetes?

It will be.

Mind you, Kubernetes was originally made written at Google by engineers that previously worked on Borg. So they will naturally design it to address problems with Borg. Kubernetes’s original project code name was “Seven of Nine”, because Seven of Nine is a friendlier Borg.

Just because someone subsequently writes another program, that does not assure the superiority of the later edition. People who wrote Borg thought Omega was going to be better, but Omega failed except for a few things that got cherrypicked back into Borg.

I particularly can't agree that k8s is a necessarily friendlier Borg. Borg is on rails; there are fewer choices for the user to make. K8s tries to be flexible enough to be used in various production environments, which leads to it being harder to use rather than easier.

Sure, and to be clear, when I say “it will be” that it’s little more than a guess on my part.

Google has enough project churn internally that services and programs do stand a good chance of getting replaced, though. There’s plenty of history.

Some of the flexibility decisions for k8s come from Omega and are mentioned as impossible to backport to Borg.

Sometimes you also get issues on k8s where a google engineer responds "it's not done yet because we really, really don't want to repeat what a fiasco it was under Borg".

So I'd say it might be better even in direct comparison ;)

I imagine part of the perceived usability problem comes from the fact that often users of k8s are also playing the role of borg-sre, whereas 99% of borg users do not need to deal with that.

You can't really predict the future if a large tech company will properly fund an open source project to reproduce what they have internally, and if that will be any good in time, especially if none of it exists. In the mean time, you need a solution to problem X.

I've even written a few libraries like that, and still stuff out there isn't that good or does not exist years later, because it's too niche, yet still a generic issue in the industry. Can't open source it although.

In an alternative timeline, Buck for example could of been the OSS solution that google decided to devote their resources to, and then in the end it would of been a good idea that FB created their own version of it.

To be clear, what I’m saying is that “this is a problem with reinventing stuff”, I’m not saying “you shouldn’t reinvent stuff.”

It seems like you might be under the impression that bazel and blaze are not the same codebase?

If Google made a good build system with features not available in open source, eventually open-sourced it, continue to use it, and lots of other people are eager to use it, how does that demonstrate that they shouldn't have invented a build system?

I guess the comment wasn’t clear enough, because that’s absolutely not what I was saying.

“Problem with X” is not the same thing as “you should not do X”.

I’m also not getting why you think that I’m ignorant of Bazel’s origins, specifically, since I was talking about Borg.

You'd replied to a comment about build systems and bazel.

Regarding Borg vs k8s, it's worth noting that k8s was never built to replace Borg or to even try to match the feature set of Borg. There are some nice things about k8s (like the ease of running an entire cluster on your local machine), but since k8s tries to be all things to all people it's not even close to being viable (or competitive with what Borg can do) for Google itself.

"Uber is absolutely not on the scale of those companies, and most of the time they should probably be using off-the-shelf solutions when they become available."

I agree, but I'd also add that the state of the world in 2016 when most of these infra projects were launched is very different from the state of the world in 2020. Back then, uber was not easily able to run on a bunch of OSS/cloud native solutions, but now for sure its not just possible but likely the best/most responsible way to do it [source: worked there for 4+ years]

Same thing at Postmates

I think Uber's finally recognizing this and has been adopting more off-the-shelf stuff for non-core stuff. The move to slack a few months ago comes to mind.

Hard to say. I'm sure that accounts for some of it, but at a certain scale, having a system be tailor-made to fit your requirements and workflows has a lot of value. And if you tend to be towards the leading edge of scale and tech, the tools that you should be using don't exist yet or aren't known at the time without hindsight. My shop wasn't even that large, and we built our own k8s because we needed it. Looking back historically, it was even available at the time, but definitely not as the robust and popular standard that it is today.

And once you have a whole system built, you're really going to have a hard time justifying re-doing your whole architecture to move to a product that does the same thing. Basically that only happens when you have a new head of engineering who comes in and goes, i don't care how many engineers you have to throw at it, and i don't care how much custom functionality you depend on that you lose, just figure how to replicate everything you do today using k8s.

This is revealed if you go to any of their tech talks too.

That's not to discount the engineers working there, they indeed do have a number of bright engineers. But the no accountability comment is spot on.

Uber Engineering sounds like they are collecting Google's open source tools and name-dropping them in their blogs to market themselves as "cool tech".

One companies boring tool is another companies favourite hyped third-party tech tool. (Until it doesn't work for them)

My experience working at Facebook was that the "working at scale" is nowhere near as glamorous as it sounds, especially when it comes to scaling build systems and monorepos.

As a company, you have to invest immense resources in evolving the infrastructure and it is a never-ending task: the more capacity you build, the more your organization will grow in ways that tax that capacity. The goal posts are ever-receding.

As an individual developer, all that fancy distributed infrastructure becomes a barrier that must be laboriously overcome at every step. Your expectations about what constitutes a "fast" operation get gradually distorted. Working on "small" side-projects (a few thousand files or less) feels effortless in comparison.

Not to say you shouldn't seek out the "big tech" experience; just that it is not all roses.

They just had a laid-off recently, and still not profitable given how much $$$ they have burnt over the years. I am not sure if you wanna work there.

FANG such as FB and Google has some big monorepos, and you can try that if you are interested.

Scale is not a prerequisite, but you do have to sell it to your company. If not a greenfield, porting a codebase over is not easy, and you have to do your organizational excellence homework too. Monorepo, well reviewed code, team not in permanent crunch, high-quality tooling culture.

Just like Kubernetes, Bazel is not a silver bullet. But if you get all your ducks in a row, switching over is a gamechanger and will help you out even at the smallest scale.

You want to join Uber because you think Bazel is cool tech?

Having spent the last few weeks of my life converting our build system to Bazel I guarantee you it's far from what I would consider cool. It's more of a necessary evil because the alternatives are even worse.

> Out-of-the-box software solutions rarely work for a codebase as large and complex as Uber’s Go monorepo.

That's a weird way to put it. Bazel is the open source variant of what Google uses for its monorepo, which is several orders of magnitude larger and arguably more complex.

Point is that take two sufficiently large monorepos & they'll have enough differences between them to require custom work

That Google had to come up with Bazel for themselves only supports that point. It isn't obvious whether it'd then be applicable to other existing cases without customization

Lots of knobs that vary here tho, such as where one draws the line between configuration & customization

The version of Bazel we (Google) uses internally has a lot of features that are not out of the box in open source Bazel. Mostly around distributed compilation or object caching.

The public version also offers remote execution and remote caching.

https://docs.bazel.build/versions/3.1.0/remote-execution.htm... https://docs.bazel.build/versions/3.1.0/remote-caching.html

What's not open-sourced is mostly the interaction with Google internal infrastructure.

This is excellent! The only reason we haven't moved some of our Go codebase to a Bazel monorepo is the IDE integration.

I have been tinkering with a few ideas to make the existing tooling work with Bazel, but the effort is larger than I had originally expected.

If you also want to migrate off of Makefile and also want reproductibile builds, try out Earthly. Normal companies can't do Bazel because it's too alien and requires deep investment.

https://github.com/earthly/earthly

Disclaimer: I am Earthly's creator.

Does anyone have experience using Bazel (or a similar build system) together with create-react-app? Specifically, is there a way do it without ejecting?

I work on a project with a Go backend and a React frontend, and having to update all those React dependencies myself is what keeps me from moving to a system like Bazel.

Super interesting! There is also a video explaining their system that might be worth watching: https://vimeo.com/358691692

Wait, didn't Uber decide that Bazel didn't scale enough (speaking as a xoogler, lol) and made their own build tool? Or am I misremembering?

That seems like a ridiculously complex pipeline to maintain and onboard into. I personally dislike this "do it all" monorepos in most cases, there are cases were they work, but they also break a lot of other things.

I have not seen metris or blogs that prove this "uptick in build efficiency" or an increase productivity.

While I do like the idea behind bazel, I hate repeating deps in things like "go_repository" with gazelle.

go mod vendor && bazel run gazelle

No go_repository duplication.

The article says that monorepos are more efficient but also that "As the monorepo grew, the build target list increased to a point where it became too long to pass it through Bazel’s command line interface.".

Monorepos are not efficient. They are easier to manage when a team is small but as the team grows and you have more and more deliverables with separate versioning you are introducing control structures in your automation. Complexity explodes!

Anyway, all this does matter if you don't make any profit :)

Google is a monorepo as far as I know, and they are doing fine.

Indeed, in their paper from 5 years ago Google claimed 300000 commits per day across 9 million source files, compared to this article claiming 10000 commits per month on 70000 source files at Uber. Whatever the differences are between blaze and bazel, it must be the case that the former can easily scale to this size of repo.

I like to look at Google's GitHub commit messages to get an idea of the pace of their revision history. Yesterday they committed something with a Piper revision of 311324901. A month ago it was 306514102, and a year ago it was 248381230. That's about 160k revision numbers per day.

Google monorepo is a productivity boon.

Disclaimer: ex-googler.

huh. last i checked with the people that work there the tooling google has around managing the repo, build, test etc is insanely good. if you’re thinking you can do monorepos because google think again. you don’t have the tooling and you don’t even know what the tooling is

But the number of targets is a function of _their_ code, not of using a monorepo, no?

If you need to build N libraries and M executables then you are going to have N+M targets (assuming building for one arch only) whether you use a monorepo or not... Either way you are going to issue N+M build commands.

Also, in bazel you can do

  bazel build //...

to do a full build of all targets in a workspace. If they are not doing this but instead passing each target name individually then that probably means that they are only building a subset of everything, and even that subset is too much to pass in a single command line invocation. I'll grant you that it seems excessive to have that many targets, but again I don't see having so many targets as an explicit issue of the monorepo.

That's a "holding it wrong" kind of affair, isn't it? Who expects to be able to "bazel test `shell thing with more than 4MB of output`"? Wouldn't a judicious application of either `...` or `xargs` fix this?

Agreed. Although the number of targets seems excessive tbh, but to each their own.

A monorepo just lets you have commits across projects. It says nothing about how you build things.

>Monorepos are not efficient. They are easier to manage when a team is small but as the team grows and you have more and more deliverables with separate versioning you are introducing control structures in your automation. Complexity explodes!

The complexity exists whether it's a monorepo or many separate repos. A monorepo lets you encode that complexity as versioned code in your build system. Separate repos encode it across people's heads, wikis and who knows what else. Hiding complexity doesn't mean it doesn't exist, just that it will bite you 10 times as hard eventually.

Reply on news.ycombinator.com