Hacker News Reader

Comment by trzy | Hacker News Reader

Last year I tried doing a side project with a talented ex-Google buddy who insisted we set up Bazel to replace my simple Makefile. Three weeks later it still wasn’t working on my Windows box. We had a mixed Python and C++ code base and I like to use MinGW64 gcc on Windows. He blamed Windows and tried to get me to switch to Mac (no thanks, lol) and eventually he lost interest and gave up. The project went on to win an OpenCV funded competition and became the basis of a startup — good job, GNU Make!

So the answer IMHO to “when to use Bazel” is “never” :)

View on HN

In general, if someone wants to beef up the tooling in my project that I know is fine to begin with, I still say "I don't mind, go ahead, but I will continue using my old tooling until yours works with no regression." Half the time, they abandon it. I don't even know how Bazel works cause I've never felt enough pain with makefiles etc to research an alternative.

Also, Google tooling knowledge doesn't transfer very well to outside projects. Everything is special on the inside there. They use an internal version of Bazel called Blaze that's totally integrated with everything, with entire teams dedicated to the tooling, and only a few officially supported programming languages, so of course it works smoothly.

I used it for several years inside Google, and TBH it was often painful in the ways external users have described in this thread. When I was doing something very "normal" from the point of view of Google's monolithic codebase, i.e. targeting a C++ binary intended to run on Google's distributed computing platform, I found it smooth enough, if awfully verbose. When I was doing something less vanilla, but not necessarily uncommon, e.g. targeting Android, iOS, Windows, it was painful, and I spent a lot of time working around it or understanding the chains/mountains of clever things others had built to work around it.

I understood some of the design decisions and tradeoffs for why it works the way it does, but I didn't like it. I certainly wouldn't reach for it for a personal project. Possibly I still just didn't "get it". I've used dozens* of build systems at this point, but hey.

* possibly exaggerating

When was this? I recall they were just getting around to cross-platform builds around 2015 or so (as I had helped in porting some QUIC stuff in my 20% time). I would image 7 years later the picture looks very different over there.

> Google tooling knowledge doesn't transfer very well to outside projects.

A good number of their open source projects fall apart in real-world use. If you visit their repositories you'll often get the sense that beyond working on interesting problems, their engineers have no desire to properly maintain these projects for the next few years. The gamble of relying on a Google product unfortunately also applies to their open source projects.

I roast Google at every opportunity but this does not seem fair to me.

What about Golang? Angular? Dart? Even the bigger stuff like Chrome and Android while a bear to deal with are integrated with every day by a lot of companies.

What is some Google OSS that has failed in the way you mention?

Angular, famous for betraying their devoted developer community with a complete breaking change when going from 1.x to 2.0. No migration plan whatsoever. It was a totally new framework which happened to share the same name. After a few years they relented and put in some work to help migrate existing users, but by that point most people has already put in the work to migrate towards a framework that wouldn't just leave you behind.

>No migration plan whatsoever. Not 100% true, there were a lot of workarounds and documentation how to upgrade.

Sure it wasn't great, but the few projects I did with Angular 6,7,8 was kinda nice. I still have an AngularJS project in production which is also nice :)

Of course any new JS project is now done in Svelte (nothing simpler) or as nice

PS. It's nice to be nice :D

Angular 1.x was (is?) still viable. Its not a breaking change because its an entirely optional shift. I was working in Angular 1.x at the time and while there was a lot of hype for 2 we all knew Angular 1.x wasn't going anywhere.

The migration plan is to use React instead.

Maybe that's because the average tenure at google is so short?

Average tenure at Google is so short because they have grown ridiculously. The actual attrition rates are slower than industry norms.

This is correct. People were always shocked how many SWEs were newer than them after working for half a year or so.

"ex-google" is a really valuable notch to have on your CV

> until yours works with no regression

and there still no guarantee of no such regressions one month later.

I give it a trial period still. But yes, one time a guy switched our NodeJS project to Typescript. It worked for a little, then stuff started breaking (mainly involving imports/modules). He left for unrelated reasons, so I immediately took us back.

A shame, TS really is nicer when you get used to it.

My experience is that Windows is a constant source of headaches if you're supporting development and builds on multiple platforms and trying not to just write every build-related thing twice, Bazel or no Bazel. Likelihood of it being a PITA goes up fast the more complex the build & tools (probably why Make did better than Bazel).

I assume it's OK if it's your only platform—though I got my programming start on Windows, with open-source languages, and distinctly recall how managing the same tools and builds got way easier and more reliable when I switched to Linux. Luckily WSL2 is getting semi-decent and can call out to Windows tools, so it's getting easier to standardize on one or another unixy scripting language for glue, even if some of your tools on Windows are native.

Part of the trouble with it, though, is that there are multiple ways to end up with some kind of linux-ish environment on it, and that all of them introduce quirks or oddities. You end up having to wrestle with stupid questions like "which copy of OpenSSH is this tool using?" It's probably less-bad if you're in a position to strictly dictate what Windows dev machines have installed on them, I suppose. Git is installed, but only with options X, Y, and Z checked, no other configurations allowed; WSL2 is installed and has packages A, B, and C installed; and so on, crucially preventing the installation of other things that might try to pile on more weirdness and unpredictability.

Those kinds of messes are possible on macOS or Linux or FreeBSD or what have you, but generally don't happen. I think it happens on Windows because every tool's trying to vendor in various Linux-compat dependencies rather than force a particular system configuration on users, or having to try to deal with whatever maybe-not-actually-compatible similar tools the user has installed. So they vendor them in to reduce friction and the rate of spurious bug reports. Basically, the authors of these tools are seeing the same problems as anyone trying to configure cross-platform development with Windows in the mix, and are throwing up their hands and vendoring in their deps, which compounds the problem for anyone trying to coordinate several such tools because there's a tendency for everything to end up with weird configurations that don't play well together.

All excellent points but Windows is a very widely used platform (perhaps the most widely used, by some metrics) and it’s actually quite nice for development. It’s hard to take tools seriously that don’t work on it.

Unpopular opinion but I think the Unix approach of lots of little tools with arcane configuration files all blaming each other sucks. I get why it exists, why it became popular, and why it remains popular in some scenarios, but I don’t feel beholden to it.

>and it’s actually quite nice for development

If we are talking about stuff like backend web development then I'd say no, it's actually a pain to work with - few people bother deploying to Windows these days, many libraries/frameworks treat it like a second class citizen, tooling doesn't translate well (or at all).

Even frontend webdev is very unixy due to node/npm and popularity of Macs in that space.

When you're talking about OpenCV, GPU related things, game dev then the situation is reversed.

So it highly depends on what you're developing.

I think we each live in our own bubbles. I see very few people on macs or linuxes, almost everyone uses Windows and they're fine with web development. What's so unixy about node/npm? They feel like very good windows citizens.

For the most part npm has worked for me on Windows, but it's so incredibly slow compared to even npm in a Linux VM on the same host. I understand this is mostly due to FS overhead related to a multitude of small files.

Obviously this isn't a deal breaker in most situations, but it is a bit annoying. I've observed similar performance problems running Webpack on Windows in the past.

Add your npm directories to the virus file exclusion lists of Windows Defender. Note that some of the directories may not be obvious (those under AppData for example). The speed improvements are massive for tools like these that touch lots of small files.

Also take a look at this talk on NTFS, file system speed, and how the presenter optimized the Rust installer: https://youtu.be/qbKGw8MQ0i8 . There is an awful lot that can be done to tools that deal with lots of files to perform well under Windows too.

There’s a possibility this is due to corporate antivirus. Many AV solutions (seemingly not Windows Defender) scan every file that’s written, massively slowing down IO, but they can’t scan VHDs. Excluding the folder where node_modules is or running npm inside WSL2 has good performance on my work laptop, in a random windows folder not so much.

There was a HN post a while back where a MS engineer was explaining why windows file system access will never be as fast as Linux.

> I understand this is mostly due to FS overhead related to a multitude of small files.

No. It was the VFS design. They have lots hooks and extension points for 3rd party, there is no way to improve without breaking backward compatibility. It was the reason why many antivirus was broken in the first few releases of windows 10.

Node didn't even run on Windows for a while and then when support was added it was very buggy for a long time - it clearly has roots in Mac/Linux ecosystem and windows was a second class citizen. Can't comment on recent support because haven't done front-end on Windows >5 years.

The only time I see Windows frontend devs is full-stack .NET shops or low cost outsourcing centers. If nothing else then just for practicality - you need to test on Safari and iOS so might as well get Mac devices.

Are you in a .Net shop?

Right—in some sense it's not really Windows' fault that something like Git is barely-portable and relies on a rube-goldberg-machine pile of scripts and binaries in several languages with the result that it has to ship half a unix environment to work somewhere that's very non-unix, or that open-source software in general treats Windows the way web developers treat Firefox.

OTOH, in practice, if you're using a lot of open source tools and have developers working on every single halfway-plausible OS except windows, things are probably pretty OK, but then you throw Windows in the mix and suddenly the time you spend supporting your builds & tools shoots way up. That makes it feel like it's Windows' fault.

Git is written in C. It hasn't been a "rube-goldberg-machine pile of scripts and binaries in several languages" for a long time (almost a decade).

While there are fewer Perl and shell commands than in the past, several remain. Scroll down to git-add—-interactive.perl to see:

https://github.com/git/git/tree/master

My install directories sure look like it's still that way. And I'm not using a decade-old git. I discovered this fact about git far more recently than a decade ago, in fact (though I was using it back then).

Has git rebase --interactive been rewritten in C yet or is it still a bunch of Perl?

    $ file -L /usr/lib/git-core/git-rebase 
    /usr/lib/git-core/git-rebase: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=95d963a593ffeeb306e8525aacce02f1cbc7dabd, for GNU/Linux 3.2.0, stripped

I specifically said git rebase --interactive. I know the non-interactive rebase is in C.

Android is an even bigger pain, suddenly you have to call Java code just to map memory and can't even have a simple C api for something as low level as that.

As a development and/or build environment? Never used it that way, myself. Agree that as a build target, and as far as its various first-party developer-facing features and libraries and SDKs and such go, it's got weirdly-many rough edges and half-baked-but-heavily-promoted junk for such a major platform.

Mercurial worked very nicely on Windows for a lot of years that Git didn't.

It was a complete waste of Mercurial developer's time that would have been better spent creating a GitHub equivalent.

If it doesn't come from Microsoft, Windows developers don't care.

Learn that lesson or repeat the mistake.

> It’s hard to take tools seriously that don’t work on it.

In my experience, most software development tools don't work on it, at least not as well as on Unix-like systems. But I'm not a game dev.

Visual Studio has pretty good native support for CMake. Going from Make to CMake would have been a better decision than trying to use Bazel.

There's a standard find_package/find_library macro and definitions for OpenCV (which is your use case here). Then use either Microsoft's C++ package manger or Conan to actually install the libraries.

I had a recent issue on windows with build times increasing 10X, turned out the Unix approach of spawning lots of processes was triggering the corporate antivirus. Didn't cause the build to fail but the result was the antivirus was scanning everything the process touched or wrote to disk. And make + gcc opens and closes a huge number of files.

I mean, Python (moreso its ecosystem) is notoriously difficult to get working w/ Bazel idiomatically. Bazel is advertised as a language agnostic system (and to be fair, it's better at it than, say, Buck), but in practice it tends to work better w/ stacks that already have some degree of hermeticity (Go, Java), whereas YMMV very much with the more loosey-goosey stacks (Python, JS).

IMHO, Bazel is a classic example of Conway's law[0], and it falls squarely in the "big corp" class of software. You have to be running into issues like 6+ digit CI compute costs or latency SLOs in hundreds-of-teams monorepos before Bazel really starts to make sense as a potential technical solution.

[0] https://en.wikipedia.org/wiki/Conway%27s_law

I've used bazel at medium size (100+ engineers, multiple languages) and tried to use it at a small size (20 engineers, almost all go with a tiny bit of C) and I think the "never" and "unless there's no other way" answers from this interview are pretty good.

Bazel probably wasn't a good fit for that project, especially if your makefile was simple.

But where it does work well, is when you have a large complex codebase in a monorepo, and you need reliable caching to keep build times down.

C + Makefile + CCache: I just `find . -name "*.h"` and throw that as the dependency of every C file, and then let ccache/make figure out the rest. My code base runs upwards of 10+ million LOC, and incremental builds are in the "hundreds of milliseconds" and "clean" builds (ccache clean, mind you) are about 30s. Truly clean builds (on a scratch system) are ~2 minutes. We do spend time making sure our very long files still compile very quickly -- we're careful to stay away from code that exercises pathological paths through the compiler. I've found that a good laptop can compile ~20kloc/(s-thread) throughput with parallel builds (it varies from 1000–100000loc/(s-thread)).

That's specific to c. Bazel is often used in cases where you have multiple languages in use, and possibly dependencies between projects using different languages.

> I just `find . -name "*.h"`

That sounds like it would cause problems with not triggering a build if a .h file is deleted. And would trigger unnecessary builds if any .h file is modified. But if your builds are that fast, maybe that isn't a problem.

It is specific to C/C++, for sure -- I started with that.

ccache is going to be as fast as any checksum dependency tracker --- otherwise you're relying on a timestamp, which isn't any better than "bare" Make. We use `*.d` rules from the `-MD` flag to prevent the "deleted header" case.

Considering the brevity of the Makefile (it looks like a "basic" build), performance, and the practical robustness, it's in a pretty great trade-off space.

From what you've described I definitely don't think it would be worth migrating what you have to bazel. My point is just that there are environments where it does provide benefits.

At the risk of looking dumb here, don't incremental builds take care of this, how is caching different? ? Make gives me incremental builds and my makefile lets me define the inputs and the outputs. What am I missing with Bazel?!

1. Make uses mtime to determine whether inputs have changed. Bazel uses a hash of all inputs to the target (file contents + other inputs like environment variables)

2. Make lets you define inputs and outputs, but doesn't enforce them. Bazel sandboxes build actions; if you try to import a dependency that you haven't listed as a direct or transitive dependency, the build will fail. This is leads to users never (or at least very rarely) having to run `bazel clean`.

Haha, I like the frankness :)

To take it a step further. I never left the "comfort" of bash-build and bash-deploy scripts.

In most of my projects (pro and personal) there is a deploy-aws-prod.sh and deploy-aws-dev.sh (or some variation)

None is longer than a 10-20 bash commands.

These "projects" are webservice, ml-models, batch-processing(think distributed clusters)

It's ugly and perfect-enough at the same time.

YMMV

I mean was there anything wrong about the makefile? Did your product need reproducible builds?

This is becoming more and more of a pet peeve of mine, about to become an outright annoyance; the adding or changing of tools without things actually improving, or the purported improvement not actually being relevant or even close to the most important thing that time should be spent on.

https://mcfunley.com/choose-boring-technology is a good starting point for moving away from this mindset.

I had a Go project up until earlier this year, I could set it up myself. I picked makefiles to build it (not bazel, that would be overkill; not mage, that would mean more coding), and it worked just fine. I never had a compelling reason to move away from it.

Actually it was more than fine, it was a relief; my previous experience with tools like that had been Maven (where everything is XML and you need a plugin to do basic things like move or remove a file) and the Javascript ecosystem, from before everything was merged into package.json / nodejs-style. Dependency management too.

I typically want reproducible builds to never ever have to deal with build issues and to be able to free up that space to think.

Boring technologies are easier for people better and making due with what they have and accepting sometimes very large limitations.

ADHD makes boring tasks harder for instance, and boring technologies give you no escape from the tedium.

With a little fancier technology though, you at least have hope ;)

I used bazel for my latest project, mainly as an excuse to learn it. I ended up spending waaay too much time debugging bazel instead of working on my code, and I still can’t properly support windows because dependencies don’t build.

I will never use bazel again. Not worth the effort required, and doesn’t work out of the box as advertised.

That's like saying you should never build a house with concrete foundations because they take so long to dig and pour and the first layer of bricks doesn't need them anyway!

Get back to me when you work for a company with a monorepo that has to build and test everything in CI for every change (taking something like 200 CPU hours) because they didn't have the foresight to use Bazel.

Software development isn't housing development. You lay a concrete foundation knowing exactly what is being built on top. The typical way of developing software is like adding stories to your house until it falls over, then rebuilding it stronger, which is perfectly fine in the software world. Otherwise things get overengineered.

In practical terms, the makefile or whatever is usually good enough.

If you're big enough to roll your own software ecosystem -- which means you have thousands of developers working on hundreds of closely coupled projects in a single code base -- and you need repeatable builds, then Bazel makes sense.

Bazel isn't designed to make things easy. It isn't even designed to make things fast (though it is easy to add that on due to repeatable builds). Bazel is designed to prevent you from shooting yourself in the foot; providing strong controls on project dependencies, both dependencies of your project as well as other projects' dependence on your project.

That means in 99% of companies the answer is no. You're better off with CMake or something thing like that.

> then rebuilding it stronger

That's a very rare occurrence in software. Honestly it's probably about as common as in housing. You can rebuild parts of a piece of software but the build system is generally really foundational and almost never changes in my experience. Especially if the software isn't a library so nobody except the main developers has to interact with the build system.

The build system can change with hardly any of the code changing. This is evident in how I've never even had to think about the build system at my SWE day job. It's evolved in the meantime.

What's often impossible to replace without a full rewrite is a backend's DBMS, and people tend to underestimate how tight of a coupling that is. Only gets worse with abstraction.

> company with a monorepo that has to build and test everything in CI for every change

That smells like covering up an architecture problem (tight coupling) with a build tool. Not a recipe for success.

I don't think that's actually the case. Bazel, AIUI, understands the entire dependency tree and can cache steps effectively, which a Makefile cannot (alone) do. (You could certainly build it into one.)

CI tooling is orthogonal; your CI tooling can invoke Bazel or Makefile+custom caching, it wouldn't really matter. But the problem in CI is that you lack state; whereas a developer's machine might e.g., have a bunch of .o files from a previous build, CI will be starting from a clean slate¹. So builds take longer. My understanding of Bazel is that it has decent support for caching and can thus makes those rebuilds a lot faster, which it has b/c it understands exactly the inputs to the steps at hand. (Which, if you write your own caching layer, you'd have to figure out.)

… but … I've also seen the same thing the top commenter has, with Bazel: endless suggestions to use Bazel … but migrating a large existing codebase is pain, and if you're not willing to put your work where your mouth is … then the status quo prevails.

¹While here CI lacking state is a sort of negative b/c of the time we need to rebuild it, normally, I think this is a good thing: it means you're constantly verifying you can build from scratch. Or, at least, something close to scratch, modulo things like vendoring. (I.e., if you do an "apt-get" from Ubuntu's servers in your CI pipeline … well … your build obviously isn't quite from scratch.)

Having experience FAANG monorepos, the other commenter was entirely correct about them papering over tight coupling with tools.

It makes more sense when you realize it was really built to deal with statically linked C++ - there's no guarantee of ABI, so you really need to ensure that all your dependencies are built the same way as your project to ensure compatibility (and the dependency graphs in monorepos get DEEP).

For any language with a stable ABI that utilizes dynamic linking, the complexity is far more then the benefit. The whole monorepo concept and all the tooling that goes with it really makes more sense when you think of it in that context of statically linked code that can't easily be packaged into something as portable as a JAR, so it's critical to build it all the same way.

Google uses a monorepo intentionally, and for good reasons. They're also a massive company. For them, a complex build system makes sense... and even that system isn't exactly Bazel.

Google uses a monorepo because they were popular in 1999, not because they had a specific reason to do so[1]. As the codebase grew, they never changed and instead built tools around it.

That's a perfectly fine choice if you have bodies to throw at the problem and can deal with the consequences of that choice, but most people can't. Google can and so it does. If it wasn't as successful as it is, it probably would have collapsed under the strain of the people and systems required to manage that code.

[1]: https://qeunit.com/blog/how-google-does-monorepo/

Ok, could've been chosen for 1999 reasons back then, but there are good reasons today. It's easier to build things collaboratively, esp across teams, when there's just one repo and one branch (I don't count release branches). Yes it takes effort to support due to its size, but that's contained within a few teams, and everyone else benefits from it.

Other tooling decisions place a burden on every SWE. Their config language is a weird trash that everyone has to sorta learn. Their deployment and integration testing tooling have similar qualities, and it's not cause of the monorepo. One consequence is that many teams cannot feasibly deploy microservices, even though it'd often make sense to, which is even worse if they're using C++.

What if I stick with companies that have enough foresight to not cause every change to build everything?

In CI? Then hooray! Your company is using Bazel (or Buck or Pants or similar).

You are talking about a two men project becoming a one man project. You are not the target of a complex build system. At that size, you would be fine with basically anything.

I’m really glade that Meson and Ninja exist however. I hate GNU Make like few things in my toolbox. M4 really is an awful language.

Don't use tools outside what they're designed for...

Specifically, Bazel is really at home with Java/C++ and Linux. Sure, it kinda works elsewhere, but you should be considering other options.

i think its more like for super small startups, the best move is always to keep things as simple as possible (like basic build/deploy scripts) and work more on the product.

Reply on news.ycombinator.com