Hacker News Reader

Comment by ceronman | Hacker News Reader

There is available tooling to help you with that, not only has to be discipline.

A monorepo oriented build tool such as Bazel will make sure that you don't compile more than needed. This is extremely efficient. In fact, in my company with a few hundred projects we tried it and it was 10 times faster than our previous Maven based workflow. These tools also help you with dependency management so you keep your code organized. The killer feature is that you finally have true Continuous Integration because the build tool will figure out any dependencies of the code that you change, rebuilding it and testing to make sure that nothing is broken.

View on HN

I don't trust incremental builds (whether via Bazel or any other tool) for an actual deployable build - I've seen them go wrong too many times.

For the local edit-test cycle incremental builds are great, but any decent IDE will do them in the multirepo case just as in the monorepo case.

Wait what? As long as the build process is reproducible, why do you care if it's incremental or not? If you don't trust bazel, people have been using distcc+ccache since forever.

The concern is that an incremental build is not reproducible. In theory, it is, but in practice, sometimes it is not.

Doing a clean build for the final artifact that is going to be released is pretty standard practice, IME.

Have you used bazel? The number of times I've even considered using `clean` (the drop some caches make everything mostly clean though still use the global cache command) in my 2+ years with it I can count one one hand, and most of those were unnecessary.

Just as an anecdote, in all my time as a professional software developer, the number of projects I have worked on that have enjoyed reliable, reproducible, incremental builds is zero. The effects of global optimisations change. The output code winds up varying based on the order of incremental changes. Updated dependencies get overlooked because of race conditions or networking glitches or filesystem bugs or a hundred other unlikely but not impossible failures.

Incremental builds are great for quick day-to-day development, but in my experience they are invariably unsuitable for robust production builds. For me, those should use a git clone or equivalent into an empty directory and then a completely clean build of everything, always.

> As long as the build process is reproducible, why do you care if it's incremental or not?

I wouldn't care if it was reliable. My experience is that it isn't.

> If you don't trust bazel, people have been using distcc+ccache since forever.

And it's been causing inconsistent builds all the while.

I've been using bazel (and the Google-internal version, blaze) for years now. The builds are extremely reliable and are used to deploy loads of projects across Google every day. I no longer work there, but my new company is also using bazel and we never run into issues with unreliable builds.

I think you're falling into the trap of "past solutions didn't work, therefore the problem is intractable", which is a special case of the "throwing the baby out with the bathwater" fallacy. Sometimes a new tool comes out that fixes the problems of past tools.

> I think you're falling into the trap of "past solutions didn't work, therefore the problem is intractable", which is a special case of the "throwing the baby out with the bathwater" fallacy. Sometimes a new tool comes out that fixes the problems of past tools.

Where's the breakthrough then? What's the magic that allows bazel to succeed where dozens of similar efforts have failed? Sometimes a new tool does fix the problems of past tools, but that's almost always due to some new insight.

Not sure exactly, since I haven't used any of the dozens of similar efforts. It sounds like you're a lot more familiar with the area, so here's some resources:

* Bazel's FAQ: https://bazel.build/faq.html

* A description on how Blaze works: https://google-engtools.blogspot.com/2011/08/build-in-cloud-...

Unfortunately the design doc for Blaze is internal and AFAIK not available publicly. I imagine they took a look at a number of the similar efforts and made decisions based on what those approaches did wrong.

While I can relate to your point, if you actually experience Blaze at Google (and Bazel too, but it has other ecosystem immaturity issues not related to reproducibility) you will quickly change your mind. I can totally see why prior tools you used may have put bad taste in your mouth.

My point was about the the fact that by declaring your dependencies explicitly, Bazel allows you to compile only the things you depend on, not everything in the monorepo. And once you have built your dependencies once, you don't have to build them again.

If you think about it, a package manager workflow (like maven, nuget, etc) is also a form of incremental compilation. You compile your libraries only once and publish them in binary form in some sort of repository and then import them. The difference is that if you want to make a change in both a library and an application this becomes hard to do with packages, while very easy with a monorepo and Bazel.

> If you think about it, a package manager workflow (like maven, nuget, etc) is also a form of incremental compilation. You compile your libraries only once and publish them in binary form in some sort of repository and then import them.

The difference is that that's explicit and visible, rather than being "magically" figured out by the build system. And the compiled binaries are 100% reproducible everywhere since it's literally the same file from the repository everywhere. So it's much more reliable, understandable, and debuggable.

> The difference is that if you want to make a change in both a library and an application this becomes hard to do with packages, while very easy with a monorepo and Bazel.

Having a library and application so tightly coupled that changes have to be simultaneous is poor practice, and will likely get you into trouble when the time comes to deploy them. Better to enforce that you test old application + new library (or vice versa) up front, than hit it for the first time in production.

> A monorepo oriented build tool such as Bazel will make sure that you don't compile more than needed.

Not at all true in my experience. I use Buck, which is very similar. It constantly recompiles stuff it doesn't need to by any standard. I've seriously considered avoiding Mercurial bookmarks in favor of manual patch management to see if I can avoid triggering the cases where it gets stupid. Maybe it's a problem specific to Buck, but my conversations with peers at other companies suggest not (and only a tiny percentage of developers at any of these companies get a choice of which tool to use).

The other issue is that "more than needed" is subject to more than one interpretation. There's "more than needed" as code was actually written, and there's "more than needed" as code should have been written. Tooling can help with the first but not the second. If developers are sloppy about how their header files are structured, then a change to one that's widely used will make it "necessary" to recompile a lot of code - including code that didn't actually use the part that changed. Oddly enough, though, that "necessary" recompilation wouldn't occur if those same header files were split up. (BTW some languages make good design more difficult by requiring private implementation details in public header files, but that's a separate conversation.) Tooling can optimize what it can see, but what it sees might already be a tangle of dependencies added only because a monorepo made it easy.

It sounds like it might be worth trying Bazel (I haven't used either). The parent made a suggestion of a specific tool with one anecdatum about performance, and your argument is that a different tool doesn't do the same thing.

My real argument, as represented by the majority of the text I wrote, was that tooling only solves one part of the problem. The lesser part, at that. But I guess that went over the downvote brigade's heads. Ditto for the fact that I did mention conversations with peers who used other tools (including Bazel BTW) and had similar experiences. Seriously, folks, read before you nitpick.

You are describing modifying someone else's unrelated code will cause the rebuild of your code right? This looks like your targets unintentionally depend on the other code.

To find out how/where the dependency was introduced, you can use bazel query 'somepath(your_target, some_other_file)'. Not sure if Buck have an equivalent, but bazel should work with Buck repo.

> This looks like your targets unintentionally depend on the other code.

Sometimes it's unintentional, for example an include left in after it's no longer needed. Other times it's intentional but lazy. For example, I often see commits that pull in one dependency just to use one tiny little utility function, but that dependency pulls in ten others, which each pull in ten others, and before you know it any change in a hundred files forces yours to be recompiled even when that tiny utility function wasn't touched.

A responsible developer would refactor the header file to keep the number of spurious rebuilds down, but the world is full of irresponsible developers. With multiple repos, the public immutable part of an API and the private fluid part tend to find their way into separate files pretty quickly. With a monorepo they don't, which is kind of ironic because one of the purported benefits of a monorepo is that it's easy to reach into "someone else's code" and refactor it. Unfortunately, people just don't seem to use that ability for good often enough unless review culture forces them to.

> Tooling can optimize what it can see, but what it sees might already be a tangle of dependencies added only because a monorepo made it easy.

In practice this is almost never because of "a monorepo" and more because people don't like being good about including what they use (see: any Titus Winters talk ever). If, for example you have :test_foo and :foo_lib, things are fine and dandy. When you modify foo_lib you'll run :test_foo and nothing else. But often people are lazy and you have :foobarbaz_lib which is built from three source files, foo, bar, and baz. Each of those has a test, but all the tests depend on :foobarbaz_lib, because that's all that exists. So when you change foo, you run :baz_test, even though in practice, baz might depend on foo, and not the reverse. You fix this by making :foo_lib, :bar_lib, and :baz_lib each with their associated :target_test and explicit dependencies, and then :foobarbaz_lib depends on all three libs (or doesn't exist at all, and everyone who needs :foo_lib uses :foo_lib directly.

So yes, absolutely, when/if you start (essentially) globbing your imports, bazel/buck/pants stop being awesome. So don't do that.

> In practice this is almost never because of "a monorepo"

I think you mean in the practice you've seen. I've seen some practice too, and it seems much more prevalent in monorepos. Yeah, still a small sample, correlation does not equal causation, yadda yadda, but I still think it's because monorepos make it so damn easy to reach out and add a huge tree of dependencies without having to think about it. My whole point is that it's possible to do the right thing, but requires more discipline.

> add a huge tree of dependencies without having to think about it.

Keep in mind that the alternative is "I have to build (or pull a prebuilt version of) the entirety of every repo that I depend on". An entire repo is always going to be larger than a single target. Yes you're right that pulling in the entirely of {{some large tool}} because you want `tool.strings.util.mangle_name` is bad.

The alternatives are

1. (multi-repo) You stick mangle name in its own repo (think `left-pad`)

2. (monorepo) Depend on `tool` in its entirety

3. (either, though more common in multi-repo) Everyone re-implements mangle_name. If the way the name is mangled is important, and it changes in the future, you have major issues and incompatibilities. (think IPV4 -> IPV6 for an example where a "global" `parse_addr` function would have made everyone's lives easier). You can bet on the assumption that mangle_name will never change, but then you aren't doing software engineering[1]

All of these are the wrong solution

4. Core utilities like this get factored out into their own files. In a multi-repo environment, everyone depends on the entire core/util repo. In a monorepo environment, you depend on the granular files you need.

4 is the only correct solution, and note that in a monorepo environment it results in you depending on fewer things than in a multi-repo environment. Yes there are a lot of ways to do things wrong, But that's true no matter what tooling you use. When used "correctly", monorepos have the best outcome, and the bad outcomes aren't actually that bad (sometimes longer builds and binary bloat if you don't tree-shake). You always need discipline. Monorepos don't need more.

[1]: https://www.youtube.com/watch?v=tISy7EJQPzI

Reply on news.ycombinator.com