Turborepo 1.2: High-performance build system for monorepos

Turborepo author/founder here....

Yes, it's possible. Databricks currently uses both Turborepo and Bazel together in their monorepo[1]

[1] https://twitter.com/elado/status/1504216742393876483?s=20&t=...

> Is Bazel designed in a way that make it impossible to do JS monorepos well?

Not impossible, but you really need to go all in with it and follow its conventions and practices. See this for the main docs: https://github.com/bazelbuild/rules_nodejs

One thing in particular that doesn't work well in the bazel world is doing your own stuff outside its BUILD.bazel files. If you're used to just npm install and jam some code in your package.json scripts... that doesn't usually work in the bazel world. If you have a lot of logic or tools in your build you'll likely need to go all in and make bazel starlark rules or macros that recreate that logic. Nothing is impossible, but expect to spend time getting up to speed and getting things working the bazel way.

Yes, it's possible to use Bazel w/ JS tooling. At Uber we run a 1000+ package monorepo w/ Bazel and Yarn. Someone else mentioned rules_nodejs if you want to go with the popular option that is more or less in line with the Bazel philosophy[0]. We use a project called jazelle[1] that makes some different trade-offs that lets us do some interesting things like not busting the whole world's cache when lockfile changes, and source code generation (including automatic syncing of BUILD files to package.json)

[0] https://news.ycombinator.com/item?id=30959893

[1] https://github.com/uber-web/jazelle

> Is it possible to integrate Turborepo with general-purpose monorepo build tools? Bazel, in particular?

It's definitely possible, but I think the practical limitations would make it too complex to reason around and maintain. You'd end up creating two separate and overlapping systems to declare dependency graphs and input sources and manage caching and execution.

I haven't yet seen a case where the two are actually interleaved. Currently at Databricks, we use Bazel to provide the correctness guarantees and interop needed for CI, and we use JS-specific tooling (non-Bazel) locally to meet our performance needs, where the usage profile is different and where we're willing to make correctness tradeoffs.

> (Is Bazel designed in a way that make it impossible to do JS monorepos well?)

There are limitations in Bazel that don't play nicely with modern JS conventions. For example, Bazel's standard sandbox is based on symlink farms, and Node.js and the ecosystem by default follow symlinks[1] to their real locations, effectively breaking sandboxing. A FUSE or custom filesystem (Google's version of Bazel takes advantage of one[2]) would be better but is not as portable. As another example, Bazel's action cache tries to watch or otherwise verify the integrity of every input file to an action, and when node_modules is 100k+ files, this gets expensive and is prone to non-determinism. Bazel does this for correctness, which is noble but results in practical performance problems. You need to do extra work to "trick" Bazel into not reading these 100k+ files each time.

The problems feel solvable to me, but not easily without adding yet more configuration options to Bazel. The influx of new JS-specific tooling is a reset to this, building the minimum viable set of functionality that the JS ecosystem specifically needs, without the burdens of being a general purpose build system.

[1] https://nodejs.org/api/cli.html#--preserve-symlinks [2] https://dl.acm.org/doi/10.1145/2854146

I'm with you in that a lot of TS use seems dogmatic or ritualistic today. I have a strong feeling in the near future we're going to see the bow string snap back and simple zero build, basic dynamic use of pure JS comes back in vogue.

> Typescript is part of the problem

Typescript is a solution to the jank-fucking-tastic type munging JS does (see: [1] About 1:20 in) and the problems that ensue.... if all you are doing is just making menus appear/disappear on click, by all means keep to JS.

The build steps/tooling are useful when you want to build actual applications rather than decorate a marketing page, and also when you need to support legacy browsers, being able to work with modern sensibilities and get code that'll work in IE11 is a blessing.

> the unnecessary complexity of the Javascript ecosystem is mostly about gatekeeping and aesthetics and i will die on that hill.

JS has much more of a "flavour of the week" problem than more mature ecosystems like PHP, I put that down to a relatively poor stdlib by comparison, rather than aesthetics or gatekeeping.

    1. https://www.destroyallsoftware.com/talks/wat

No, I’m saying I’d like to install eslint at the root of the repo and turbo could just spawn a process to use the root install of eslint for each package in the monorepo using the root config. Could do the same with prettier etc. just use a sub process to spawn and control the directories it runs against.

I’d like to see this for other tools too. I know turbo knows what the root of my repository is, this would make tools and reusabilty more efficient. It’s be even better if I could pass arbitrary arguments to actions like that so commands know where they are operating

It gets tiresome having to copy configuration and what not. Just less files to manage and less places to have to update things overtime.

It’s almost like you need gulp tasks again or something like that and the whole idea is to get away from task runners right? Yet I still need to manage my per package configs and pipelines on top of turbo

Turbo still forces you to have to think about so many other parts of the pipeline. At least nx has plugins but it’s support outside of their happy path for customizations is horrid, but turbo just lacks any thought to this problem of re usability on a task basis.

I don’t want to litter the package.json for each packages with every dependency they need to operate actions be it formatting, minting, etc because that means I have to check them all for different versions, make sure new packages have the same layout etc. A task runner like turbo should be aware enough to know how to do segmented task running by spawning processes for each package without needing a copy of whatever tool installed for each package

HMAC are symmetric signatures, which means that anyone who can check the signature can also poison the cache (so pushing only from the CI are reading/checking on the dev machines would not be possible)

The primary difference is that in Nix it's fairly easy to wrap arbitrary build systems, so you can inject third-party dependencies into your codebase without vendoring and adapting their build system (plus, through nixpkgs[0], the majority of all relevant software has already been wrapped).

In Bazel you have to do a lot of that work yourself. Google has essentially ~unlimited manpower to do this for the things in their third_party tree, but for any other size of organisation this is not the case and people resort to all sorts of ugly hacks.

Depending on your needs these different types of projects can also coexist with each other in Nix. To use an example from my work, we have a Nix-native build system for Go[1] and code using this[2] co-exists with code that just uses the standard Go build system[3]. Both methods end up being addressable on the same abstraction level, meaning that they both turn into equivalent build targets (you can see both in a full build[4] of our repo).

And for what it's worth, some of the things Bazel gets extremely right (such as having a straightforward mapping from code location to build target, universal build command) are pretty easy to do in Nix (see readTree[5], magrathea[6]).

[0]: https://github.com/NixOS/nixpkgs

[1]: https://code.tvl.fyi/about/nix/buildGo

[2]: https://cs.tvl.fyi/depot/-/blob/ops/besadii/default.nix

[3]: https://cs.tvl.fyi/depot/-/blob/third_party/gerrit-queue/def...

[4]: https://buildkite.com/tvl/depot/builds/13067

[5]: https://cs.tvl.fyi/depot/-/blob/nix/readTree/README.md

[6]: https://cs.tvl.fyi/depot@f0e6d3498d6c0f905977ea9432c311b6808...

The main thing to be aware of with Bazel is its sandboxing philosophy. Instead of using the files in place, it basically creates a copy of the relevant file structure for any given build (i.e. minus the files that might exist on a given folder but be irrelevant for a build). Because of this, it generally requires strict hermeticity guarantees from everything you're doing, so many things have to be done in a "Bazel way" (i.e. there are specific ways in which you're supposed to download dependencies and there are restrictions in terms of file creation in e.g. tests that might deal w/ I/O)

The way that Bazel approaches flexibility is by being programmable (via a Python-like language called Starlark). There are open source rulesets for various languages that you can use to get off the ground faster, but they are usually fairly opinionated, so if you want to deviate from the happy path, you're going to hit face first into a fairly steep learning curve about depsets, graphs, file/symlink permissions and the like.

Another "weakness" of Bazel is that it's not aware of ultra granular file-to-file dependencies out of the box (e.g. how Golang imports work vs Java's), so either you have to manage dependency graphs manually or you invest in wrapper tooling like gazelle to automate some of the BUILD file generation for you.

If you're doing something new it's pretty straightforward to start with bazel and do everything from the start 'the bazel way'. If you're pulling in an existing codebase and a lot of dependencies along with it, you could be spending a lot of time to make it work 'the bazel way'. There's a very narrow path that bazel forces you to be on--this is what makes it work so well but make it difficult to stray off that path.

Personally I don't think this is a good idea. Yes, you can make it work if you for some reason have to use both tools. However, both Nix and Bazel are tools that expect to "control the entire world" and whichever way you decide to layer that onion will contain some amount of pain (not to mention that both are very complex tools and you're adding significant onboarding cost by using both).