Buck2: Our open source build system
engineering.fb.com
engineering.fb.com
I don't see many details about the sandboxing/hermetic build story in the docs, and in particular whether it is supported at all on Linux or Windows (the only mention in the docs is Darwin).
It's a good sign that the Conan integration PR [2] was warmly received (if not merged, yet). I would hope that the system is extensible enough to allow hooking in other dependency managers like vcpkg. Using an external PM loses some of the benefits, but it also dramatically reduces the level of effort for initial adoption. I think bazel suffered from the early difficulties integrating with other systems, although IIUC rules_foreign_cc is much better now. If I'm following the code/examples correctly, Buck2 supports C++ out of the box, but I can't quite tell if/how it would integrate with CMake or others in the way that rules_foreign_cc does.
(one of the major drawbacks of vcpkg is that it can't do parallel dependency builds [3]. If Buck2 was able to consume a vcpkg dependency tree and build it in parallel, that would be a very attractive prospect -- wishcasting here)
[1] https://buck2.build/docs/developers/windows_cheat_sheet/ [1a] https://github.com/facebook/buck2/blob/738cc398ccb9768567288... [2] https://github.com/facebook/buck2/pull/58 [3] https://github.com/microsoft/vcpkg/discussions/19129
> There are not yet mechanisms to build in release mode (that should be achieved by modifying the toolchain).
> Windows/Mac builds are still in progress; open-source code is mostly tested on Linux.
Source: https://buck2.build/docs/why.
Looks like local mode just inherits whatever environment the buck daemon was spawned in.
The remote execution thing is configured with a docker image to run things in, and only specified files are coped into the container instance, so it's somewhat hermetic. Docker containers aren't really reproducible, and there's only one image per remote execution backend, so that's kinda the weakest link (especially compared to something like Nix's hermetic builds, where the build-visible filesystem only contains the things you declared as dependencies).
IIUC, the publicly available Remote Execution services out there are specified by docker, so we chose to have OSS buck2 align to that.
As noted, local mode doesn't do anything else at this point, but we've discussed exactly this to help developers identify dependency declarations earlier.
One thing I find annoying with all of these general, language-agnostic build systems though is that they break the "citizenship" in the corresponding language. So while you can usually relatively easily build a Rust project that uses crates.io dependencies, or a Python project with PyPi dependencies, it seems hard to make a library built using Bazel/Buck available to non-Bazel/Buck users (i.e., build something available on crates.io or PyPi). Does anyone know of any tools or approaches that can help with that?
If you want to see an approach of bazel to pypi taken a bit to the extreme you can have a look at tensorflow on GitHub to see how they do it. They don't use the above-mentioned building rule because I think their build step is quite complicated (C/C++ stuff, Vida/ROCm support, python bindings, and multiOS support all in one before you can publish to pypi).
I'm cautiously optimistic with this latest work. I'm glad at least this isn't some unsupported personal project but something official from Meta.
Buck2 is at a higher level than Shake - the rules/providers concepts pretty much force you into a pattern of composable rules. The fact that Meta has lots of languages, and that we were able to release those rules, hopefully means it's starting from the point of view of a working ecosystem. Writing those rules took a phenomenal amount of effort from a huge range of experts, so perhaps it was naive that Shake could ever get there on only open source volunteer effort.
https://bazel.build/external/module
This means your packages are just Git repos + BUILD files.
I mean, this is kind of the whole point. A language agnostic build system needs a way to express dependencies and relationships in a way that is agnostic to, and abstracts over, the underlying programming language and its associated ecosystem conventions.
If the output is libraries for some ecosystem (perhaps with bindings to something written in Rust or C), one needs to be able to build packages that others not invested in that build system can consume.
In plainer language, I'd say the observation/motivation is that not only do compiling and linking benefit from incrementality/caching/parallelism, but so does the build system itself. That is, the parsing of the build config, and the transformation of the high level target graph to the low level action graph.
So you can implement the build system itself on top of an incremental computation engine.
Also the way I think about the additional dependencies for monadic build systems is basically #include scanning. It's common to complain that Bazel forces you to duplicate dependency info in BUILD files. This info is already present (in some possibly sloppy form) in header files.
So maybe they can allow execution of the preprocessor to feed back into the shape of the target graph or action graph. Although I wonder effect that has on performance.
---
The point about Java vs. Rust is interesting too -- Java doesn't have async/await, or coroutines.
I would have thought you give up some control over when things run with with async/await, but maybe not... I'd like to see how they schedule the tasks.
Implementing Applicative Build Systems Monadically
https://ndmitchell.com/downloads/paper-implementing_applicat...
It has been remarkably convenient for our Python monorepo.
Since Pants 2 has a core in Rust, I wonder if you considered Starlark vs. Python? I can see advantages to Python, but it seems like there are good open source implementations of Starlark now.
With Pants 2 and Buck 2, now we're on the 3rd or 4th generation of the Bazel-like systems :)
I really appreciate tooling that is written in Rust or Go that produce single binaries with minimal runtime dependencies.
Getting tooling written in for example Python to run reliably can be an exercise in frustration due to runtime environmental dependencies.
Why have an OS at all if every program is just going to package everything it needs?
It feels like we cheapened out. Rather than get good & figure out how to manage things well, rather than driver harder, we're bunting the problem. It sucks & it's lo-fi & a huge waste of resources.
You’re probably thinking of static binary. I believe that OP is comparing a single binary vs installing the whole toolchain of Python/Ruby/Node and fetching the dependencies over the wire.
This would definitely sound like a conspiracy theory, but I'm quite sure that hardware vendors see this technological development as, at least, a fortunate turn of events...
As a long time Linux desktop user, I appreciate any efforts to improve compatibility between distros. Since Linux isn't actually an operating system, successfully running software built for Ubuntu on a Fedora box, for example, is entirely based on luck.
Where-as with the conventional dynamic library world one would just update openssl or whomever & keep going. Or if someone wanted to shim in an alternate but compatible library, one could. I personally never saw the binary compatibility issue as very big, and generally felt like there was a while where folks were getting good at packaging apps for each OS, making extra repos, that we've lost. So it seems predominantly to me like downsides, that we sell ourselves on, based off of outsized/overrepresented fear & negativity.
the computational model of a fleet of long-lived servers, which receive host/OS updates at one cadence, and serve applications that are deployed at a different cadence, is at this point a niche use case, basically anachronistic, and going away
applications are the things that matter, they provide the value, the OS and even shared libraries are really optimizations, details, that don't really make sense any more
the unit of maintenance is not a host, or a specific library, it's an application
vulnerabilities affect applications, if there is a vulnerability in some library that's used by a bunch of my applications then it's expected that i will need to re-deploy updated versions of those applications, this is not difficult, i am re-deploying updated versions of my applications all the time, because that is my deployment model
Beyond app delivery there's dozens of different utils folks rely on in their day to day. The new statically compiled world requiring each of these to be well maintained & promptly updated feels like an obvious regression.
Linux distros are thankfully moving towards embracing static linking, rather than putting their heads in the sand and pretending that dynamic linking isn't on its last legs.
Dynamic linking on *nix has nothing to do with 70s era programming languages.
Did you consider the possibility that the incompatibility between monomorphization (possibly the dumbest term in all of programming) and dynamic linking should perhaps saying something about monomorphization, instead?
Given that dynamic linking as a concept came out of the C world, it has everything to do with them.
> Did you consider the possibility that the incompatibility between monomorphization (possibly the dumbest term in all of programming) and dynamic linking should perhaps saying something about monomorphization, instead?
Yes, I considered that possibility.
What elements of dynamic linking do you see as being connected to "70s era programming languages"?
> Yes, I considered that possibility.
Then I would urge you to reconsider.
there is no practical downside to a program including all of its dependencies, when evaluated against the alternative of those dependencies being determined at runtime and based on arbitrary state of the host system
monomorphization is good, not bad
the contents of /usr/lib/whatever should not impact the success or failure of executing a given program
There's also the problem that dynamic loading relies on almost all the same mechanisms as dynamic linking, so you can't get rid of those mechanisms just because your main build process used static linking.
smaller executable sizes, shared code in memory, synchronized security updates, are all basically value-zero, in any modern infrastructure
there is no "growing chorus" of developers or users saying otherwise, it is in fact precisely the opposite, statically linked binaries are going extremely well, they are very clearly the future
"modern infrastructure" seems like a bit of a giveaway of your mind set. yes, i know that there's a lot of stuff that now happens by having your web browser reach out to "infrastructure" and then the result is displayed in front of you.
But lots of people still use their computers to run applications outside the browser, where "modern infrastructure" means either nothing at all, or it means "their computer (or mobile platform)". the techniques mentioned in this subthread are all still very relevant in this context.
the infrastructural model i'm describing doesn't require applications to run in browsers, or imply that applications are slower, actually quite to the contrary, statically linked binaries tend to be faster
the model where an OS is one to many with applications works fine for personal machines, it's no longer relevant for most servers (shrug)
end-user systems account for at best single-digit percentages of all systems relevant to this discussion
(mobile is not relevant to this discussion)
Not disagreeing that there many upsides to statically linking, but there are (other) situations where binary size matters. Rolling updates (or scaling horizontally) where the time is dominated by the time it takes to copy the new binaries, e.g.
> the model where an OS is one to many with applications works fine for personal machines, it's no longer relevant for most servers
Stacking services with different usage characteristics to increase utilization of underlying hardware is still relevant. I wouldn't be surprised if enabling very commonly included libraries to be loaded dynamically could save significant memory across a fleet .. and while the standard way this is done is fragile, it's not hard to imagine something that could be as reliable as static linking, esp in cases where you're using something like buck to build the world on every release anyway
yes: if you're a company that tends to only run 1-20 applications, no, the memory-savings probably won't matter to you. that matches quite a large number of use cases. but a lot of companies run way more workloads than anyone would guess. quite a few just have no cost-control and/or just don't know, but there's probably some pretty sizable potential wins. it's even more important for hyper-scalers, where they're running many many customer processes at a time. even companies like facebook though, i forget the statistic, but sometime in the last quarter there was quote saying like >30% of their energy usage was just powering ram. willing to bet, they definitely optimize for binary size. they definitely look at it.
there's significant work being put towards drastically reducing scale of disk/memory usage across multiple containers, for example. composefs is one brilliant very exciting example that could help us radically scale up how much compute we can host. https://news.ycombinator.com/item?id=34524651
i also haven't seen the very important very critical other type of memory mentioned, cache. maybe we can just keep paying to add DRAM forever and ever (especially with CXL coming across the horizon), but the SRAM in your core-complex will almost always tend to be limited (although word is Zen4 might get within striking distance of 1GB which is EPIC). static builds are never going to share cache effectively. the instruction cache will always be unique per process. the most valuable expensive fancy memory on the computer is totally trashed & wasted by static binaries.
there's really nothing to recommend about static binaries, other than them being extremely stupid. them requiring not a single iota of thought to use is the primary win. (things like monomorphic optimization can be done in dynamic libraries with various metaprogramming & optimizing runtimes, hopefully one's that don't need to keep respawning duplicate copies ad-nauseum.)
i do think you're correct about the dominant market segment of computing, & you're speaking truthfully to a huge % of small & mid-sized businesses, where the computing needs are just incredibly simple & the ratio of processes to computers is quote low. their potential savings are not that high, since there's just not that much duplicate code to keep dynamically linking. but i also think that almost all interesting upcoming models of computing emphasize creating a lot more smaller lighter processes, that there are huge security & managability benefits, and that there's not a snowman's chance in hell that static-binary style computing has any role to play in the better possible futures we're opening up.
the benefit is that a statically linked binary will behave the same on all systems and doesn't need any specific runtime support above or beyond the bare minimum
this is important if you want a coherent deployment model at scale -- it cannot be the case that the same artifact X works fine on one subset of hosts, but not on another subset of hosts, because their openssl libraries are different or whatever
static linking is not stupid, it doesn't mean that hosts can only have like 10 processes on them, it doesn't imply that the computing needs it serves are simple, quite the opposite
future models of computing are shrinking stuff like the OS to zero, the thing that matters is the application, security (in the DLL sense you mean here) is not a property of a host, it's a property of an application, it seems pretty clear to me that static linking is where we're headed, see e.g. containers
> smaller executable sizes, shared code in memory, synchronized security updates, are all basically value-zero, in any modern infrastructure
This highlights the fact that you're extremely focused on one particular model of development, one where a single person or group deploys software that they are responsible for running and maintaining - often software that they've written themselves.
This is, obviously, an extremely appropriate paradigm for the enterprise. Static linking makes a lot of sense here. Python's virtual environments are basically the approved workaround for the fact that Python was built for systems that are not statically linked, and I cherish it for exactly that reason. Use Go on your servers - I do myself! But that doesn't mean it's appropriate everywhere.
Sometimes developers in this mindset forget there's a whole other world out there, a world of personal computers, that each have hundreds or thousands of applications installed. Applications on these systems are not deployed, they are installed. The mechanism by which this happens (on Linux) is via distributions and maintainers, and dynamic linking needs to be understood as designed for that ecosystem. Linux operating systems are built around making things simple, reliable, and secure for collections of software that are built and distributed by maintainers.
I'm firmly on the side of the fence that says that dynamic linking is the correct way to do that. All the benefits you mention are just a free bonus, of course, but I care about them as well. Smaller executable sizes? Huge win on my 256 GB SSD. Synchronized security updates? Of course I care about those as an end user!
it's not that i forget about these use cases, it's that i don't really consider them relevant
tooling that supports industrial use cases like mine is not really able to support end-user use cases like yours at the same time
linux operating systems may have at one point been built around making things as you describe by distribution maintainers, but that model is anachronistic and no longer useful to the overwhelming majority of its user base, the huge majority of software is neither built nor distributed by maintainers, it is built and distributed by private enterprises
> it's not that i forget about these use cases, it's that i don't really consider them relevant
Yes, exactly! It's an extremely myopic vision. You've spent this long thread arguing against dynamic linking on the basis of what is only a small fraction of total human / computer interactions! By "not relevant" you mean not relevant to the enterprise. I grant that of course - but these uses cases are (by definition) relevant to hundreds of millions of PC users.
> the huge majority of software is neither built nor distributed by maintainers, it is built and distributed by private enterprises
The overwhelming majority of the software I run is built and distributed by maintainers. Literally, there are only a few exceptions, like static-built games that rarely or never change and are (unfortunately) closed source. I daresay that's true for the majority of Linux users - the vast majority of the software we install and use is not "built and distributed by private enterprises".
This reality is what Linux-on-the-desktop is built for. There are millions of people who are going to want to continue using computers this way, and people like me will continue contributing to and developing distributions for this use case, even if shipping static or closed-source binaries to Linux users becomes common.
like i'm not sure you understand the scale of enterprise linux. a single organization of not-that-very-many people can easily create and destroy hundreds of millions of deployed systems every day, each with a novel configuration of installed software. i've seen it countless times.
1. My point about Linux on the desktop is that there are in practice users like me who are already getting the (many) advantages of dynamic linking, and don't want to give up those advantages. To the point that some of us are going to support and work on distributions that continue the traditional Linux way in this area. In your view, the ecosystem has moved to software being built and distributed by private corporations. I don't think this has happened - on Windows software was always built and distributed this way; on (desktop) Linux it never was and largely still isn't!
2. My point about the desktop in general is that this use case matters to the vast majority of computer-using human beings much more than enterprise. The number of deployed containers that get created and destroyed every day doesn't change that fact, nor does the fact that Linux users are merely a tiny fraction of this desktop use case. This is what creates the myopia I was talking about - you're thinking about metrics like "number of systems deployed" whereas I'm thinking of number of human-computer interactions that are impacted. I don't think you can just discard what matters on the desktop or paint it as irrelevant. Desktop computing shouldn't be subordinate to the technical requirements of servers!
So to summarize the argument: (a) desktop use cases still matter because they comprise the majority of human-computer interactions, (b) dynamic linking and the maintainer model are the superior approach for desktop computing, and in fact complement each other in important ways, and (c) even if most desktop users can't take advantage of this model because of the dominance of closed source software and the corporate development model, desktop Linux can and does, and will hopefully continue to do so into the future.
i guess this is the crux of the discussion
linux desktop computing for sure _is_ subordinate to linux server computing, by any reasonable usage metric
i'm not trying to deny your experience in any way, nor suggest that dynamic linking goes away, or anything like that -- your use case is real, linux on the desktop is real, that use case isn't going away
but it is pretty clear at this point that linux on the server is wildly successful, linux on mobile is successful (for android), and that linux on the desktop is at best a niche use case
the majority of human interactions with linux occur via applications, services, tools, etc. that are served by linux servers, and not by software running on local machines like desktops or laptops
linux is a server operating system first and foremost
and where is the growth potential? is the industrial need going to become greatly newly empowered & helpful to this planet, to us? will it deliver & share the value potential out there? PC may be a smaller factor today, but i for one am incredibly fantastically excited to imagine a potential future 10 years from now where people start to PC again, albeit in a different way.
individual PCs have no chance. it's why the cloud has won. on-demand access wherever you are, consistent experience across devices is incredibly incredibly convenient. but networks of PCs that work well together is exciting, and we've only so very recently started emerging the capability to have nice easy to manage ops/automated multi-machine personal-computing. we've only recently emerged to maturity where a better, competitive personal computing is really conceivable.
it's been the alpha linux geeks learning how to compute and industrial players learning how to compute, and the invisible hand has been fat happy & plump from it, but imo there's such a huge potential here to re-open computing to persons, to create compelling interesting differently-capable sovereign/owned computing systems, that are free from so many of the small tatters & deprevations & enshittifications that cloud- that doing everyting on other people's computers as L-Users- unnerringly drops on us. we should & could be a more powerful, more technically-cultural culture, and i think we've severely underrated how much subtle progress there's been to make that a much less awful, specialized, painful, time-consuming, low-availability, disconnected effort than it used to be.
In a way they are. You deploy it to the store and then as people's computers download the update automatically.
A counter example to your claims about Linux is Android. Libraries are not shared between apps (beyond the android framework and libc). This is despite the fact that phones have limited storage.
You raise the world of personal computers. And I think dynamic linking is absolutely a choice that has huge advantages for these folks.
There's other realms too. Embedded software needs smaller systems, so the dynamic library savings can be huge there. Hyper-scaler systems, where thousands of workloads can be running concurrently, can potentially scale to much much much higher usages with dynamic linking.
It's a little far afield, but with systems like webasssembly we're really looking less at a couple orgs within a company each shipping a monolith or two, and we're potentially looking way more at having lots of very small functions with a couple helper libraries interacting. This isn't exactly a classic dynamic library, but especially with the very safe sandboxing built in, the ideal model is far closer to something like dynamic linking where each library can be shared than it is shared.
> Then I would urge you to reconsider.
Done. No change to my beliefs.
the overwhelming majority of software that would ever be built by a system like buck2 is written and deployed in an industrial context
the share of software consumers that would use this class of software on personal linux laptops is statistically zero
really, the overwhelming majority of installations of distros like fedora or debian or whatever are also in industrial contexts, the model of software lifecycles that their maintainers seem to assume is wildly outdated
So we make the trade off of software complexity for hardware complexity. Such is how life goes sometimes.
the size of a program binary is a distant secondary concern to this main goal
static compilation more or less solves this primary requirement, at the cost of an increase to binary size that is statistically zero in the context of any modern computer, outside of maybe embedded (read: niche) use cases
there is no meaningful difference between a 1MB binary or a 10MB binary or a 100MB binary, disks are big and memory is cheap
the optimization of dynamic linking was based on costs of computation, and a security model of system administration, which are no longer valid
there's no reason to be offended by this, just update your models of reality and move on
It's popular to be cynical & conservative, to disbelieve. That has won the day. It doesn't do anything to convince me it was a good choice or actually helpful, that we were right to just give up.
i won't try to refute your personal experience, but i'll observe it's relevant in this discussion only to the extent that your individual context is representative of consumers of this kind of software in general
that static linking provides a more reliable end-user experience vs. dynamic linking is hopefully not controversial, the point about security updates is true and important but very infrequent compared to new installations
definitely i am not using buck or bazel or whatever to build binaries that go on those little sticks
The impetus for dynamic linking on Multics and MTS was the ability to upgrade libraries without having to recompile software, and to reuse code not originally designed or intended for (e.g. different compilers or languages), let alone compiled with, the primary program. Both of these reasons still pertain, notwithstanding that some alternatives are more viable (e.g. open source code means less reliance on binary distribution).
"the primary program" is the atomic unit of change, it is expected that each program behaves in a way that is independent of whatever other files may exist on a host system
There are multiple problems with Python packaging which ought not exist, but are there and make lives of Python users worse:
* Python doesn't have a package manager. pip can install packages, but installing packages iteratively will break dependencies of packages installed in previous iterations. So, if you call pip install twice or more, you are likely to end up with a broken system.
* Python cannot deal with different programs wanting different versions of the same dependency.
* Python version iterates very fast. It's even worse for most of the Python packages. To stand still you need to update all the time, because everything goes stale very fast. In addition, this creates too many versions of packages for dependency solvers to process leading to insanely long installation times, which, in turn, prompts the package maintainers to specify very precise version requirements (to reduce the time one has to wait for the solver to figure out what to install), but this, in turn, creates a situation where there are lots of allegedly incompatible packages.
* Python package maintainers have too many elements in support matrix. This leads to quick abandonment of old versions, fragmented support across platforms and versions.
* Python packages are low quality. Many Python programmers don't understand what needs to go into a package, they either put too little or too much or just the wrong stuff altogether.
All of the above could've been solved by better moderation of community-generated packages, stricter rules on package submission process, longer version release cycles, formalizing package requirements across different platforms, creating tools s.a. package manager to aid in this process... PyPA simply doesn't care. That's why it sucks.
Node basically works fine. You get a huge node_modules folder, sure. But it works.
Python is a complete mess.
A well engineered virtualenv solves most Python problems.
pnpm and other tools deduplicates that
Come back ten years. We'll talk.
Let's go through the points which I was referring to:
"Python doesn't have a package manager. pip can install packages, but installing packages iteratively will break dependencies of packages installed in previous iterations. So, if you call pip install twice or more, you are likely to end up with a broken system."
"Likely" seems like a stretch here since it's pretty damned rare that I've come across this when using virtual environments. With a virtual environment, you have an isolated system. Why are you installing packages iteratively in the first place? Use a requirements.txt with the packages you need, then freeze it. If you end up with a conflict, delete the virtual environment and recreate a fresh one, problem solved.
"Python cannot deal with different programs wanting different versions of the same dependency"
It does when you're running your applications using virtual environments. Again, you say that it's irrelevant, but this is literally what this shit solves. I come from a world where multiple applications are run on separate docker containers so this doesn't really apply anyway, but if you had to run multiple applications on the same server you can set the PYTHONPATH env variable and venv binary to the virtual environment when running each application.
"Python version iterates very fast. It's even worse for most of the Python packages. To stand still you need to update all the time, because everything goes stale very fast. In addition, this creates too many versions of packages for dependency solvers to process leading to insanely long installation times, which, in turn, prompts the package maintainers to specify very precise version requirements (to reduce the time one has to wait for the solver to figure out what to install), but this, in turn, creates a situation where there are lots of allegedly incompatible packages."
Maybe I'm misunderstanding what you are saying here, but this seems like a retread of your first point with some casual opinions thrown in. If you delete the venv and re-install all the packages at once, shouldn't it resolve dependency issues? "Insanely long installation times"? Seems to be a lot quicker than maven or gradle in my experience, and much easier to use. I get a lot of dependency issues with those managers as well, so this doesn't seem to be a unique problem for python, if it really is a problem when using virtual environments.
"Python package maintainers have too many elements in support matrix. This leads to quick abandonment of old versions, fragmented support across platforms and versions."
I admit I don't know anything about this. Maybe it's true, but I imagine this is true for community packages of just about any language.
"Python packages are low quality. Many Python programmers don't understand what needs to go into a package, they either put too little or too much or just the wrong stuff altogether."
This is not only purely subjective opinion, it's not even one that seems to be common. Maybe that's true for less popular packages (and again, I'm not convinced it wouldn't be the same for less popular packages in other languages), but the ones most people use for common tasks I often see heralded as fantastic examples of programming that I should be reviewing to level up my own code.
"All of the above could've been solved by better moderation of community-generated packages, stricter rules on package submission process, longer version release cycles, formalizing package requirements across different platforms, creating tools s.a. package manager to aid in this process..."
I'm not familiar enough with the politics, culture, and process of maintaining Python's packages or package management system to speak to any of this. It seems like this would generally be good advice regardless of the state it's currently in. But these are broad, systemic solutions that require a revamp of the culture and bureaucracy of the entire package management system, a completely different set of tools than the ones that already exist (that would likely create backwards incompatibility issues), and no meaningful way to measure the success of these initiatives because most of your complaints are subjective opinions. Furthermore, at least half of your complaints seem to already be mitigated using virtual environments and industry best-practices, so I'm struggling to see where any of this is helpful.
The fact that the Buck team felt they had to do a from scratch rewrite to build the features they needed just goes to show how hard it is to design something robust in this area.
If there are any people in the Buck team here, I would be curious to hear if you all happened to evaluate waf before choosing to build Buck? I know FB's scale makes their needs unique, but at least at a surface level, it doesn't seem like Buck offers anything that couldn't have been implemented easily in waf. Adding Starlark, optimizing performance, implementing remote task execution, adding fancy console output, implementing hermetic builds, supporting any language, etc...
[1]: https://waf.io/
Personally, I always go for declarative CMake first, then waf as soon as I find my CMakeLists looking like something other than just a list of files.
I’ve considered before creating a simple declarative language to build simple projects like that with waf, but I don’t like the idea of maintaining my own DSL for such little benefit, when CMake works just fine, and everyone knows how to use it. I feel like I’d end up with my own little TempleOS if I decided to go down that rabbit hole.
The Nickel rationale doc has some thoughts on why this might be the right call: https://github.com/tweag/nickel/blob/master/RATIONALE.md#tur...
From my (limited) experience with another deliberately limited configuration DSL (CUE), I think more power in such DSLs will pan out better in the long run. Of course, it's not all one or the other: a powerful build DSL can still enforce useful discipline, and a Turing-complete language can still be thoughtfully designed around a special purpose. I think Nix demonstrates both pretty well, actually.
Contrast this with Gradle, which is currently digging itself out of a hole by forcing authors to declare all inputs and outputs of their tasks so it can serialize them, but you can literally do anything Java can throughout the entire process. This is the kind of Herculean task which is neatly sidestepped by tightly controlling the DSL environment (inputs/outputs) as does Nix.
I'd bet there are a more than a few repos that do get (at least) hundreds of commits as a highwater mark. My guess is lots of engineers + mono-repo + looming code-freeze deadline can do that like clockwork.
Edit: Robots too as sibling pointed out. A single human action may result in dozens of bot-generated commits
1) Automated refactoring
2) Automated merges when CI passes
Configs that can be generated should just be generated by the build.
But that's a different topic
- automatic security / vendoring updates (e.g. https://github.com/renovatebot/renovate)
- automated cross-repo syncs, e.g. Google has processes and tools that bidirectionally sync pieces of Google3 with GitHub repos
This depends entirely on the quality of dev tools available.
Also, commit =/= shipped code: you may have a automated commits and keep a human in the loop before shipping, by way of rejectable Pull-Request (or the proprietary equivalent).
A simple library upgrade will result in a wave of commits/bot-authored PRs
1. Human makes a change to a core library, changing it from v1 to v2
2. Bot identifies all call-sites and refactors to v2-equivalent, creating 50 PRs for 50 different teams.
One change, 51 commits.
[1] https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...
The monorepo case is also a little bit outside what I was originally talking about. I was mostly refering to individual services/libraries/apps
And sure, maybe having a restricted build definition (whether by using a restricted tool or by doing code review etc.) moves the complexity somewhere else, like into the actual code implementation. But it's easier to manage there. The build system is the wrong place for business logic, because it's not somewhere most programmers ever think to look for it.
I understand where the sentiment comes from, having seen one too many example of people struggling to implement basic logic in cmake or groovy, that would be a oneliner in python. But completely opening up the floodgates is not the right solution.
Escape hatches into GP languages can still exist but the interfaces to them need to be strict, and it’s better people see this boundary clearly, rather than limping around trying to do GP inside cmake and failing on correctness anyway. Everything else should like parent say just be a list of files.
Dependencies need to be declarative and operations hermetic.
Otherwise the spaghetti of complexity will just keep growing. Builds and tests will take forever due to no way of detecting what change affects which subsystem, what can be parallelized and even worse when incremental builds stop working.
By constraining what can be done, you also empower developers to do whatever they want, within said boundaries, without having to go through an expert build-team. Think about containers, it allowed every team to ship whatever they want without consulting the ops team.
That works if you have one team - of if all teams work the same way. If you have multiple teams with conflicting requirements[1], you absolutely should not constrain the build because you'd be getting in the way.
1. E.g. Team A uses an internal C++ lib an online service and prefers an evergreen version of it to be automatically applied with minimal human involvement. Team B team uses the same lib on physical devices shipped to consumers/customers. Updates are infrequent (annual), but have to be tested thoroughly for qualification. Now your build system has to support evergreen dependencies and versioned ones. If you drop support for either, you'll be blocking one team or the other from doing work.
That said, the more your build systems makes easy without having to write code the better.
One of the key requirements is that Buck2 had to be an (almost) drop-in replacement for Buck1 since there's no way we could reasonably rewrite all the millions of existing build rules to accommodate anything else.
Also Buck needs to support aggressive caching, and doing that reliably puts lots of other constraints on the build system (eg deterministic build actions via strong hermeticity) which lots of build systems don't really support. It's not clear to me whether waf does, for example (though if you squint it does look a bit like Buck's rule definitions in Starlark).
The documentation and examples for waf seem to be around building one project, in one language, with an output of statistics and test results. I am sure this is a simplification for education and documentation purposes, but it does leave a vague area around "what if I have more than 1 or 2 build targets + 5 libs + 2 apps + 3 interdependent helper libraries?"
Buck seems to be different in that it does everything waf does but also has clear `dep` files to map dependencies between various libraries within a large repository with many, many different languages and build environments.
The key thing here being, I suspect that within Meta's giant repositories of various projects, they have a tight inter-linking between all these libraries and wanted build tooling that could not only build everything, but be able to map the dependency trees between everything as well.
Pair that with a bunch of consolidated release mapping between the disparate projects and their various links and you have a reason why someone would likely choose Buck over waf purely from a requirements side.
As for another reason they likely chose Buck over waf. It would appear that waf is a capable, but lesser known project in the wider dev community. I say this because when I look into waf, I mostly see it compared against CMake. Its mental state resides mostly in the minds of C++ devs. Either because of NIHS (not invented here syndrome) or fear that the project wouldn't be maintained over time, Meta may have decided to just roll their own tooling. They seem to be really big on the whole "being the SDK of the internet" as of late. I could see them not wanting to support an independent BSD licensed library they don't have complete control over.
These are just my thoughts, I could be completely wrong about everything I've said, but they're my best insights into why they likely didn't consider waf for this.
However, if you’re going to build a custom package management system anyways, there’s no reason you couldn’t build it on top of waf. Again, the core is tiny enough that one engineer could realistically hold the entire thing in their head.
But I don’t think we’re going to get it right speculating here lol. I’m sure there was more to it than NIHS, or being unaware of waf.
Fixing that or moving to a daemon or whatever starts to take more time than just redoing it from scratch, and if the whole thing is 10k lines of python, it's something a domain expert can mostly reimplement in a week to better serve the fb specific needs.
And if it is too slow, there's profiling support for tracking down bottlenecks, and many different ways to optimize them. This includes simply optimizing your own code, or changing waf internal behavior to optimize specific scenarios. There's even a tool called "fast_partial" which implements a lot more caching than usual project-wide to reduce time spent executing Python during partial rebuilds in projects with an obscene number of tasks.
> Fixing that or moving to a daemon or whatever starts to take more time than just redoing it from scratch, and if the whole thing is 10k lines of python, it's something a domain expert can mostly reimplement in a week to better serve the fb specific needs.
Well, considering Buck just went through a from-scratch rewrite, I would argue otherwise. Although, to be fair, that 10k count is just for the core waflib. There are extra modules to support compiling C/C++/Java/etc for real projects.
(also, waf does have a daemon tool, but it has external dependencies so it's not included by default)
Based on what, the idea that waf fits their needs better than the tool they wrote and somehow wouldn't need to be rewritten or abandoned?
> Even if the interpreter were 100x slower to start and execute than it currently is, that time would be negligible next to the time spent waiting for a compiler or other build task to complete.
This wrongly assumes that clean builds are the only use case. Keep in mind that in many cases when using buck or bazel, a successful build can complete without actually compiling anything, because all of the artifacts are cached externally.
> There's even a tool called "fast_partial" which implements a lot more caching than usual project-wide to reduce time spent executing Python during partial rebuilds in projects with an obscene number of tasks
Right, the point that this is a concern to some people, and that there's clearly some tradeoff here such that it isn't the default immediately rings alarm bells.
But discussions about optimization are pointless without real world measurements and data.
Especially considering how poor Python's support for shared memory concurrency is...
With waf, the build system is trivially included in the source, and so your project always uses the right version of waf for itself.
There’s no way Waf can handle code bases as large as the ones inside Facebook (Buck) or Google (Bazel). Waf also has some problems with cross-compilation, IIRC. Waf would simply choke.
If you think about the problems you run into with extremely large code bases, then the design decisions behind Buck/Bazel/etc. start to make a lot of sense. Things like how targets are labeled as //package:target, rather than paths like package/target. Package build files are only loaded as needed, so your build files can be extremely broken in one part of the tree, and you can still build anything that doesn’t depend on the broken parts. In large code bases, it is simply not feasible to expect all of your build scripts to work all of the time.
The Python -> Starlark change was made because the build scripts need to be completely hermetic and deterministic. Starlark is reusable outside Bazel/Buck precisely because other projects want that same hermeticity and determinism.
Waf is nice but I really want to emphasize just how damn large the codebases are that Bazel and Buck handle. They are large enough that you cannot load the entire build graph into memory on a single machine—neither Facebook nor Google have the will to load that much RAM into a single server just to run builds or build queries. Some of these design decisions are basically there so that you can load subsets of the build graph and cache parts of the build graph. You want to hit cache as much as possible.
I’ve used Waf and its predecessor SCons, and I’ve also used Buck and Bazel.
You mean, multiple gigabytes for build metadata, that just says things like that X depends on Y and to build Y you run command Z?
I highly doubt that, after doing all those tricks, you still end up with 100GB - 1TB of build data.
I guess the downside is that the system would be more complex overall, but you can probably get 80% of the result with not so large changes
Memory management, broken build scripts, etc. can all be handled with Waf as well. In the simplest case, you can just wrap a `recurse` call in a try catch block, or you can build something much more sophisticated around how your projects are structured.
Note, I'm not trying to argue that Google/Facebook "should have used X". There are a million reasons to pick X over Y, even if Y is the objectively better choice. Sometimes, molding X to be good enough is more efficient than spending months just researching options hoping you'll find Y.
I'm just curious to know if they did evaluate Waf, why did they decide against it.
Isn‘t that already a no-go, to write a performance critical system in a slow programming language?
I’ve seen some projects with 100MB+ ninja-files that even ninja itself, proud for being written in optimized c++, takes a second or two to parse on each build invocation. Convert that to python and you likely land in 5-20 sec range instead. Enough to alt-tab and get distracted by something else. Google code base is likely even larger than this.
A background daemon that holds the graph in memory would probably handle it. In the big scheme such a design is likely better anyway. But needs a big upfront design and is a lot more complex than just reparsing a file each time.
Side note: For some, even the interpreter startup is annoying. Personally I find it negligible, especially after 3.11 you can almost claim it’s snappy.
Having spent a lot of time with Buck and Bazel, there are just so many little things you run into where you go, “Oh, that explains why Buck or Bazel is designed that way.” These design decisions permeate Buck and Bazel (Pants, Please, etc.)
I just don’t see how Waf can be used as a base. I really do see this as a new “generation” of build systems, with Buck, Bazel, Please, and Pants, and everything else seems so much more primitive by comparison.
The thing about waf is that it’s more designed like a framework than a typical build tool. If you look at the code, it’s split into a core library (thats the <10k loc I estimated), and additional tools that do things like add C++ or Java build support.
That’s one of the reasons I like Waf, since it becomes a powerful toolkit for creating a custom build system once you strip away the thin outer layer. There is no one-size-fits-all build system, so a tool that can be molded like waf is very powerful imo.
I guess it’s hard to get that point across without experiencing it. There are just so many good design decisions everywhere. For example, extensibility comes easily because task generator methods are “flat”, and ordering is implemented via constraints. This means you can easily slip your own functions between any built in generator method to manipulate their inputs or outputs. It’s like a sub-build system just for creating Task objects.
Also, I don’t want to give the impression that I think waf would have been a better choice for these companies. I’ve kind of been defending it a lot in this thread, but my original point/question was just to know if they evaluated waf/what they thought about it. After so many comments I feel like I might be coming off as hostile… which isn’t my intention.
Waf does seem like a “do things as you like” framework, and I think that notion is antithetical to the Buck and Bazel design ethos. Buck and Bazel’s design are, “This is the correct way to do things, other ways are prohibited.” You fit your project into the Buck/Bazel system (which could be a massive headache for some) and in return you get a massive decrease in build times, as well as some other benefits like good cross-compilation support.
One fundamental part of the Buck/Bazel design is that you can load any arbitrary subset of the build graph. Your repository has BUILD files in various directories. Those directories are packages, and you only load the subset of packages that you need for the targets you are actually evaluating during evaluation time. You can even load a child package without loading the parent—like, load //my/cool/package without loading //my/cool or //my.
The build graph construction also looks somewhat different. There is an additional layer. In build systems like Waf, you have some set of configuration options, and the build scripts generate a graph of actions to perform which create the build using that configuration. In Buck/Bazel, there is an additional layer—you create a platform-agnostic build graph first (targets, which are specified using rules like cc_library), and then there’s a second analysis phase, which converts rules like “this is a cc_library” into actual actions like “run GCC on this file”.
These extra layers are there, as far as I can tell, to support the goals of isolating different parts of your build system from each other. If they’re isolated, then you have better confidence that they will produce the same outputs every time, and you can make more of the build process parallelizable—not just the actual build actions, but the act of loading and analyzing build scripts.
I do think that there is room to appreciate both philosophies—the “let’s make a flexible platform” philosophy, and the “let’s make a strict, my-way-or-the-highway build system” philosophy.
Did FB fork Bazel in the early days but retain basically everything about it except the name? Why didn't they just...adopt Bazel, and contribute to it like any other open source project?
Buck (https://github.com/facebook/buck) has been open sourced for nearly 10 years now.
The lore I've heard is that former Googlers went to Facebook, built Buck based on Blaze, and Facebook open sourced that before Google open sourced Blaze (as Bazel).
The first pull to the Buck github repo was on May 8, 2013 (https://github.com/facebook/buck/pulls?q=is%3Apr+sort%3Acrea...). The first to Bazel was Sep 30, 2014 (https://github.com/bazelbuild/bazel/pulls?q=is%3Apr+sort%3Ac...).
Amusingly, the cake was in German but most of the Bazel team didn't really speak German. But it was yummy.
Skylark was a later evolution, after the python scripts grew out of control, and a cue that fb took from Google long after Buck had been near-universally deployed for several years.
Over time Facebook has been working to align Buck with Bazel, e.g. the conversion to Starlark syntax so tools such as Buildozer work on both systems. I believe Buck2 also now uses the same remote execution APIs as Bazel, but don't quote me on that.
In the years that followed folks left Google and joined other companies and created similar build systems because blaze had a lot of advantages at scale. Facebook made Buck, Twitter made Pants. Blaze was still closed source inside Google. They all used the same python looking language.
In 2012 Twitter open sourced Pants: https://blog.twitter.com/engineering/en_us/a/2016/the-releas...
In 2013 Facebook open sourced Buck: https://en.m.wikipedia.org/wiki/Buck_(software)
In 2015 Google finally open sourced most of blaze, but renamed it bazel for copyright reasons. Some might argue they waited too long because clearly there was a lot of demand for such a system. :)
After that Twitter (mostly?) migrated to bazel and Facebook sort of stalled out on Buck. But then recently they decided to rewrite it from scratch to fix a lot of the architecture problems resulting in Buck2.
Buck2 looks pretty impressive and hopefully it gets the bazel folks moving faster. For example the analysis phase in bazel is very slow even inside Google, and Buck2 shows an alternative design that's much faster.
On my machine `buck2 clean && time buck2 build :cxx` takes 6.2 seconds.
`bazel clean && time bazel build :cxx` takes 19.9 seconds.
Compiling gazebo v0.8.1 (/home/jwb/buck2/gazebo/gazebo)
error[E0554]: `#![feature]` may not be used on the stable release channel
--> gazebo/gazebo/src/lib.rs:10:49
error: no such command: `+nightly-2023-01-24`
Cargo does not handle `+toolchain` directives.
Did you mean to invoke `cargo` through `rustup` instead?They've clearly learned from Bazel though! I like the idea of not needing Java to build my software, and Starlark is battle tested / might make transitioning off Bazel easier.
Even Facebook's Buck launch blog does not offer this story of Bucks lineage and although the author worked on the Closure compiler at Google that is not all of Blaze.
https://engineering.fb.com/2013/05/14/android/buck-how-we-bu...
Also that seems an interesting thing an independent person could write about, but whatever claims Meta made on a topic like that would be heavily scrutinized. Benchmarking is notoriously hard to get right and always involves compromises. It's probably not worth making a claim vis a vis a "competitor" and triggering backlash. If it's significantly faster than Bazel that will get figured out eventually. If not the tool really is aimed at Buck1 users upgrading to Buck2 so that is the relevant comparison.
And use the compile command json file to power clangd. I'm not a vscode person but I would hope the vscode c++ plugin would support that
I'm not well-versed on the internal implementation details of bazel's depsets, but one interesting thing about tsets that may further differentiate them is how they are integrated into the core, specifically that we try hard to never flatten them there. The main two places this comes up are: (1) when an action consumes a tset projection, the edges on the DICE graph (our incremental computation edges) only represent the direct tset roots that the action depends on, not the flattened full list of artifacts it represents and (2) when we compute the input digest merkle tree for uploading an actions inputs to RE, that generally doesn't require flattening the tset as we cache the merkle trees for each tset projection node and can efficiently merge them.
Is anyone using Buck/Bazel and also using frameworks like Spring, or React, for example?
I really liked the cross-language aspect of Bazel. Having one command that could compile everything and produce a deployable container in a highly reproducible way is great. It really cut down on "what's your compiler/tool version etc."-type back-and-forth during debugging with other engineers.
The bazel JS/TS rules were tough to work with when we first started using it for JS (2018 I think), especially since we were using create-react-app at the time, and that didn't mesh well with the way bazel wants to work. It's gotten a lot better though.
If I was making the choice from scratch in a new company/codebase, I think it'd really depend on the team. You kind of need broad-based buy-in to get the full benefits IMO.
Regarding 3rd party packages, if you update them regularly and they are depended on by intensive builds, that’s going to be the reality in any build system worth using. A build is a DAG and if one node changes then it’s children must too, if you want any guarantee that the update isn’t going to break fresh builds. As with any build, if you want fewer rebuilds you need to be diligent with assigning dependencies to things that need them (so unrelated things don’t rebuild), and when things can be divided and conquered then they should be.
Bazel it isn’t perfect at reproducibility and thats what made me abandon ship. It isn’t perfectly sandboxed, for instance - it still uses your system compilers, still has access to your system libraries. This means that you can accidentally depend on something without specifying it, which violates my requirement for true reproducibility.
These days we use Nix, which is ultimately a better environment for developing reproducible builds if you’re happy to shun Windows. It is also a superior language to Starlark IMO - declarative makes much more sense than imperative in something that has to later resolve to a DAG. In terms of using it as a build system, it’s lacking unless you put in the legwork. The default is to use other build systems within Nix, but that leads to a lot of unnecessary rebuilding during development, so we just span up our own Bazel-like library for Nix to do the heavy lifting.
Overall, I would say that it has been very much worth it, as it eliminates some classes of developer problems almost entirely that come up in companies of any size (e.g. "works on my machine"). I also feel when it comes to time spent setting it up, it's also a net positive over alternative systems where we would've had to spend time tuning e.g. GH Actions CI caches or make Docker build more reproducible.
In my experience Buck was spending a huge amount of time in GC, so this doesn’t surprise me. It must have been (ab)using Java in such a way that massive amounts of stuff were sprayed across the heap.
My biggest gripe with Bazel is how when you're off the beaten path suddenly it feels like the ecosystem really doesn't want you to just solve problems yourself. Meanwhile in this Buck2 documentation, directly talking about adding good support for tools outsides of community-provided things.
I still am not a superfan of the awkward way that custom implementations get declared (which I think comes from needing to support super-giant projects? But it's jsut awkward) and all the naming suffers from Google-like "we cannot call them functions but must call them factories" NIH things... but at least there are clear docs.
Edit/Add: cf the part about language rules in this comment from a person who says they're a former bazel developer: https://news.ycombinator.com/item?id=35477309
* The fact that it is written in a compiled safe language is a breath of fresh air. I personally like Java the language and understand why Bazel was originally written in Java and how it has done a great job at "hiding" it, but it's still there. In particular, Java's memory and threading models has been problematic for certain scenarios. (I haven't kept up with the language advances and I believe there are new ways to fix this, but adopting them would require a major overhaul of Bazel's internals.) Plus Bazel being written in Java prevents it from being adopted in smaller projects that are /not/ written in Java--a bummer for the whole open source ecosystem.
* The complete separation of language rules from the core is great. This is something that Bazel has wanted to achieve for a long time, but they are still stuck with native C++ and Java rules (it's really hard to rewrite them apparently). Not a huge deal, but in Buck2's case, their design highlights that it's clean enough to support this from day one.
* The "single" phase execution is also nice to see. Bazel used to have three phases (loading, analysis, and execution) and later managed to interleave the first two. However, the separation is still annoying from a performance perspective, and also introduces some artifacts in the memory model.
* It's good that various remote execution facilities as well as virtual file systems have been considered from day one. These do not matter much... until they do, at which point you want the flexibility to incorporate them. Bazel used to have this in the Google-internal version (there is that ACM paper that explains this), but the external version doesn't. For example, there is a patch to support an output tree on a lazy file system courtesy of the bb-clientd project, but after years it hasn't been upstreamed yet.
* And lastly, it's also great to see that what they open sourced is what they use internally. Bazel isn't like that: Google tried to open source a "cleaner version" by removing certain warts that were considered dead ends... and that has been both good and bad. On the one hand, this has been key to developing Starlark to where it is today, but on the other, this has made it hard for certain communities to adopt Bazel (e.g. the Python rules were mostly unusable for a really long time).
Now, a question: Buck2 uses the Starlark language, but that does not imply that they implement the same Build APIs to support the rules that Bazel has. Does anyone know to what extent the rules are compatible between the two? If Buck2 supported the Bazel rules ecosystem or with minor changes, that'd be huge!
* At the BUILD/BUCK file level. I imagine you can get close, but there are details between each other that will be hard to overcome. Buck2 doesn't have the WORKSPACE concept, so that will be hard to emulate. Bazel doesn't have dynamic dependencies which means that things like OCaml are written as multiple rules for a single library, while Buck2 just has a single rule. I think it should be possible to define a macro layer that was common enough so that it could be implemented by both Bazel and Buck2, and then open source users could just support the unified Bazck build system. * At the rules level. These might be harder, as the subtle details tend to be quite important. Things like tsets vs depsets is likely to be an issue, as they are approximately the same mechanism, but one wired into the dependency graph and one not, which is going to show up everywhere.
[0] https://github.com/facebook/buck2/search?q=nixos&type=issues
Actually, the current tip of trunk for buck2 can't build on NixOS right now with buildRustPackage due to a problem with prost. I should probably file a root cause issue about that soon...
crane.dev with nightly-2023-03-15 from oxalica/rust-overlay built here okay (after I figured out all the BUCK2_BUILD_PROTOC, missing Cargo.lock, etc tricks).
I did have some weird trouble with trying to import buck2 into my flake as a non-flake input, with complaints about "failed to resolve patches" for prost, but putting the flake.nix into the buck2 source tree worked.
- https://github.com/thoughtpolice/buck2-nix/tree/main/buck/ni...
You should mostly be able to just 'callPackage' that and have it work if you have rust-overlay applied. Check the corresponding 'flake.nix' if you need it.
The prost issue with buildRustPackage is preventing an upgrade, so ideally the minor patches to prost can get upstreamed, but I need to (again) file a ticket about moving this along.
Also, in case of our builds, we can benefit only so much from being faster during build phase because it is all the other bits like SonarQube scans, pushing artifacts to Artifactory, misc housekeeping bits, annoyingly slow Octopus deployments etc, that add most of the time to the long deployment cycle. Sometimes I think if a dedicated Go utility that takes care everything build related (parallelizing when possible) would make things faster; it will have the full picture after all. But then we will be reimplementing all the features of these various tools which is maybe ok at FB scale but would be too much for a smaller shop.
And as soon as you have to manage PRs for multiple repos with a new cross-cutting feature or scheduling changes in the correct order you understand why they are so appealing.
That said, bazel’s set of prepublished rules, reliance on the jdk, etc, make it not worth the burden, imo/e.
I think less ambitious, but similar tools are where it’s at. We use please for this reason, and are generally quite happy with how it balances between pragmatism and theory.
In any event, having your build tool be a single binary is a major win. I’d rather use make than anything written in python or Java just because I don’t have to worry about the overhead that comes with those other tools and their “ecosystems”.
I've yet to try BUCK on small projects though - I personally default to Makefiles in that case.
On thing I definitely wouldn't use it is for Android development. The Android Studio integration is much worse than gradle's and adding external dependencies means you have to make BUCK modules for each one.
I would however use it for large-scale projects or projects with more than a dozen developers.
Make has quite a few flaws, but it is near universal and a good location to centralize all required project work. Even if the entire task is defined in a shell script elsewhere. That being said, I have being looking longingly at just[0] which is not just concerned with compiling C, but has task automation builtin from the beginning.
Deployment can be done with `run`, but again I wouldn't recommend it for deployment to real environments, starting a local devserver with run is a common pattern though.
Build systems like maven can read from the entire filesystem (e.g. it reads from ./~m2) and you might end up with artifacts that depend on the state of the machine. This makes debugging production issues harder. You can of course be careful with other build systems to stay reproducible but it's easy to make mistakes and buck will enforce that you don't.
Companies like Meta and Google have huge monorepos and are using multiple languages. It's common for developers to deal with multiple languages and for projects to depend on other languages. Buck can deal with that very naturally and avoids the engineers to deal with multiple build tools.
There are other upsides and downsides listed on their website.
Here’s how I used Bazel (and how I now use Nix).
I am provided a configuration file that specifies what a given instance of my program must do. I use language A that can natively understand this configuration to code-generate a file in language B (which is significantly more suited to the performance requirements of the program)
This file is then built along with generic program code. It is used to process a lot of data.
As an interface to this program, I have a HTTP interface that can communicate with it. It needs to understand the kinds of outputs the program will produce, so some of this HTTP interface is also code generated in language A. The interface is interactive so typescript is generated and then compiled too.
In order to process the output from the program, I need to produce extensions for languages that users of the output use: Python and R. These need to understand the kinds of data being used, so are also code generated - then built. They’re then tested against requirements defined by the config (so the tests are also code generated).
Each of these stages have dependencies required when building and dependencies required when running, and there are several languages involved.
I also need to be able to express the entire process as a function, because it’s something that needs to be run very frequently on different configs - sometimes in collections of configs that need everything built at once. It needs to be build on several different machines, sometimes desktops and sometimes remote servers, sometimes on clients’ hardware (depending on the needs). I need confidence that something that works on my development machine will work, in entirety, on a completely different machine with minimal prior setup. And I need it to be easy to do, easy to maintain, and I don’t want to mess around with many different build systems that have entirely different use cases, entirely different ideas of how build/runtime environments should be handled, entirely different languages to configure them - and many of them are rigid and don’t have a concept of functions, they’re just “state your dependencies, kthxbye”.
All of the above is absolutely trivial with bazel if you know where to tread lightly (e.g. surrounding Python environments). It’s also very easy with Nix once you get used to it, and you don’t need to tread lightly there - it has stronger guarantees.
In Nix, the sequence of events that lead to your build are figured out on the language level. A build is this sequence of events.
In Starlark, you have to do things in a specific order - and that can get messy. It has a bunch of built-in phases that mean WORKSPACE, .bzl and BUILD files all have different levels of ability. A workspace can define a dependency (e.g. a http_archive), load a file into bazel, then call a bazel function from it. If you want to do that outside of a huge unmaintainable file, you have to break the steps down into multiple bzl files - one that does the download, then another that loads the dependency and calls it. And then you can call the latter from your workspace. BUILD files can do neither of the above and can only really define builds. So you end up with awkward file arrangements to load the right dependencies.
In Nix, anything you evaluate is a Nix variable. A function is a variable, a derivation (target) is a variable. You can refer to it natively and do what you want with it.
In Bazel, everything is referred to in a global register via labels, which are just strings. If you want two versions of the same thing that are the result of a function call, you have to give them different labels, so you can end up screwing around with string manipulation to generate unique identifiers.
This also feeds into a deeper issue. Bazel loves its global registry. Names from your dependencies are also your dependencies - so if the author of a project decides it hates the users and wants to enforce a “com_github_owner_repository” as its workspace name (because who doesn’t love Java-style conventions), you’re stuck using “@com_github_owner_repository//:library” throughout your codebase, anywhere you rely on that dependency. And to load in that dependency’s own dependency, you have to either load a special file and run special functions that the author provides, or you have to copy a bunch of code into your WORKSPACE file, making sure you give things the naming conventions used within the dependency.
In Nix, you name things whatever you want. If something you depend on has dependencies, it manages them itself - you can override them if you want.
Bazel doesn’t have much of a registry of third party repos. The onus is on you to scour the web, find an archive you want, grab the hash, and sometimes write your own build file to explain how it gets put together.
Nix has a truly vast repository of dependencies, which can be library dependencies or programs (in the latter, it’s the largest repository of applications out there - bigger than even debian and arch provide).
Bazel is maintained by Google, and maintainers make it very clear that they are spread thinly and don’t enjoy doing it. Some projects are completely stalled due to lack of maintenance. When bugs are found in the internals, they don’t get fixed for a long time. It’s based on Blaze, just like VSCode is based on Visual Studio, and there’s a feeling that it is being held back when things that could benefit users would be incompatible with their own system. There’s always the looming apprehension that Google are going to abandon it as soon as the next shiny thing comes along, or as soon as the people who work on it get promoted. Want to submit a PR? Sign this agreement first, then we might look at your PR, then we might merge it, might say no, but (note: speculation) an eerily familiar yet unattributed push might arrive a few months later from a googler.
Nix is full FOSS, with the standard open process you’d expect.
In favour of Bazel.
In Bazel, because you’re expected to follow a certain flow, errors are friendlier and easier to figure out. Bazel knows what you’re supposed to be doing and can help. When it builds things and fails, the intermediate files are saved in an easy-to-find place within your local directory (as a symlink) so it’s easy to explore and figure out the issue.
In nix, a derivation (target) truly resolves down to the tiny operations that put it together. It doesn’t know what you’re trying to do with it, so a typo at the topmost definition of a derivation can end up giving you horrific errors about low level list operations. If an error occurs in the code you’re building, intermediates can be found by copying a huge nix store path from the terminal and navigating to it, but it is a laborious process.
Starlark is based on Python, and it’s familiar to a lot of people. The overhead of learning the language is just a case of “oh it’s Python but without X, Y, Z, and… without fstrings!?”.
Nix is an entire language, with its own conventions, its own library, and its own syntax. It’s easy enough to learn if you’re used to picking up new languages (and honestly is one of my favourite languages now), but that’s an overhead. Most people think imperatively and eagerly, and nix is functional and lazy, so it can be hard to figure out how to accomplish something.
Bazel has an actual build system. It has native support for a bunch of languages. Nix is very much DIY - if you’re using make or CMake then the mkDerivation can do a lot of things for you, but for a developer using the two technologies together is unpleasant. We went as far as making our own build system in Nix that invoked e.g. gcc directly rather than try to work with cmake.
Interesting, so twice the bang for your buck.
stopped reading there.
The only reason people use it is to have an excuse to rewrite things in something new, free from "legacy".
In particular the idea of writing something entirely generic that works for everything is a waste of time. The build system should be tailored to building your application in the way that matters to you and making the best of the resources that you have.
They're all made to follow the same conventions so neither of them specifies anything redundant.
How about doing away with the build system entirely? Build systems seem like something that shouldn't exist. When I create a new C# .NET app in Visual Studio 2019, what "build system" does it use? You might have an academic answer, but that's beside the point. It doesn't matter. It just builds. Optimizing a build system feels like a lack of vision, and getting stuck in a local maxima where you think you're being more productive, but you're not seeing the bigger picture of what could be possible.
Building software can be complex sometimes, and _pretending_ it's a "microoptimization" to care about that is unnecessarily diminutive.
It's like if I said "I don't really know to understand the details of how a CPU works in my daily job. What about we just removed CPUs entirely?".
Is it worth putting those resources into a new build system? Apparently Facebook thinks so. You don't have to agree, you can use Visual Studio.
It just seems like you say "if I don't care about build systems, nobody should".
Whatever code we write as developers is only very tip of a the iceberg of software that comprises any substantial application. Controlling what goes into that iceberg, and how it's assembled is an essential part of the engineering of software. The details and quality of your build system determine the composition and construction of that iceberg, not to mentioned the reliability and velocity of your development process.
Even 'basic' local build systems like CMake, maven/gradle/ivy, sbt, lein, cargo, go, ... bridge dependency management and task execution. They decide what goes into the software artifact you ultimately distribute (or deploy), and how that's assembled.
At the scale of buck, bazel, ... tools of that shape are necessary to make forward progress in that's composed of internal dependencies that are managed by different teams, written in different languages, targeting a variety of environments, that are so numerous they require distribution to complete in reasonable timeframes, and require absolute.
I'm not VS/C# user, but MSBuild is definitely a build system, and both in and the developer definitely have care about these complexities, even if they come under the heading of "IDE" instead of "Build System".
Also:
As the joke in my first comment implied, if you can't identify the build system, you're probably the build system
This isn't really just a FAANG thing. I've seen this throughout my career before the FAANGs at much smaller places, but projects needing multi-platform/multi-language/multi-arch scenarios.
You do have a point, that the issue is that one shouldn't have to learn the depths of a build system to do what one needs. The trade off is how are the knobs exposed to a developer.
Another way to think about it, these domain specific languages like Starlark and otherwise are just where the knobs are being stored. As much as Visual Studio ends up writing these to vsproj/vcproj or msbuild files, it's just that these are stored in another form.
Whether you should care about the details and knobs of your build system is strictly a function of the first kind of scaling. Every team building software of appreciable size should think a bit about their build system, and even small teams might benefit from Buck or friends if their iceberg is big enough.
Supposedly the C# compiler can compile millions of lines per second, but my not millions of lines of code project takes a minute to compile so it must be wasting time doing something, and could use some optimization.
Refining individual ecosystems to be effortless to use is one thing, and it’s a goal worth pursuing for the benefit of those who are happy in that ecosystem. For those who rely on bringing different technologies together, though, there can be a lot of complexity and nuance.
The point of these projects is to take that complexity and nuance and provide an effective “super ecosystem” to piece them together in. That’s why some of the biggest proponents of these technologies are Google and Meta - they have many interacting parts in many different languages that need to come together and work effectively.
It's nice that you've lived in a world where you haven't had to concern yourself with the concerns of how your code is built, but please understand that some of us actually need or want to delve into this. It can't always be magic.
That's a build system!
> I can just use a BASH script
Yep you can. Many build systems start out like that. Complexity tends to creep in.
...msbuild?
I mean I don't know how to use visual studio to build software, but I get along fine, so why should we optimize for your preferred interface and not another one that might be better?
They do mention that it supports virtual file systems through Sapling, which now encompasses their long known EdenFS. I'd like to know more but right off the bat Sapling says it is:
> Sapling SCM is a cross-platform, highly scalable, Git-compatible source control system.
Git compatible eh? Thats a lot less foreboding. (It is however a total abstraction over git with it's own distinct cli tools.)
I hope there are some good up to date resources on Sapling/EdenFS. I have heard some really interesting things about code kind of auto getting shipped up to the mothership & built/linted proactively, it just at commit points, which always sounded neat, and there seem to be some neat transparent thin checkout capabilities here.
Buck2 notably though tries to be a good persistent incremental build system, and it needs much more awareness of things changing. It does integrate with Sapling for these kind of reasons.
So the boundaries are a bit blurrier than they used to be between scm/build/ci/CD tools.
Buck2 is actually used internally with Git repositories, so using Sapling / hg is definitely not a requirement