Microsoft Build Accelerator – open-source build engine for large systems
github.com
github.com
I know this wasn't supposed to be a humorous announcement but I couldn't help laughing out loud at that! Kudos to the managers at Microsoft who now seem to be asking "Why not?" instead of "Why should we?" when the topic of releasing code to the community is raised.
Are you critical of the size of the monorepo, or the number of builds?
Former employer was a big C++ shop in finance. Of around 1000 employees, roughly 3/4 of those were developers. They definitely could take advantage of something like this. I dont know how many 100s of million of LOC they have between C++ and later C#, but I was responsible for around 3 million alone (largely generated). A full coordinated firm wide rebuild could take weeks.
There are many many companies out there that have huge megarepos.
There's the number of compilation units within a lib vs overall. Typically you can parallelize within a module, but not externally unless you have some smarts.
Edit: I use module in this sense as a producible result, not the future language concept of modules.
We (the developers) were tasked with enormous responsibilities. I was back office, but responsible for managing a client/server for all non security reference data. There were easily over 200 data objects modelled. No direct DB access was allowed except for the owning service. Although it ended up around 3M LOC, it wasnt as bad as it seems, because only about 10% was manually written code. A lot was generated C++ (and aside: I was able to write a C++ wrapper around a generic API that exposed some 300 types through a home grown reflection esque API amd the Python wrapper never had to be updated necause it could use this reflection and run time cose generation to generate a strongly typed API at runtime; to my knowledge, the Python code has remained unchanged since about 2006 despite the underlying C++ API changing constantly, when I first wrote it - I get occasional updates from former coworkers).
The big problem was the dependency management and scale. At least at the time I was there, neither were done well.
Scale was a problem because of tight coupling between libs. Upgrade a core lib? Everyone had to rebuild. Want to upgrade a 3rd party dependency? Firm wide rebuild that took a min 2 weeks. It was a mess. We were supposed to be client/server to minimize dependencies, but we so tightly coupled our clients to our servers, we just exacerbated the problem. A few us could handle multiple client versions with a single server, but most couldn't. Don't recommend.
I wouldn't be surprised if they had at least 50 million LOC in total. Actually, now that I think of it, 100 million was more likely. And that was almost 10 years ago...
I mean we had Ballmer under whom you'd likely never have seen anything (or only trivial, non monetizable things) open sourced and likely no Linux support.
Now under Satya, we have MS open sourcing lots of projects and embracing Linux as a first class citizen. Probably far a complete list of Linux support, off the top of my I can name: VS Code, building for Linux via VS proper, Cmake support, WSL, SQL Server on Linux, SQL server odbc drivers, .Net Core and Linux support in Azure.
Even if projects are completely open, such as WSL and connhost, they have github projects at least for bug tracking that allows end-users to directly interact with the teams responsible for those projects. Personally, I've filed several issues against WSL and I've gotten fairly quick responses and resolutions usually appear in a few weeks time (I'm in the Fast Insider's ring at home).
But without things like WSL and native OpenSSH, I doubt I would even have looked; I would have stuck with MacOS forever or just gone native Ubuntu.
https://docs.brew.sh/Homebrew-on-Linux
I'm also in the process of switching back from Macos to Windows thanks to WSL.
I _have_ to wonder why we didn't just pick up Bazel, which is Google's open-source distributed build engine for large systems, which also happens to have been stable for years. Perhaps its Windows support wasn't up to snuff at the time, but it feels like that would have been easier to fix that than to build a whole new build system.
Regardless, congrats again! So cool to see this out in the open.
(I work at Google and have contributed a very tiny bit to improving Bazel on Windows, though it is not related to my actual work at Google.)
I am yet to see what Bazel does better, beyond faster Android builds, which is quite easy given Gradle and how Android team implements their plugins.
So far the key differences from maven, which I am most familiar with, are:
* The module format is much lighter, making it more practical to split a project into thousands of modules, which can then be built incrementally. This leads not just to faster build times, but finer-grained tracking of dependencies between modules
* The driving factor for us: Bazel has built-in support for specifying the sha256 hash of external dependencies. Verifying authenticity of external dependencies is not possible in maven without significant extra work. In Bazel it's built in.
* Much easier to set up bazel targets to perform once-off tasks like seeding a local database through your build system. While can you can do this with maven, you usually had to resort to plugins and the declarative style made it complex. Most maven projects I worked on had external shell or python scripts for these sorts of tasks.
For me personally the biggest downside has been that it takes over enough of the build-system responsibilities in IntelliJ that other tooling I'm used to that integrates with the IDE, such as Chronon or JRebel, doesn't work.
Bazel has a steep enough learning curve and enough additional complexity that I probably wouldn't recommend it for most projects unless you are at a certain size and complexity level where you can benefit form it.
>> Bazel has a steep enough learning curve
I'd argue for basic tasks it has far less of a learning curve than e.g. Gradle or, god forbid, something like CMake.
Bazel was first announced and made open-source in early 2015, but by that time Domino had already been in development for at least a year. I remember following the early internal discussions at Microsoft on the One Engineering System or "Apex" initiative which first provided the impetus for what became Domino; some of those discussions were quite animated :P
Also, while I don't know BuildXL and Bazel well enough to compare their goals and relative strengths, I will say that BuildXL was developed at Microsoft, which has historically had a great diversity of build toolchains and workflows across all its software products. Just on the toolchain side, Microsoft has many different build engines/make tools, unit test frameworks, tools and conventions to register an automated test to run on a certain schedule (for products too big to run every test on each and every build), build and test server farms, continuous integration tools to orchestrate those server farms, tools to email people and/or file bugs on test failures, and so on...
From my limited understanding, Google has never been this way at all, and so some of the lessons and ideas incorporated into BuildXL may seem strange to Bazel's designers.
Anyone know of anything similar but without that steep learning curve?
The only catch is that while the language and tools are simple, Nix is really a pure functional language and you have to treat your build process like code rather than config. It's easy to go down rabbit holes...
(I considered Bazel but it's very half-baked and lacks all of Blaze's proprietary toolchain support.)
https://github.com/Microsoft/BuildXL/blob/master/Documentati...
Seems what drove this was a 90 build times of 90 hours for and end to end build of Office. What I can gather this has a means of capturing all read/write operations for a build step and placing it in a cache store to determine if a change necessitates the rebuild of a component. Since it can hook in at a lower level, it isn't sensitive to time stamps for building. Actually quite interesting
Anyone else notice this?
Ant did it in XML, Premake uses Lua, CMake has its own, etc