Simplifying build in C++ | Hacker News Reader

Simplifying build in C++

mropert.github.io

71 pointsingve41 comments

It does not satisfy all of the items on the author's wish list, but I have used and enjoyed Bazel[1] as perhaps the nearest build system in existence that provides a notion of package management to C++. It does not hold prebuilt binaries to satisfy dependencies. But it does cache builds.

Nor does Bazel use C++ syntax to describe the build, instead using a variant of Python called Skylark, but this feels (to me) much less onerous to pick up than the custom scripting language CMake uses.

Bazel also provides a notion of visibility of dependencies that allow you in effect to control which build targets are exposed to other targets as allowed dependencies. And allows the build tool to specify a toolchain (compiler, linker, etc.) and machine architecture. This information is used when building all the dependencies, which in effect satisfies the author's request for compatibility between packages. You might even argue that a (caching) build from source may be better than attempting to serve prebuilt binaries.

[1] https://bazel.build

+1 for bazel. It also scales to large builds and handles java as well. We've seen big decreases in build times by using it as it does caching, parallel execution and good dependency analysis.

Getting up to speed can take a little doing though as the original docs weren't great and there are some counter intuitive things about it. I think the docs are improving but some things a little odd like everything gets dumped into ~/.cache/bazel without any command line option to change [1]. Feels like the first time I used git where you suspect that the authors designed it to be difficult to keep the lazy away.

https://docs.bazel.build/versions/master/output_directories....

For C++ theres also 'GN' from Chrome, that is inspired in Bazel, and generate ninja build files.

You can also specify a toolchain, have custom actions for asset or code generation, etc..

GN itself is in C++, unlike the older build system for chrome 'GYP', and it produces a binary tool, that can process meta build file with a syntax very similar to Bazel.

https://chromium.googlesource.com/chromium/src/tools/gn/

https://chromium.googlesource.com/chromium/src/+/lkgr/tools/...

I encountered GN for the first time while looking at Google's nascent Fuchsia OS. It looked interesting but I have dug further.

I would be wary of depending on gn. Google is really only interested in supporting it for their open source projects.

What software do you build with bazel? How do you distribute it?

Would you use it for an open source project? It feels onerous to ask users and contributors to install a moderately esoteric build tool (which in turn requires Java) to build a C++ project.

I currently use it for java and c++ builds inside a largish tech company. Build systems are definitely not one size fits all though so I would encourage you to pick one that meets needs of your project. If you want to build for different platforms like unix, windows, iOS, android and have a big complicated build then I'd encourage you to check out bazel. If your project needs are pretty simple I'd say just start off with a simple makefile.

The premise of matching modules and package one-to-one is I think incorrect. The way I see it, a package is just a library with a description of its dependencies, and possibly an already compiled binary. It could contains multiple modules. Plus, you probably need to link with a specific package version, and the module proposal doesn't have any way to specify a module version. So you can't just extract package dependencies from module usage.

> One last item remains (and possibly the most controversial): not using a language other than C++ to describe how to a package a C++ project

Totally agree on this point. Why learn all the quirks of the CMake language just to express the structure of my program?

Raw C++ is unsuitable as a build language, so you'll have to layer a DSL on top of it. Either way you're learning a new language.

As to "why learn all the quirks of CMake," the selling points of CMake are its widespread availability, maturity, and coverage. For example, just on the packaging side, it knows how to create tarballs, .debs, .rpms, Windows .msi files, multiple flavors of Mac installers...

Less mature build tools require you to implement this stuff manually. CMake mostly handles it for you (and has the scars to prove it).

I do like cmake, however I think that actually C++03 is the one unsuitable, C++17 is quite capable.

All those actions can be easily modeled in a libraries coupled with a DSL.

C++ as it exists now is unsuitable, but I can imagine an alternate history where C++ actually allowed arbitrary compile time execution (better than macros and constexpr). Then someone would have made a library for build system logic, and all the user would have to do is g++ project.cpp.

Herb Sutter has a proposal for metaclasses, which involve running some compile-time C++-like code. I don't think it would stretch to this, but it's heading in your direction.

I never understood why people insist of having a single language for all parts of development. Most developers already (should) speak a few languages, so one extra is fine, I guess. I strongly believe in DSLs.

It's not that they "insist", it's just that when you look at a modern general-purpose programming language you often ask yourself, why couldn't I use it for such and such particular purpose?

This is what makes me love Rust/Cargo so much. You can create pre-build orchestration in your build.rs in the same language as you build your project.

Initially I thought they've lost their mind to replace domain-specific scripting languages with Rust, but I've come around to love it.

It has enabled Cargo use its own package ecosystem at build time as well, so build scripts can easily reuse other crates instead of reinventing the wheel for each project.

It also saved Cargo from needing to be a build system that works with everything and supports everyone (the midlayer mistake[1]) and instead 3rd party build-time crates and *-sys library wrappers provide the flexibility.

[1] https://lwn.net/Articles/336262/

build.rs is a fallback though, the main "language […] to describe how to a package a [Rust] project" is Toml and the Cargo.toml file (http://doc.crates.io/manifest.html), build.rs is the escape hatch if the main manifest file does not suffice.

Python uses a Python script as the main entry point to the package, and it's commonly problematic as doing all kinds of weird shit so your package builds (on your machines, until you change your configuration) is a smaller step than looking up what the system already provides, and thus the default. It also makes static package in(tro)spection much more difficult.

> build.rs is a fallback though, the main "language […] to describe how to a package a [Rust] project" is Toml and the Cargo.toml file (http://doc.crates.io/manifest.html), build.rs is the escape hatch if the main manifest file does not suffice.

I get the feeling that they're making the exact same mistake that Cabal/cabal-install did in the Haskell world[1] -- they just don't have complex-enough builds and/or a big enough ecosystem yet to know it.

(In Cabal/cabal-install there's a "Custom" build type where "anything goes". Unfortunately that means that it's basically impossible for any higher-level tooling to have any idea of what's going on in the build... and hence it thus powerless to introspect anything useful. It's a huge pain.)

[1] Not that I blame them, it's just such a tempting thing to do rather than completely redesigning the build system to accomodate everything and provide enough introspection/metadata. So it goes, I guess...

I wouldn't say it's "the exact same mistake", but rather, the usual thing that happens with technology: you ship a first version, feel out where the drawbacks are, and iterate.

There has been some discussion about making things more declarative, if this is ultimately the right approach, etc. But you have to do something before you can know for sure what the problem space entails.

In our case, one of the largest uses of build.rs is to link to things written in C and C++; there's no unified tooling there, so if you wanted to make some sort of DSL to handle this, you'd have to also solve that problem, which is a huge undertaking.

Right, "mistake" may have been a bit too harsh, but I regularly work with systems written in multiple languages and the amount of necessary hackery (aka "duct tape") is absolutely crazy. The abysmal quality of all the mainstream build systems is just one of my pet peeves. (Incidentally, mainstream also includes "make". It's horrific, though it did get some basic things right.)

I am pretty convinced that "we" (as an industry) already actually know 95% of what the 'problem space' of a build system is, but everybody just wants to greenfield and try to do their own. Why? Because it's more fun! There's no actual real rationale for reinventing everything every few years. I guess it comes about from a lack of attention to history and a lack of attention to the state of the art, e.g. Shake in Haskell-land.

Sorry to be so negative. I'm sure that Cargo at least works really well according to the 80-20 rule :).

Considering ~60% of the use cases for build.rs is wrapping C/C++ builds I'd love to know exactly how you expose metadata for that ;).

That's a fair point, but even mandating "choose one of Ninja/CMake/etc." would be helpful in terms of at least getting a little bit of metadata out. This is obviously non-trivial, and I'm not sure how much metadata is even retrievable from my examples, but it's better than nothing. (Ninja and CMake operate at sort of different levels, but that's OK for this purpose, I think. At least you can ask things like "if X changed, what else would change?" without actually having to run a compiler.)

(AFAIUI Cabal:Custom had exactly the same reasoning, btw. "It's too difficult! Let's just give up and allow anything.")

It might be a practical reality that this is just unavoidable if you want to achieve success, but at a meta-level I think deserves thinking about whether builds should actually be allowed the full Turing Complete treatment. At least, theoretically: As long as we continue 'allowing' Custom for practical reasons and without serious thought, there can be no progress.

(Btw, I'm quite mad which it comes to this type of thing. I spend way too much time thinking about build systems, etc.)

EDIT: This problem manifests EVERYWHERE: Docker builds randomly guess wrong about whether inputs have changed, cabal-install gets it wrong, SBT/Zinc doesn't quite track dependencies correctly, etc. etc. and it's infuriating.

I like Nix. Only problem is it doesn't seem to work on damn Windows. Nix has, in my setup, made the build system topic almost irrelevant, since it allows for simpler build scripts to play well with other dependencies. Plus, it works with stuff in any language and polyglot projects are a bliss. I really hope it catches traction.

I also like Nix. Can you share your particular workflow?

Also, how does your workflow handle ‘snapshot’ / temporary dependencies?

I still haven't learned how to use a build system yet. For me it's just way easier to put the individual build commands in a .sh file and run it.

You should at least use a makefile. It's pretty simple, and mostly the same as shell, but it will automatically detect changes, and only recompile the files that need to be recompiled.

Honestly, I think very few engineers fully learn a build system. The norm seems to be learning just enough to hobble along. That said, you should really try out GNU Make, CMake, or similar.

I have...

If you're using C or C++ you can actually get suprisingly far using this approach + ccache. Most C preprocessors are ridiculously fast at this point.

C++ really needs an ABI standard. It would make builds and package management much easier.

I doubt the OS and compiler vendors will ever agree on one.

Right now the "best" solutions are COM (aka UWP) on Windows, and the C++ Itanium ABI description.

Even C does not have one, just that most mix language ABI with OS ABI, which happen to be the same on OS written in C.

If you pick something else, not written in C, for example IBM i, then the ABI is actually TIMI, the OS/400 bytecode format.

MBIs, the binary format for modules, might be the answer but until then there are many issues to tackle and agree upon.

It is impossible to have a standard ABI for a language that is supposed to be compiled down to machine code and run as native executable. The ABI is tied to the OS and specific machine. At best you can have standardized machine specific ABIs.

Which is exactly what we have.

Machine specific ABI would already be great. Whenever I need a new library it takes forever to figure out how to integrate it correctly and add to the build. From observation this also trips over most newbies who don't know all the subtleties.

Decoupling build from dependency management I think is the great feature of conan.

https://www.conan.io/

Besides CMake, Bazel, and Conan, what build system or related project(s) are worth looking at? Which one(s) would you recommend? (See other threads as well)

As i've said in another comment, theres also GN, the build system used for Chrome.

It's bazel-like, in C++, so you can have a binary tool, and it outputs ninja build files.

https://chromium.googlesource.com/chromium/src/tools/gn/

https://chromium.googlesource.com/chromium/src/+/lkgr/tools/...

I'm not saying whether it's good or bad but for completeness one should probably look at MSBuild.

There's a pretty interesting compiler wrapper called "fastbuild" that really lives up to its name.

C/C++ build environments are fascinating to me because there are so many. They solve this swarm of overlapping problems in many different ways.

https://buckbuild.com

Seems like the horse is out of the barn, across the hay field, down the street, and over to the next town on this one.

No love for autotools?

Reply on news.ycombinator.com