A new CMake Scripting Language?
zenodo.org
zenodo.org
This deck does not address the drawbacks of doing this at all, and the only recognition of this problem is in "Stretch Goal: Research development of tool(s) to translate legacy CMake to new CMake language."
The current CMake language is fine. Nothing amazing, but good enough.
But every moment that I spend configuring CMake is a moment I don't spend doing something actually useful. I don't want to disparage the team's work--CMake is awesome software--but the way they see it is very different from the way typical developers see it.
No. "Legacy cmake" has been made obsolete with the release of cmake 3.0, almost a decade ago. There is absolutely no excuse to use anything other than the declarative, target-based "modern cmake" style.
I'd go as far as claiming that 95% of all use cases boil down to simple straight-forward one-liners with modern cmake, with the remaining 5% corresponding to putting together custom cmake find modules, packaging, and deploying runtime dependencies.
> (...) now there need to be two more languages (slide 13) (...)
I'd argue that if anyone feels the need to write scripts to do something in cmake, they are already doing something awfully wrong to start with.
"made obsolete" != expunged from codebases. I'm sure there are plenty using cmake 2.x conventions that haven't been updated, and need maintaining. And in order to either maintain or upgrade, one needs to know both conventions.
No, not really. You only need to know the declarative, target-based modern cmake. Most of the ad-hoc nonsense done with legacy cmake is either features that since the 2.x days were added to cmake, or nonsense that should never have been written to start with.
No you're completely missing the point. If it exists and needs to be managed then anyone in their right mind migrates their cmake project to modern cmake.
Why? Because odds are any weird thing happening in legacy cmake projects happened because either cmake didn't supported things way back then or old timers made a mess for themselves, needlessly.
Maintaining code does not require you to keep messes around. You're expected to pay legacy debt, learn from mistakes, and right whatever wrongs you made. With cmake projects, this means modern cmake. No excuse.
> and still needs to either be maintained or re-written.
It's not a "or". No one forces you to not fix mistakes. You only keep them around if you wish to, and if that's your own personal decision then the responsibility of creating that problem is on you, not on the tooling.
I'm talking from experience. I've maintained a dozen or so legacy C++ projects, some of which with auto tools, qmake, and the deprecated imperative cmake style put together over a decade ago. Porting stuff to modern cmake is trivial. There is no excuse.
If your old lib can't be build due to a deprecated build system, that's already the greatest possible disaster for a build system, as all your automatic tests etc might not run anymore and you lack a basis for releasing anything at that point. You need to setup an entire new project for recovery.
I seems common for people to try to have CMake run Python scripts, install stuff on the system, and do all sorts of custom tasks. I'm sure other build systems also allow devs to make a mess if they want to. Maybe some are less powerful, so that people cannot make such a mess even if they want to. But "the tool is too powerful, I will shoot myself in the foot because I don't want to learn the basics" does not sound like a good argument to me.
> The current CMake language is fine. Nothing amazing, but good enough.
I'd argue these two statements are in conflict.
I came to c/c++ after several other languages and ecosystems -- the c++ dependency/build ecosystem is by far the worst.
CMake does work. It's underlying capabilities are great, even. But "modern cmake" is sort of a joke syntactically. Generator expressions are indecipherable. Semicolon delimited arrays, no distinguishing between input/output variables... I personally would love for a different DSL frontend to CMake.
> I don't want to disparage the team's work--CMake is awesome software--but the way they see it is very different from the way typical developers see it.
I have to agree with this. The underlying core of cmake is actually pretty awesome. Modern cmake is an improvement in capability. The team deserves credit for that.
Yeah. CMake works, it can do everything you want somehow, but it's a total mess, and the documentation isn't great.
Just a simple wrapper API (like libcurl-easy, for example) would go a long way.
I completely disagree, I almost always find what I need in the docs.
> Just a simple wrapper API
IMO, just spending a couple of hours learning the basics would go a long way. Usually that is not done.
But its biggest issue, arguably, is the extreme proliferation of bad information. Everything on StackOverflow and similar sites is outdated and bad practice for the most part. You're right that introducing a new language which deprecates the old will make this an even bigger issue.
What am I missing?
Similarly there's things that have been ad-hoc standards forever, like "make clean" to remove all build and test artifacts, or "make test" for running tests.
Many of my coworkers only use it in this way and are unaware make looks at files/timestamps.
The irony of using “hype” to describe make/CMake/autotools while referencing Zig as an alternative example.
Building a build tool written in C or C++ requires a build tool — hence the existence of make/CMake/autotools, which can bootstrap themselves from a minimal environment.
If you ship a library, then you need to provide a way to install it somewhere, and ideally a way to find it (e.g. pkg-config, at least on Linux).
If you need to use CMake because your dependencies use CMake, you are doing something wrong. I typically build dependencies with all sorts of build systems, I just don't mind. I just need them to be installed somewhere such that CMake can find them.
C/C++ compilers don’t have a scripting language built in to construct your build system on top of?
I mostly use distools to build python modules but I have a couple projects where there is code generation, building static libs, &etc… and I find it easier to use cmake to just pop out a python module.
Cmake can also be convinced to make distro packages, like rpms and debs, without too much trouble — not sure how to do that in any other build system.
Lol CMake was a huge reason I got out of C++ development almost a decade ago. I was tired of scripting my own shitty build systems, especially ones which couldn't even manage packages. I never actually need the power CMake offers, but the stuff I do need (reproducible package management) CMake punted on. 99.9% of the time, all you need is a build tool that takes a list of dependencies and a lockfile (or similar).
I'm very happy for you but almost all the C++ projects I've worked on have needed muuuuuuch more complex build system scripting than this.
But there is a philosophical question here: I think that developers should master their basic tools to the point where they don't make a mess writing a CMakeLists.txt or a build.gradle.
It feels like many devs think that a tool is too complex if you need to learn it.
But that was too boring, so people turned to gradle and we're back at ad-hoc scripting in a bizarre Turing-complete language. Believe me, I've hated maven with a passion (and still do) but this episode taught me build systems are nothing but a zero-sum game and typical junior-grade pastime.
Update: the story with Node.js is a bit different, but package.json was soon augmented by package-lock.json and yarn at about the same time Webpack and similar became kitchen sinks requiring config-generator tools such as create-react-app. The morale is, the grass is always greener on the other side I guess. Though tbh, cmake always was absolute garbage making autotools look sane in comparison. Now I hear there's legacy/unmaintained and new cmake syntax; what were people thinking?
Just because it accepts XML doesn't mean your build system is meaningfully declarative.
An example: the main software I work on, along with a few other software I know, has to generate a SDK as part of its build so that people can build plug-ins for it. How often do you have to do this in java or JS ? almost never. But this requirement implies having a build system powerful enough to do that.
Another example: another software I work on must produce a Max/MSP package ; this implies:
- working with multiple versions of the Max SDK which changed their structure on disk
- setting different build options such as file naming depending on the target architecture: .dll on win32, .dll64 on win64
- generating a relatively complex tree structure which combines a Windows and a macOS build together because people don't want the hassle of having two different download options
- generating some stub C file that will declare a function to be exported
- generating some .json file with various build information, build date, etc
- generating a symbols.txt file which contains all the symbols that are known to be explicitly exported to minimize binary size ; the build system has to: list the symbols (one per plug-in we build), write them to a symbols.txt file, pass that file to Apple's dyld like this:
set_property(TARGET ossia-max APPEND PROPERTY
LINK_FLAGS
"-s -exported_symbols_list '${CMAKE_CURRENT_BINARY_DIR}/symbols.txt'"
)
these are the requirements we have. can you solve these requirements with just Java's XML ?
Yes, actually. They don’t come up very often, but maven in particular has the ability to do everything you’ve mentioned. Most of them built right into the standard plugin set, like setting various compiler options, adding build information, or even platform specific build shenanigans (usually when dealing with javafx).
Maven also has a rarely used escape hatch plugin (ant run) which allows you to run custom scripts if you really need to.
The canonical way to do such a thing is through the java annotation processing api [1] and using a tool like java poet [2]. Before you did that, you'd probably decide if you wanted to instead use bytecode generation with a library like bytebuddy [3]
But, assuming for some reason, you wanted to torture yourself and actually consume a java.in file and apply a regex, then you'd probably pull out the "maven-replacer-plugin" [4] and configure that for the task at hand. (or use your favorite templating language plugin. There's a million of them).
Though, to be fair, this really isn't something that comes up in regular java programming due to the nature of the ecosystem. Anything you'd want to codegen likely already has a library and anything you didn't would receive (legitimate) push back.
> And could you really say that this would be cleaner than the same thing done with cmake?
Probably not. But not really the main point. cmake becomes a problem when you want to do anything more complex than string replacement. Imagine, for example, if you did want to use a code generation library within a project. The strength of maven is a new team dev only needs maven. Further, if they need to update that library they only need to change that maven file.
What happens with cmake if you wanted to update a minimum dependency?
[1] https://www.baeldung.com/java-annotation-processing-builder
Never needed 18 programming languages to build. I actually don't think CMake is harder than Gradle or SwiftPM. Most people I see complaining about CMake actually write bad CMakeLists, so the way I see it is that they don't want to learn the basics. And that's maybe why e.g. gradle works better: for many projects, the IDE is probably able to automatically handle an imperfect and not-so-readable gradle configuration, so people don't need to learn gradle.
But the result is the same: they don't really master their build tool.
Rust tries to avoid that by hard-coding availability into the libraries themselves, but the result is less reliable. For example I cannot use `libc::mkostemp` on macOS because Rust has (incorrectly) hard-coded that it is not available on that platform.
CMake is able to detect it, but we pay a cost in build system complexity. I wish instead this detection could be done at runtime.
But your C++ project never needed 18 languages.
If you feel it does, you need to take a step back and take a serious look at the mess you're making because you're making at least 17 mistakes.
Tell me, what exactly does your build need that is not met with the following:
project(foo LANGUAGES CXX)
add_executable(foo_exec ${SOURCE_FILES} )
target_link_libraries(foo_exec ${MY_TARGETS} )
https://mesonbuild.com/Wrapdb-projects.html
Writing new wraps involves essentially porting the target project to meson though, so it's far from trivial.
It is harder to get 400 dependencies than with other ecosystems, I'll admit that. But I see this as a feature. I can't even start to imagine how people deal with security when the most basic app they write gets tens (hundreds?) of transitive dependencies. I guess they don't. Nobody cares about security.
This assumption implies that C++ developers are all one-track wonders that lock themselves in a basement and have no contact with any other programming language or tooling.
> But the tooling is just insanely cumbersome (...)
It really isn't. Node.js with npm or yarn is a far greater mess than anything that happened in C++.
I'd say that 9 out of 10 times people complain about the C++ ecosystem, they are actually talking from a place of ignorance and at best are only complaining about the mess they needlessly dug themselves into.
Case in point: who in their right mind feels the need to bolt on a different scripting language to cmake, a build system that only rarely needs a damn if statement in the project? These guys are clearly deeply invested in making their lives far hard and worse than they need to be. I mean, where's the need?
If anything, this wave of contrarian bullshit targeting C++ only shows the high volume of inexperient and clueless developers who decided to skip learning the basics and instead thought it was a good idea o dive head-first into wheels reinvented very poorly, and from there in proceed to depict their own mess as representative of everyone else's experience.
You can just link all your files together in a single command without make
Not sure why a makefile is so bad.. it’s just a file generation tool which can check out-of-date files and specify dependencies by pointing out which input files are needed for an output.
It should be read as: “Make file”
Makefiles don't work for extremely large and complex applications. That's the whole reason we have junk like CMake and Gradle.
That's funny. After thirty-odd years of C++ development, it's only in the last year or so that I've started to take CMake seriously, since a critical mass of other people seem to have (inexplicably) decided to treat it as something akin to an official C++ build system.
No, CMake is terrible. It is in fact quite a feat how god damn terrible it manages to be.
I have built cmake systems from scratch and replaced others with vanilla make. I am currently stuck dealing with a cmake project that would make all of Byzantium blush.
I agree that cmake is terrible. The more I learn to use it properly, the more I hate it.
In addition:
If you think cmake is solving any problems that vanilla make does not handle better, then either (1) you don't understand how to build and package software, or (2) haven't spent an hour or two reading the gnu make manual and "recursive make considered harmful".
I guess a bazillion
QRCODE_INC := $(shell pkg-config --cflags qrencode || true)
# or whatever the make version of this is:
$(if $(QRCODE_INC), ohgawd, ,)Presumably all of those are tested in CI on each PR?
The cmake you linked to is a classic example of the "not knowing how to build or package software" cmake anti-pattern.
As for QRCODE_INC, I'm guessing that it is just an include directory. Put qrcode.h in /use/include, (or /usr/local/include, or wherever the compiler looks by default) or set an environment variable to tell the build to look in nonstandard paths before looking in system directories.
Problem completely solved.
(Objection one: What if the installer for qrcode.h puts the header in a strange place? Solution: Fix libqrcode's package.)
(Objection two: I don't want to pollute standard paths with conflicting library versions, and can't be bothered to set environment variable overrides. Solution: Docker)
Based on your PoV, I'm guessing pkg-config is similarly misguided, since those fools allow the end user to specify the prefix, includedir, libdir, and any cflags and ldflags for ther setup. Pssh, idiots. If they'd just "sudo make install" everything, they'd be enlightened
> Presumably all of those are tested in CI on each PR?
Clearly not "2^26 possibilities," but not "linux/amd64 or gtfo" either: https://github.com/keepassxreboot/keepassxc/releases/tag/2.7...
I don't usually hate tools. I often hate what people do with them, though.
Now to use a tool correctly, you certainly need some basics. Saying that those who manage not to make a mess with a tool are probably doing it wrong because you don't know how to do it feels... weird.
I ended up being fairly proficient with the "modern" CMake language after investing a ton of time into it, but as the size of the company grew, I ended up being a bottleneck. Every other team found themselves subject to CMake's many footguns, and I'd end up having to step in to help.
It was maddening, but I 100% get that other teams just wanted to focus on getting their work done, rather than fighting with a poorly-documented and frustrating build system.
I, for one, would be very happy to see a new CMake scripting language. Ideally, one that is a lot more friendly for beginners and occasional users.
Getting your project to build properly should be part of "getting the work done". Modern software is way too much about stacking up hacks to "get the work done" instead of writing proper code.
Sure, there could be a better way than CMake. But it's not that hard to learn the basics.
Eg, you'd think downloading a file in cmake would be fairly straightforward, but it's not really.
Now I could build the download into the test itself, but that's extra stuff that's not part of the test proper.
I also never got why we need DSLs for build systems. It seems like Turing completeness is often inevitable, no matter if people don't like it. There is no reason that a build file couldn't be mostly declarative with a general-purpose programming language though.
If you're still struggling with this, a solution is to use ExternalProject to create a "superbuild" top level project that has various subprojects being built with different toolchains.
The same reason they show up anywhere: to have the correct level of abstraction, or its cousin "signal/noise ratio." Trying to debug "gcc -o build build.cc && ./build" sounds like the 8th circle of hell to me. I can show a concrete example: https://github.com/facebook/sapling/blob/0.1.20221201-095354... So, when this raises OSError or FileNotFoundError due to some botched assumption, I guess one just attaches pdb to it and now your job is to be a python expert
Or, my other favorite "what is wrong with you:" https://hg.mozilla.org/mozilla-central/file/FIREFOX_NIGHTLY_...
Or, so as to not just be hating on python, this junk: https://github.com/brave/brave-browser/blob/v1.48.33/package... and its https://github.com/brave/brave-core/blob/master/build/comman... friend
All of those are in a general purpose programming language, and they require spelunking through thousands of lines of code to surface the actual commands and dependencies that are buried therein. And, if you become a domain expert in mach, where else are you going to apply that expertise?
Well, debugging cmake files is not any better, in my experience. And I can't even attach a debugger. I can rely on printf debugging or pray that cmake with --trace or --trace-source has the relevant information I need (often it doesn't).
edit:
Well, your disgust is justified against ad-hoc imperative build systems that happen to use a general-purpose programming language. It doesn't have to be like this. I liked the idea of scons, for example, unfortunately it had other problems.
$ cd mozilla-central
$ find python/mach python/mozbuild/mozbuild -name '*.py' -exec cat '{}' ';' | wc -l
83088
$ find keepassxc -name CMakeLists.txt -o -name '*.cmake' -type f -exec cat '{}' ';' | wc -l
866
I'm curious: is there any way to estimate how much US taxpayer support went into creating CMake? Would CMake exist were it not for the DOE and/or NSF grants that Kitware got?
Getting that kind of support isn't a fault of CMake, but I think it's a meaningful property in considering and comparing different software development projects, and evaluating their long-term health.
I'm not impressed, for two reasons:
* Backwards compatibility for the CMake language is not desirable. It would be better to allow files to be translated to the new language completely. Sure, CMake could support having files in the old language and files in the new language, but the CMake language is so bad that a break from it is the best path forward.
* The CMake language is not the only drawback to CMake, unlike the claims in the presentation. CMake's model is hamstrung as well, in several ways.
Here are some ways in which the CMake model is hamstrung:
* Generating a build file for another build system. This means that CMake always gives up control of the build. It can only tell the other build system the dependency graph and then sit back and watch. It can't regulate the build itself.
* No dynamic dependencies. If you need dynamic dependencies, you need to call into CMake to generate a new and separate build file for the second build system and then call that. And because the two build files that CMake generated don't know anything about each other, it's hard to regulate the use of computing resources between them; you either have to complete the one and start the other, or you risk over-extending on your computing resources.
* No way to generate a target that doesn't call an outside process. More generally, the only way to generate a target (unless my CMake knowledge is old) is to have the second build system create a child process. This is strictly less powerful than having the same scripting language available during configure also available in targets. For example, LaTeX is only properly built with a loop that checks for a fixed point. If you can't loop in your target, you have to use an outside bash script or something of the sort to make up for it.
* No way to have targets that do not generate files and have other targets depend on those targets. If a target does not generate a file, CMake does not know how to make other targets depend on it (unless my CMake knowledge is old).
* The model of "configure in Turing complete language and then build" also means that configuring often has to happen more than once before a build. You see this if you use `ccmake`: you configure, and new options appear. You set the options and configure again. More options might appear. Using plain `cmake` hides this by using defaults, but that just means the user might end up with a build they didn't want because they didn't get a chance to set all of the options they might care about.
Anyway, rant over.
That's the critical selling point of CMake in my estimation. When I'm on Windows, I can generate a Visual Studio project and use its excellent debugger and code analysis tools. When I'm on Mac, I can generate an XCode project. Or a CLion project, or Makefiles, or...
Decoupling the build "recipe" from the build toolchain is critical for collaborative, cross-platform, cross-architecture projects, IMHO.
> No way to have targets that do not generate files and have other targets depend on those targets.
Isn't that exactly what an INTERFACE library is for? <https://cmake.org/cmake/help/latest/command/add_library.html...>
CLion’s native support for CMake is the only thing that makes the IDE story palatable, but everything about CMake’s architecture resists treating CMake as a declarative format and performing that kind of intelligent integration.
1) Two editable sources of truth, one of which is only sometimes overwritten when performing a target build, depending on whether the CMake inputs have changed.
2) Non-universal native IDE constructs can only be expressed if CMake supports them, and if you use IDE-specific CMake features.
VS is as good as it gets, and it isn’t that great. By comparison, Xcode project support is absolutely terrible.
CMake has a semi-declarative semantic representation of your project and its targets, but has to lossily translate that into a number of different, incompatible project formats, all of which are only capable of representing a different incompatible subset of CMake’s features. In several cases, CMake’s features behave differently (or not at all) depending on the generator being used.
I think GP already thoroughly elucidated most of the issues with this model; all of these issues fall out of the necessarily lossy, inconsistent conversion to incompatible output formats.
Except that you don't need to, Visual Studio has support for CMake built in (since 2017).
I've already addressed some of this in another comment ([1]), but you bring up more points, so I'll address it here.
CMake's model forces it to generate build files for other build systems. The better model does not force the build system to, but it also does not preclude doing so.
Separately from actually running a build (and that's important), my build system will be capable of generating files for VS, XCode, and other IDE's, but the files it generates will simply tell the IDE's how to integrate with my build system by telling them how to call it with the correct build profile. And yes, it will generate multiple possible build profiles, allowing VS users to select the solution the want to build from within VS.
But this means that my build system will not only integrate better with IDE's than CMake, it will still retain full control of actual builds.
This includes using debuggers and analysis tools, by the way. I wouldn't consider my work done on IDE integration until all of that is as easy as possible, and definitely as easy as CMake.
> Isn't that exactly what an INTERFACE library is for?
Maybe? It might be enough. At first glance (and I fully acknowledge that I could be wrong here), it seems like CMake expects the "target" to be a library of some sort. That's not the sort of thing I would personally want.
I would want to be able to depend on any target for any target.
But again, I could be wrong. I've never been able to understand the CMake docs.
That is a hard requirement to using Visual Studio and XCode, which I wager account for more than 90% of C/C++ projects.
Here's how my build system will do it: the programmer will list default build profiles. Then the build system will generate Visual Studio or XCode files that include all of those default build profiles, but those VS or XCode files will simply call my build system with the correct build profile.
Thus, when a build is run from VS or XCode, my build system is still in full control.
In essence, instead of generating build files for use by other build systems, my build system will generate files to tell VS and XCode how to call my build system. It will tell VS and XCode how to integrate it.
My build system will actually have a command-line option to reverse the dependency resolution. Instead of building a given target, updating its dependencies as necessary, it will build the default target(s), marking the files given on the command-line as "changed" and resolving dependents up to the default target(s).
This will mean that building the "current file" will be a simple command-line switch that can be part of the file generated for that IDE.
But yes, I'm going to put in a lot of work to make IDE's work as smoothly as possible.
Anyway, CMake is terrible - for all the reasons you have identified - so I wish you luck!
Funny you should mention Bazel and Buck; I'm coming for them too! I'm also coming for Nix; I want its advantages to be easily accessible for mere mortals.
I think I can do better alone than those teams have because they have all made assumptions that weren't great. I'm learning from their mistakes.
Also, UX matters, and I'm putting a lot of effort into that.
But the other reason is that if you do Nix right, the capabilities of Bazel and Buck just appear.
Thank you for the encouragement!
I'm curious about that... Nix has no notion of "providers" AFAIK, but these are key for optimization and composition of targets in Bazel / Buck(2)
That said, can you explain providers to me? I looked up the docs, and I'm no closer to understanding them.
I want to understand them and their uses do that my build system can have the same capabilities.
Every Bazel rule (e.g. cxx_library) has attributes and an internal implementation function in Starlark. The implementation can read the attributes and coordinates the work of actually calling programs (like Clang) and collecting the outputs. The collected outputs are exposed via a list of data-structures of type "Provider". When a target depends on other targets, it can see the providers of its direct dependencies. This allows targets to build on their dependencies. In C/C++, for example, the Providers contain the header files, the objects for linking etc.
I would definitely write (or at least read) a few Bazel rules before writing a new build system that improves on it!
You have a point there; I didn't because everything I've seen is that people like the results of Bazel, but they hate writing rules, so Bazel rules seemed like something to not emulate.
But point taken.
And thank you for the explanation. It seems providers allow for multiple outputs on a target. My build system allows that, but bundling such outputs together is a good idea that I need to add.
With CMake, the new stuff still sucks.
For me, Nim and Nimble work pretty well.
I don’t understand why people are surprised to see similar complexity creep in the “reborn” CMake ecosystem.
There has never been a one size fits all all purpose glue. It’s not surprising to me that neither is it possible to make a one size fits all “glue my software together” stack.
Plus the idea seems to be to add it on top, so if it flops, no big deal, the old stuff will still be there.
PROG=foo
SRCS=foo.c
MAN=
.include <bsd.prog.mk>
https://wiki.netbsd.org/tutorials/bsd_make/#index1h2 https://crufty.net/help/sjg/bmake.html
xmake.io
PS: I know that make has many shortcuts to compile a C/C++ codebase but in essence its targets as file system time stamps.
But to your question, the answer is kinda. There is support for a few other languages built in (off the top of my head, Ada and Fortran I believe) and you can provide a custom backend as well for your own language support but most people just use add_external_command
ada support in cmake is terrible. in the end you have to call gnatmake several times, if you want a build that always work. terrible.
There are much better alternatives:
c++ is doomed
That's been my favorite part of CMake (even before CLion used it as their defacto project format): if you can get a run of cmake to complete (which, granted ...), then you can open the project in a sane IDE and have library and build targets available