A new decade, a new tool
vector-of-bool.github.io
vector-of-bool.github.io
And the point is, he's making npm/maven for C++. Did C++ not already have a fully featured package manager until now? Why not?
There are several competing projects, but no clear winner yet. And a lot of people a companies prefer to build their stuff themselves for additional control.
> Why not?
It's complicated. In a few words, it's practically impossible to build a general tool that could suit everybody's needs.
Indeed and that's not what npm or maven aims for either, no tool fits everybody's need, not even npm! Probably there is a deeper issue than "it's hard", or if it's "just hard", why is so much harder for C++ than for the JavaScript folks?
C++, on the other hand, doesn't work like that. Source code needs to be compiled to be usable and the exact way it is supposed to be compiled varies enormously between different projects. Not to mention that there are usually lots of compiler options and project-specific macro definitions that you might want to change to get the exact binary you need. Compilation matters, it matters a lot, and it cannot be easily abstracted away without sacrifising performance and flexibility.
There are C++ package managers that exist and work - they tend to be your OS' package manager, because of the problems they will have to work around.
Because C++ has a lot more flexibility in core components than JavaScript does, which aren't easy to resolve at the package manager level.
Things projects may _require_ to build correctly:
* Multiple compilers that don't always have compatible extensions that sometimes share the same flags, and can falsely report to be each other when queried.
* Multiple simultaneous differing versions of clib to link against. They may requiring differing specific versions of specific implementations.
* Specific kernel libraries to link against. You can't easily build and link against a specific OS kernel in isolation.
* Projects may expect specific paths, and ignore or break attempts to change these. Not being able to compile a project that has a history of twenty years isn't an option.
* Arbitrary execution with intentional side-effects. It's bad, but there is a very long history of projects that you're dealing with here.
* CPATH and CXXPATH are globals. Relative path linking is possible, but difficult and a pile of hacks. You can force a node_modules style to exist, and some projects do, but most are a mix of relative and global, and the global can override your local and break things.
No, it doesn't. I'd say it was due to age and fragmentation; most of the more recent languages have, in effect, a single "vendor" who can coordinate something like that. Whereas C++ has historically one big proprietary and one open vendor who fought each other (Microsoft vs. GNU), plus a long tail of proprietary implementations (e.g. Intel); more recently there is clang.
The language itself doesn't have packages.
In practice, the role taken by a package manager in the Free world was occupied by autoconf, which does a good job of building across a huge variety of systems but is a pain to use. There was also much more of a tradition of "this is what you're getting, deal with it": rather than a program specifying precise version requirements, autoconf probes the capabilities and then adapts accordingly.
Build system in the Free world is make, with occasional jam if you're a big boost user. Microsoft of course do something completely different with MSBuild. Visual studio does have an integrated package manager, nuget, but it's orientated at C# use.
This is a widely held meme, but I think its flat wrong. The reality is that most C++ project's have a very wide spectrum of types of dependencies and build steps. The history of competing vendors certainly adds to the non-homogeneity, but it isn't the only portion. Even if all of our C and C++ dependencies were built and installed by a common package manager, we would still have dependencies on a variety of packages in Fortran, Python, Bash, Tcl, Verilator, NGSPICE, Octave, and so on. Running the compiler and linker represents only a handful of the many steps in the process.
C++'s niche lies at an intersection of performance and complexity. If the project wasn't demanding enough for C++, folks wouldn't pick it in the first place.
It's oriented for MSBuild. The package creation tooling is aimed at .Net. It's certainly possible to create high quality C++ Nuget packages (that are also huge because you have to provide multiple binaries).
The docs similarly don't seem to include a concise explanation but at least are a bit more readable: https://vector-of-bool.github.io/docs/dds/guide/packages.htm...
I don't really see it either, my typical C++ project is okay with the library management the different *make tools offer. But C++ is not my go to language these days, I'm usually rather careful on which dependencies take on in those projects and where they come from, not really sure a npm/maven dependency model is something I would want there.
Well, based on what I read, he's not actually making the equivalent of npm. Instead, he's proposing a new tool ("dds") and new syntax of config files to integrate build and package management. This distinction is key because the "package manager" for javascript implies npm the the tool & package specification but also a public repository ("https://www.npmjs.com") that is funded by the Nodejs/NPM Inc entity.
Likewise, Rust's package manager points to "crates.io" repository which I believe is funded by Mozilla.
This blog article is not talking about making a C++ repository like "npmjs.com" and "crates.io". Only a syntax specification. Arguably, the destination repository (server) is more important than the command-line tool (client) because it's easier for a funded website with diskspace+bandwidth to justify the existence of a tool rather than the other way around.
> Did C++ not already have a fully featured package manager until now? Why not?
It's a seemingly simple question but it the "answer" which hasn't happened yet is complex and has multiple parts. The 2 big parts are (1) social inertia and (2) technical complexity
(1) social inertia: C++ created ~1986 has 2+ decades of usage before the idea of "let's have a process _automatically_ reach out to the internet to download dependencies" became popular. Thus, there is no well-known repository that became a Schelling Point[1] for C++ programmers to upload their C++ libraries.
Consider popular C/C++ libraries like SQLite, Zlib, OpenSSL, etc. Why don't those authors upload their libraries to a package manager repository? Because nobody made an influential website destination for it that everybody adopted. E.g. If Google Inc funded a C++ repo with disk space and bandwidth, would corporate politics prevent Microsoft and Apple from uploading their code to it? In contrast, the Javascript quickly adopted NPM that was funded by Nodejs and it was the already default baked into Nodejs downloads. Which entity in the C++ world is analogous to NPM inc to fund a repository that Google/MS/Apple/Adobe/etc would all adopt? It needs to be an entity that cuts across all those competitors. Maybe Intel? Or maybe Epic Games (they make Unreal Engine)? Or all the big competitors agree to contribute to a non-profit entity? Nobody has stepped up to the plate like NPM Inc did.
(2) technical challenges: C++ has a compile and link steps that interpreted languages like Javascript don't have. In npm, the assets are "text" (i.e. .js files like leftpad()). In C++, many library publishers (proprietary) don't offer source code (the "text") and only binary precompiled .lib files. These libraries are binary blobs that are incompatible across cpu architectures and compilers (Intel vs ARM, Microsoft MSVC vs gcc, etc). Because C++ is low-level, there's also a different set of build configs such as 32bit vs 64bit, static link vs dynamic link, and DEBUG vs RELEASE, etc. This means C++ needs to implement a much more complicated "dependency" syntax way beyond just "versions" and nobody has created that specification that everybody has agreed to use. Since Javascript doesn't have separate compile & link phases, something like "npm" is much easier to create. If Rust also has "compile & link" step, how did they successfully get people to adopt the "cargo" tool and the "crates.io" repository? Probably because Rust's ecosystem of programmers has more emphasis on shared "open source code" than the C++ programmers of the 1980s and 1990s. Also, we emphasize again that Rust's patron (Mozilla) also funded "crates.io" and Rust programmers just adopted it so the "cargo" tool has an http destination to point to download artifacts. There's no equivalent influential entity in C++ yet. In terms of package spec complexity, I haven't studied Rust's TOML configuration spec to know if it's flexible enough to handle all of the different ways that C++ practitioners build files. A lot of C++ build flexibility is put into "./configure.sh" scripts (e.g. ffmpeg, Qt, etc) so I don't know if toml spec is a superset of all that.
Rust has the compile and link step. People adopted Cargo because it is nice to use.
I don't think it's that simple and I want people to really dissect what the "nice to use" means. I contend that a fully funded backend ("crates.io" on Heroku+S3) is essential to make cargo a viable tool. A programming language ecosystem needs a Schelling Point (or a disproportionately influential C++ vendor dictating industry-wide tool usage) and the C++ ecosystem doesn't have that.
We can't replay history with an alternate events since cargo and crates.io came together but let's try a thought experiment and think of cargo in isolation:
Let's say we believe cargo's nice usage syntax (and not crates.io) is the what really drives adoption. Ok, let's hypothetically clone the github/cargo project and modify it to C++ (e.g. process ".cxx", "cpp" files instead of ".rs" files. I don't think the existence of a "nice tool" on the client side will matter for C++ programmers. There are still no sponsors paying for a C++ repository with disk+bandwidth that everyone will upload their code to. Swift.org has Apple as its sponsor. Who's the logical entity to fund an industry-wide C++ repo?
This already does exist: Conan, for example. And cppget for build2.
But is there consensus that conan.io or cppget have become industry-wide Schelling Points?
For example, I searched both repos and neither have popular dependencies like Qt nor ffmpeg. Therefore, "conan install ffmpeg/4.2.2" isn't going to work. Thus, large non-trivial projects would only be able to use a package manger like Conan/cppget for some dependencies but not all.
In Javascript land, npm is so pervasive that it can be "one-stop shopping" for fetching all js dependencies. Likewise, not only is someone paying the light bills for "crates.io", it also has instant canonical status within the Rust community. Conan doesn't have comparable mindshare in the C++ ecosystem. This is a common interpretation of the question "Why doesn't C++ have a package manager?"
Our dependency stack contains about 80 3rd party libraries. We have a custom build script to download and compile these libraries.
58 of those libraries can be built using common steps that assume the use of autotools. The remainder are riddled with special case instructions.
I wish I could say that I was more optimistic about efforts like those described in the linked-to content, but I frankly am utterly pessimistic that "dds" or any similar system will ever handle the build requirements of a project like Ardour.
Note also that a large chunk of those libraries are not even written in C++, so their authors and maintainers would not consider them part of any C++ ecosystem should one ever emerge.
https://github.com/vector-of-bool/libman
The author defines it this way:
> libman is a new level of indirection between package management and build systems. If a package manager can emit libman files, then any build system that understands them can work with that package manager. If a build system can spit out a libman-export, then a package manager that understands the format can collect the build output and distribute it.
Do you see some values in this? Would that be something helpful for your project? As far as I understand, "dds" is just one emitter and consumer of a libman manifest, but the author seems to that it could also be used in combination with other tools (such as cmake as the build tool, conan as the package manager).
If a library (either as source, or as a compiled package) provides a (correct) pkg-config file, then using that library from within a project is pretty easy, assuming the build system knows how to use pkg-config files (we use waf, and it does).
But if the library doesn't provide a pkg-config file, or provides a broken one (yep, that happens), then we either have to create one or use a different method to tell the build system where to find and how to use the library.
We don't really want a package manager per se, because we're cross-platform, and package managers for compiled libraries are almost certainly going to be platform specific.
So for our release builds, and for some of our developers, we need to start from the source code, build the library using flags we provide (because we know what we want and what we're doing), install the library (somewhere) and only then set about using it from within our project. That process has to work more or less identically for our linux, macOS and mingw/windows builds (we x-compile for windows).
Consequently, even if there's a pkg-config file and even if it's contents are correct, none of that addresses (1) the version of the library (2) the compile time flags for the library (3) actually downloading the library (4) actually compiling and linking it. It does help with "installation" because the pkg-config file goes to a known location even if the installation is somewhere non-standard.
So in summary I would say that even if libman became wildly successful, and was used by the dozens of C libraries that we use in addition to the C++ ones, it would really only function as a replacement for pkg-config and would not address the harder, bigger problems we have with 3rd party library dependencies.
people complain about all the deps but deps in C/C++ and even to some degree python are hidden since they are system deps. No one seems to blink an eye when they apt-get something and it requires 60 deps and of course the second apt-get only shows the missing deps so there is this false idea that the dependencies are fewer.
for the most part npm just works. As someone with 15-35yrs C/C++ experience and a bunch of Perl and Python it's refreshing not to have to trash my system with global deps or learn non default modes of operation, that the default is all tools and deps local.
I also get why static typing people hate JS but one thing they got right IMO is there are no namespaces. You import libraries into local variables which seems like a huge win over namespaces.
Note: I also get that c/c++ is generally harder. Different compilers, different flags, different platforms, etc. I’m not saying I have a solution. getting MS (VC++), Apple(Xcode/clang), and GNU(GCC) on the same page might be a start.
There's no big semantical difference between
var Hello = require("someorg/hey");
using Hello = SomeOrg.Hey;
In short, the node_modules package resolution algorithm is how JS does namespaces. It's the same thing, just with different syntax (and, IMO, a bit more weirdness). The fundamental thing isn't "no namespaces" but "namespaces are package-relative". I wonder whether they got that right through sheer genius, or by accident.
(I recognize that I'm being super pedantic here btw, hope that's OK)
Note that I agree with the rest of your comment, NPM is splendid and it's impressive how much they got right at the first go, especially given how NodeJS was on a hype train at the time (many things on hype trains only get tooling like this right after the first so many tries).
I recall hearing Ryan Dahl's regrets about design decisions in Node.js, a few of which had to do with the module/package resolution algorithm.
10 Things I Regret about Node.js - https://www.youtube.com/watch?v=M3BM9TB-8yA
There are some good aspects of it though. I like how modules/packages are just imported objects (the path resolution is a bit weird and seemingly inefficient, i.e., checking every parent folder's node_modules folder). It goes together well, that modules are also self-contained function scopes by default (I suppose a kind of anonymous namespace), that exports a plain object with assigned properties.
---
Wow, I just learned about C++20 Modules, looks like a game-changing feature.
Update : Verified the same issue exists in c#.
That is not unique to npm. Java/Maven, Ruby/Bundler, Go/Vendor also work like this and I believe that several other languages might work like this.
Comparing npm with C/C++ build/package tools is setting a very low bar
It really took quite a while for programming communities to accept that that local packages are a good thing, and NPM was early to the party.
And NPM does this recursively, which I think it what greggman is primarily getting at. I don't think many other package managers do this. I mean, the fact that a package's dependencies are local to it too is fundamental. You simply can't get dependency version conflicts (except if package authors do terrible global state hacks, and fortunately this is pretty unheard of). Instead, you just pull in 5 different versions of some logging library or some http client wrapper, and you don't even notice. That's not without its own downsides, but I prefer it to the steaming pile of `override` directives in our Elixir codebase has day.
Unless I am missing something, "go mod vendor", and "mvn package" will work in the same manner. They will download all deps of the app itself PLUS all transitive dependencies. No system libraries needed (or touched)
Last I checked, it'll indeed pull in all transitive dependencies, but if multiple dependencies need the same subdependency of a different version, you need to work out conflicts. JS simply doesn't have this problem, it's splendid.
Multiple unrelated projects can use the same or different version of a library without any issues
Here is the archived body of the reddit post if anybody was curious: http://removeddit.com/r/cpp/comments/egrzd0/writing_c_is_won...
The other is a 'drop dead simple' build tool. His build tool has these compile flag restrictions.
But you could still build with CMake and add all kinds of extra requirements. However, for a library that you want to share, making it simple enough to work with the simple build tool is probably a good idea.
you can have general flags (-O3/-g3) for all compilation units(CU), but there should be the capacity to add any per CU
And more good luck if you don’t control the compiler—whatever you decide may end up wrong tomorrow because the compiler developers change their minds, and of course you take all the blame because everything works except when the user uses your tool.
It took me a few minutes following the tutorial to get my project to build on Windows, macOS, and Linux.
Do I miss the challenges and rewards of working at the lowest level of the stack? A little bit. But it's more than made up for by the fact that I can produce code that's workable, readable, and extensible, and can mature and grow more powerful in months rather than years.
Of course, I now stand on the shoulders of the people who build tools like numpy and tensorflow. I have great respect for those who continue to wrestle with these demons, but I just had to tap out.
Anyway, I suspect that what happens is, as worse as the C++ compiling/linking situation is, it's not bad enough to warrant learning a new build system.
After reading this rather lengthy blog post, I am excited to try this out.
Currently, dependency management for C++ projects is a PITA.
- the author mentions his work on "libman", a common format to describe C and C++ libraries. The idea is to have a manifest format that can be emitted by a package manager, then consumed by any build tools that supports it. The main example is "Conan (a C++ package manager) generates the libman file, CMake uses it for its build process". See the project at: https://github.com/vector-of-bool/libman
- then the author presents its own build and dependency management tool, named "Drop Dead Simple build and library management tool" (aka dds). It's one tool that can both consume and generate libman manifests (at least, that's what I understood), and should be simple to use in the majority of cases. It's still very alpha, you can find tutorials and documentation at: https://vector-of-bool.github.io/docs/dds/index.html
I spent the weekend playing a bit with both projects, it's really refreshing IMHO compared to CMake & co., and quite fun. It took me just a few minutes following the tutorials to get some side project build on macOS, Windows, and Linux.
Keep in mind that all of this is still very early days, and experimental.
I guess C++ is package / build hell? I’m not a C++ developer so I’m not too familiar with the dependency story there. Anyone have a short take on what’s the problem and whether DDS is a promising answer?