Conan – C/C++ package manager
conan.io
conan.io
The primary means to distribute open source software is via source code. Automagical technologies to assemble all the source code for a dependency tree, and/or automagical technologies to fetch precompiled dependency trees with curated options, versions, component granularities, etc are extremely non-trivial problems. To date the only examples of this done with a measure of effectiveness are actually entire operating system distributions backed by software like apt, yum, pkg-src, etc.
Primarily because of the total failure in this regard of windows and osx, individual languages themselves starting advancing ecosystem centric dependency management with files retrievable from servers hosted on line. At best, this style strategy solves some of the problems some of the time, and at worst breaks the ability to make things even work the old way - ie manual download, assembly, and compilation of source code dependency trees.
In the context of largely portable, interpreted code, this sort of thing is at plausible. In the context of native code with multi-dimensional matrices of OS/Compiler/libc/versions/flags/phases of the moon, the notion of a repository of precompiled binaries to even enough cover "common cases" to be thought of as a language level feature.
In the case of source dependency management, exposing the information about where and how things are installed and reachable for use in the build environment of the foreground project is hard enough, and trying to be able to hold cross platform build recipes of some sort is literally the creation of yet another operating system. Take homebrew for example.
Perhaps some sort of unification wrapper over apt and pkg-config is a thought, but honestly can't even guess what problems can be addressed with something like this.
In all seriousness the only cross platform solution maintaining build environments is to just host all builds on linux, use mingw64 and osxcross along with apt or yum + regular pkg-config, git, etc.
This looks like someone who thinks things like pip and gem make sense trying to overlay that paradigm to a broader scope. Or maybe I'm just too old to "get it". Please clue me in?
Rust achieves it. Cargo will download dependencies built for the client's desired platform.
I may be wrong but I think D does it as well.
You are right that C/C++ has a wider range of options available to it, but that is for historical reasons.
The principal benefit of a package manager like this is that it forces the packages contained therein to follow a uniform convention of platforms, build types, and options. C/C++ has been missing this for decades.
> Cargo will download dependencies built for the client's
> desired platform.
I've been toying with a similar idea that would focus only on static libraries and would handle binaries as an optimization. Just being able to specify "fetch zlib, protocol buffers and libjpeg" and build them for me would simplify many of my C projects.
The only coherent way I could think of to manage architectural differences was to force a single build tool that understood the important compiler flags and version incompatibilities.
Building from source solves a lot of problems but somehow even today it still takes forever to compile certain projects (looking at you, grpc).
I can't speak for other languages, but for C++, it's mainly the huge build time just to get started using the project. This is obviously an issue for any C++ project, but with GRPC, having to build all of OpenSSL is particularly annoying. (Especially if you're not using secured channels!).
I also couldn't figure out how to build just the C++ static libraries easily. Instead I end up building everything and then deleting the shared libraries. That's easier to do than trying to figure out how to convince the linker that I prefer to link statically.
Once that's out of the way it's pretty reasonable. It took a while to figure out that `--cpp_out` doesn't actually generate stubs, though I'm sure it's documented.
So it's not really worse than any other package; it's just _big_, and that makes it a little more annoying. Having any kind of reasonable package manager for C would simplify things dramatically. Not having to express my dependency on GRPC as a git submodule+some clunky Makefile rules would be nice.
I tried using gRPC with Go before having to abandon the project for reasons outside my control. Looking forward to using it once again and this time take it to production. The only trouble I had was with the documentation which was scarce.
Anyway, here are some comments on how to improve things with gRPC:
1. A dedicated mini-site for each supported language. This http://www.grpc.io/docs/guides/auth.html is really confusing to read due to the cocktail of languages. All examples need to be in all languages and the user should be able to choose which language to view documentation for.
2. http://www.grpc.io/docs/guides/error.html needs to be populated. Right now I use a custom error code enum w/ 0 as SUCCESS so that my downstream consumers (app developers) get a descriptive identifier. I don't even know if there is anything else I should be aware of that gRPC offers to work with errors.
3. A how-to on setting (on a client) and accessing (on a server) custom HTTP headers in gRPC. This is needed for reading the Authorization header during authentication for example. Another example is setting the From header (email of user) and using this to enable persistent sessions. I appreciate the docs covering these two specific examples as they are rather common.
4. A simple guide on how to manually make an application speak the gRPC protocol - both as a client and as a server. Helpful for those who want to integrate other languages w/ gRPC enabled languages. Endgame is they can speak the wire protocol and understand the conventions just enough to quickly write a working client/server for their app. Great if this example uses JavaScript (most people have to learn it anyway due to the browser).
5. RPCs with multiple messages as arguments in input and/or output. All examples have SomeServiceRequest and SomeServiceResponse messages for SomeService RPC giving the impression that all RPCs are single argument in the .proto file. This isn't stated explicitly anywhere creating an ambiguity.
6. Move to Discourse from Google Groups. Much easier to use and would let us organize the discussions better. Also use this to power comments on the gRPC website for each page. I can see this step alone helping the community aspect of gRPC immensely.
7. Create a visual gRPC explorer. Ideally this should be a simple HTML/CSS/JavaScript static page running on browsers that support HTTP/2 and making it would be a good way to start supporting gRPC on the browser (refer Point 5 above).
The problem is that OS package managers install things globally. This is a nightmare when developing, because
* it makes much more difficult to handle multiple versions
* as a consequence, it makes difficult to keep libraries updated, because package authors try to be conservative in what libraries they include in an operating system targeted at non-developers (in particular, latest versions of libraries are always lagging behind upstream)
* it makes it hard to be sure that what you build on one computer will be reproducible on another one
* even in the case where you know which dependencies you need (and you don't forget something just because you had already installed it), there is no simple way to document how to install them in such a way that the next user will be able to install everything with a single command
* it requires root access when it is clearly not needed
npm seems to handle this just fine for both JS and native (c++) node modules via gyp, and it's officially supported on at least three platforms (Windows, Linux, OS X) using 3 different compilers AFAIK (microsoft c/c++, gcc and clang respectively). Granted, node enforces specific conventions for packaging and lookup (package.json and node_modules), but there's no reason similar conventions can't be created for C/C++ modules, even if they're more complicated due to language specific complexity (dynamic vs static linking). Mantaining packages up to date for a lot of open source projects could be done by a sufficiently dedicated community.
Could you describe this in a bit more detail? Does gyp (in the context of npm) build the binaries at install time, or is it a repository of a bunch of pre-built artifacts?
In my experience of installing npm packages, it doesn't seem like it's building native extensions at install time, but it's hard to say for sure.
$ npm install ws
> bufferutil@1.2.1 install /Users/coreh/Projects/test/node_modules/bufferutil
> node-gyp rebuild
CXX(target) Release/obj.target/bufferutil/src/bufferutil.o
SOLINK_MODULE(target) Release/bufferutil.node
> utf-8-validate@1.2.1 install /Users/coreh/Projects/test/node_modules/utf-8-validate
> node-gyp rebuild
CXX(target) Release/obj.target/validation/src/validation.o
SOLINK_MODULE(target) Release/validation.node
/Users/coreh/Projects/test
└─┬ ws@0.8.1
├─┬ bufferutil@1.2.1
│ ├── bindings@1.2.1
│ └── nan@2.1.0
├── options@0.0.6
├── ultron@1.0.2
└── utf-8-validate@1.2.1"Oh, that's cute, you want me to do that instead of work on my project?"
There's a huge first-mover disadvantage to this that has to be solved.
If I write a C library and someone wants to use it in their library or application, they have three choices:
1. Add it as a build dependency of their project. This is a huge pain.
- First of all because most build systems are extremely fiddly and poorly designed, so adding the dependency is usually harder than it should be. For example, because CMake, which everyone seems to be using these days, can't produce sensible Visual Studio projects, one project I've worked on maintains separate CMake and Visual Studio build systems - which means that making any moderately complicated changes to the build requires me to either manually edit the VS XML files, send the result to the trybots, and pray it works, or boot a Windows VM. That's just an example; I don't know how representative in the sense that other projects also use that dual build hack, but from struggling with CMake, autoconf, and other build systems, I'd say it's representative in the sense that build systems will always find some stupid way to break and waste your time. Awful. Any alternative is worth consideration.
But let's say we found an alternative that's not a package manager, and ended up with a nice clean system where adding a dependency on a given pkg-config file is just a matter of a few lines and will just work, including for users on weird systems. Yeah.
- Second because the user now has to manually download all the dependencies, and manually upgrade them when they get out of date. Which means you don't want to release too many upgrades, even if you're trying to do rapid development. It's nice that this is possible, I guess, but it's a real pain.
- That is, unless it's in their distribution or system package manager of choice. But going to 10 different Linux distributions, Homebrew, and maybe MacPorts for good measure (and leaving Windows users to rot), and asking each of them nicely to add your library, is not exactly a pleasant experience. Maybe they will add it themselves if an application depends on it, once your library is actually popular rather than a proof of concept. (A bit of a chicken and egg situation.) And then maybe stable Linux users will actually be able to download it in a year or so, but will always be out of date. Of course, if the project using your library wants to be on its bleeding edge, their developers can't do this themselves - they have to go the manual route.
This also doesn't really work for micro-libraries, which I'll elaborate on in a bit.
So it's not surprising that two other approaches are common:
2. "Vendor" your codebase into their codebase, either with something like git-submodule or just by copy and paste. Typically this requires them to figure out how to build your code with their build system. Works okay, but is basically just a poor man's package manager - it has none of the advantages of the previous method, such as systemwide package tracking and upgrades (either by the system package manager or the user manually remembering to do it!), reduction of fragmentation, etc.; and it is more work to maintain.
3. Don't bother with your library at all; just reimplement whatever they want themselves (in a poorer fashion).
It's hard to quantify how much this happens, but in my experience, C++ projects do seem to have a bias toward building up little piles of basic utilities rather than reaching for a library. At the other extreme would be node.js and NPM - this post, as well as a Debian thread from a few months ago it belongs to, containing much handwringing about how to fit it into their system, is a good read:
https://lists.debian.org/debian-devel/2015/08/msg00603.html
(Spoiler: It's the build dependency graph for building jQuery, which contains some 491 entries, although it lists packages multiple times when they are referenced by multiple dependencies.)
Some NPM packages are literally 5 lines of code. Arguably still better than copy and paste... maybe...
I don't think C/C++ need to go there; most languages with package managers don't. But it's a good example of how the presence of packaging tools, as internalized into a programming language's culture, significantly changes the way people write code in those languages. And I think C/C++ could be a lot more pleasant if it were easier to reach for utilities outside of the standard library. (Especially pure C, which lacks an equivalent of the STL in its standard library - while the language may prevent writing data structures as neatly as in C++, you can still express some generalized high level structures pretty well and efficiently; and I think the standard library is partly to blame for the fact that people usually don't, instead building half-assed special-purpose data structure implementations every time one is required.)
...
I know, the idea of having a different package manager for every language sounds awful. It is awful, both for the user who needs to remember how each one works, and for the developer who's screwed if they want to depend on something written in a different language. I'd be quite interested in the idea of building something that feels like a language package manager, but is actually just Nix or something with some useful defaults...
But I'm starting to be convinced that having nothing at all is worse.
You allude to the complexity of C builds. While real, a huge fraction of this is not caused by any inherent difference between C and interpreted languages, but simply the diversity and general mess of C build systems. (Of course, some low level programs really do require complex build processes, and those programs are more likely to be written in C than Ruby, but those are the minority.)
And this is something a package manager can significantly, simply by providing less flexibility and therefore more standardization. Ideally, if you can build one package from the package manager, you can build 'em all.
Somewhat belated disclaimer: I haven't given conan.io more than a quick look, so I don't know how well it actually solves any of the preceding, heh. But I've wanted to see a good C package manager for a while, and I fully intend to check this one out when I have the chance.
I have used sbt, nimble, leiningen, maven, npm, cargo, conda, and while each of those has its own issues, I would take any of them when working with C.
I have tried biicode, that was showing some promise, but it seems it is not gaining traction...
It's not clear from their docs, but if it requires a seperate repo like biicode then I think it is also doomed. A package manager can't be intrusive, ideally including a single file in the root of a project (see bower, npm, rubygems).
Their example project uses a couple special made repos, which is concerning. I'd rather see the main project repos, with configuration added.
If only there was something like a universal package manager, that provides a generic but powerful package-management platform. Something where package specifications look just like expressions in a purely-functional programming language, and the package manager implicitly understands the complete dependency graph, including all inputs to all build actions.
Something like Nix.
People have been bemoaning language-specific package managers in favor of (Nix/apt-get/yum/Homebrew/MacPorts) for years now, and again and again developers keep flocking to them, despite the supposedly-superior alternatives being readily available. Rather than continuing to complain about this state of affairs, perhaps we should try to figure out why language package managers are so successful.
The reason people keep flocking to the favorite language-specific package manager is that it is seemingly the easiest way to solve their immediate problem - get the dependencies installed so they can get on with coding. And the reason not many people go with Nix is first they don't know about it, second it initially seems scary, third the particular package they need may not yet exist.
1. only packaging once, opposed to making a .deb, a .rpm and so on.
2. as soon as its in the language repo, it's available everywhere. Even if I make an apt-get package, I'd have to wait for Ubuntu/Debian/others to include it in their repos. Or I'd have to host a ppa, requiring users to search for it and add it to their package manager. RPM fusion and AUR simplify that, but it still seems hard to have your own packages on .deb systems.
http://blog.rust-lang.org/2014/11/20/Cargo.html
Standardizing on Cargo as a build system and package manager was a massively important technical decision that was made by the Rust devs. It will have positive and negative consequences for the language, probably for as long as it is relevant.
Rust will work really well with other rust code thanks to Cargo, but what if you want to make a complex project that mixes Rust with other languages at the package level? How about inside of one library? Rust isn't any worse than other languages on this front, but it could have been way better.
OK, I like where you're going...
Something where package specifications look just like expressions in a purely-functional programming language, and the package manager implicitly understands the complete dependency graph, including all inputs to all build actions.
No, you lost me.
Look, Nix and Guix are really great ideas, I don't debate that, but they are not what people want.
People who design and build tools want them to be interesting, but people who use tools want them to be boring. They want them to be boring because boring things present the least resistance
What people want is exactly the first thing you said. They want a package manager that does the things that npm and bundler and pip+virtualenv and cargo and everything else do, but works for all languages and across all platforms, and also provides the functionality of a system package manager.
But Guix and Nix provide exactly this. They just need to be ported to more platforms and processor architectures. Language-specific packages all have giant flaws like not being able to handle dependencies that aren't packages for that language such as C shared libraries, or duplicating dependencies on disk for each project bundle, or relying upon a single point of trust without the option to build from source, non-reproducible builds, non-precise package specifications (loose versioning), etc.
Nix and Guix come with tools that act as a language-agnostic virtualenv called nix-shell and 'guix environment', respectively. For example, if a Guix user wants to build Emacs from source, they can spawn a shell with all of the needed dependencies with 'guix environment emacs' and move on with their lives. It can't get much more boring than that. This is precisely what I want as a user of a package manager: To spend less time worrying about dependency management. Nix and Guix, using the functional package management paradigm, are the only projects that offer it.
So it's specific to C and C++ (and, apparently, also Go). How is that not Language-specific?
[0] https://github.com/clibs/clib
[1] https://medium.com/@tjholowaychuk/introducing-clib-b32e6e769...
Edit: No, it is not.
I install Python locally (w/o the OS-level package manager which doesn't have Python 3.5 yet) and would like to install the SciPy stack which depends on a bunch of C libraries only supplied by the Linux distro I am using.
What I'd Like:
Something like npm's package.json or pip's requirements.txt where I can declare the semantically versioned C/C++ dependencies of a project; doing conan install should fetch and install those dependencies so that pip can (as a future feature) invoke this before installing the SciPy stack and not pause a dozen times complaining about a missing dev dependency.
Does conan solve this problem ? If no, what alternatives do I have ?
Depending on OS-supplied libraries is a royal pain because they are almost always outdated.
I'm all for one idiomatic package manager per language/platform (as it is today for a lot of recent ones) which simplifies dependency management significantly.