Single_file_libs: List of single-file C/C++ libraries
github.com
github.com
Managing dependencies in C++ is a nightmare, and I'm not even talking about the lack of actual modules inside the language itself. It's so bad that people (incl. me) tend to prefer to copy-paste libraries source code instead of setting up a dependency.
A Python, Ruby, Node or Java project would never have to head toward such a poor solution instead of using their respective package managers. But in C++ we do because the alternative (e.g. Conan) is such a nightmare to use.
PS: the original motiviation for the STB single-file libs was actually that Windows doesn't have a default place where dependencies are installed:
Just because you have five header files instead of five packages doesn't mean you can use all five headers at the same time. For example, some might require C++03 and some might need C++14 or newer.
Also, if all dependencies are single file with no non-standard dependencies, that means none share dependencies they should, like threading, telemetry, or logging libraries.
Anything in its own compilation unit would be isolated. Anything written using C++03 should compile without too much trouble in a more modern compiler.
Also threading is part of the C++ standard library.
I don't know what you mean by sharing dependencies on telemetry and logging. That sounds like adding questionable complexity to libraries that are simple and modular.
In isolation yes. But if it interacts you get events like the fact that gcc changed std::string between versions (from refcounted to one with small buffer optimisation for C++11 compliance) also introduction of move semantics and rvalue references can change interpretation of the same class declaration between compilers in different modes.
You seem to be talking about major changes to the fundamental set up of a project while not taking a comparably trivial amount of time recompiling compilation units.
How often are you changing major compiler versions and breaking standard library versions that you would have this expectation?
You've never linked to a binary? I have no idea what debian's Qt was build with or libssl or SDL etc were built with but we can link to them all the same.
> How often are you changing major compiler versions and breaking standard library versions that you would have this expectation?
You don't configure Jenkins to build against multiple compilers/versions of compiler? Isn't that super normal?
I've had so many issues with shared libs on linux in the past, I gave up on it and started to include all the necessary source of any third party library in my projects. Makes building and distributing things much easier.
No, I think there is no other way to see it.
Single files are a modern and unified package management platform. You copy the file and you use it. It does not get any simpler than that.
You are talking as if there was a kind of compromise between some imaginary disadvantages of single files. There are none, and there is no compromise. Single files are alright.
Certainly, if anyone finds that things have got desperate enough that they're looking the OP's list of single-file libraries then they should just take the one-time hit of setting their project up to use vcpkg. Once that's done, using a supported library reduces to running `vcpkg install foo`, regardless of how many source files it has (or how many transitive dependencies for that matter).
Can trivially support multiple incompatible versions of the library, nothing needs to be fetched or resolved (once you've got the single file of course), can send and distribute it easily over any channel you want (web, email, free file host, google drive, your own website, etc).
This approach lacks any features that reasonable dependency manager provides, such as providing updates compliant with semver.
* You have to wait for compilation of the whole library at least once every time you do a build of your program - certainly every time you do a clean build, and potentially even incremental builds if it's header only.
* If the library is header only (many of the linked libraries are) then you you potentially have to pay that compilation cost more than once per compilation of your program - once per every one of your source files that include it.
* Again this is specific to header-only libraries, but to avoid code bloat you'll need to turn on link-time optimisation which is far slower than just allowing the linker to do its job by only compiling definitions into a single object file. (Admittedly LTO is a good idea anyway, but adding a bunch of duplicated symbols is avoidable extra work for it.)
* Some useful libraries are realistically just too big for their authors to write the whole thing in one file (e.g. protobuf, opencv, ... in fact most libraries I use on a regular basis seem to fall into that). They could "release" the library in single-file format, similar to SQLite's amalgam, but then if there are any problems (either a bug in their code or something in your code that makes you want to look at their code) you're now not looking at the original source but some mangled version of it.
* If the library is so large that its interface needs to be split over multiple headers (think Boost or OpenCV) then you're now bang out of luck. Hopefully the library has cleanly-enough separated modules you could potentially release these separately (e.g. OpenCV core, imgproc, imgcodecs, highgui, ...) but then you're essentially back to multi-file libraries.
* Adding a library with a lot of its own transitive depedencies takes proportionally the amount of effort as the number of those dependencies, rather than being handled automatically.
One interesting thing about all of these problems is that they get worse and worse as you need more libraries in your program, or need a larger library for your program. In contrast, using a package manager (I'm thinking particularly vcpkg here) tends to add a one-time cost at the start but allows you to scale your dependency list almost for free.
If you're writing your own library, rather than a application, then there are even more problems with this approach, but I won't quite open that can of worms.
Even visual studio's compiler compiles sqlite's 6MB in under a second.
> Again this is specific to header-only libraries, but to avoid code bloat you'll need to turn on link-time optimisation
This is nonsense. This list seems like you are trying to invent problems that aren't there. The vast majority of the time you decide what compilation unit to put the definitions into and that's it.
SQLite is notably pure C, which compiles orders of magnitude faster than C++, or at least certain types of C++.
I've worked on a project - not particularly large - where using precompiled headers reduced compilation time from something like an hour and a half to more like 15 minutes. Admittedly that's a very old crusty laptop, but that's without building the libraries, which are compiled separately! Even on much faster modern machines, building the dependencies is at least an hour, maybe more.
I wonder if our opinions are so strongly at odds because we're simply working in totally different situations to start with. As I hope I've illustrated, compilation time definitely IS an issue for us, but obviously it's not for the projects you're working on - which must surely be C-based or at least C-like C++ code. I would wager more people are in my situation, but to some extent that's irrelevant. The advice to other devs has to be: if you're happy to restrict yourself to small C or C-like libraries then single file libraries are fine, but if you want to take advantage of the full C++ ecosystem be aware that there package managers (again, I'm mainly thinking vcpkg) you can use with only a little initial effort. I think it's dangerous for beginners to see articles like the one linked to here in case it makes them think that using C++ has to be like that.
> > Again this is specific to header-only libraries, but to avoid code bloat you'll need to turn on link-time optimisation
> This is nonsense. ... The vast majority of the time you decide what compilation unit to put the definitions into and that's it.
I said that this paricular objection obly applies to header-only libraries, and that qualification is even in the bit you quoted. I never claimed that there are no single-file libraries that allow separate compilation.
But I do dispute that the "vast majority" allow or require separate compilation (either as a separately supplied .c/.cpp file, or by #defining something before one of the uses of the header). That is the exact opposite of my experience. As an experiment I looked at all the libraries listed under "argv" in the linked article, and 9 of them were header only with no option for separate compilation (Argh!, Clara, CLI11, cmdline, flags, kgflags, linkom, optionparser, ProgramOptions.hxx) while only 1 (parg) allowed separate compilation.
But is it due to 'single file libraries'? What I've seen is that having fewer, fatter compilation units speeds up compilation the same way 'unity builds' do. Instead of having a single monolithic compilation unit, a balance can be used to iterate faster and use multiple cores. Compilation units can be groups of things that are changed more or less frequently.
> But I do dispute that the "vast majority" allow or require separate compilation
It isn't about the library always being made for separate compilation. The reality is that unless you are using something fairly fundamental it will probably only need to be in one compilation unit in the first place.
What does need to be shared are data structures and those ideally have as little logic and dependencies as possible.
> But is it due to 'single file libraries'?
That's a fair point, they're generally not. I just meant that if those libraries were single file, but otherwise unchanged, then the problem would only be worse. I see what you're saying about unity builds, but if you build your dependencies totally separately from the rest of your program then it's not so important exactly how fast they build. I'm imagining a situation where you leave vcpkg to build all your libraries overnight, then you spend the next few weeks iteratively working on your application code (which is how it usually works for us).
> The reality is that unless you are using something fairly fundamental it will probably only need to be in one compilation unit in the first place.
In the projects I work on, which are really quite varied, many (but not all) libraries are used throughout the codebase as vocabulary types. Again, I think this just comes down to very different codebases and uses of libraries for us.
I don't know what 'vocabulary types' are, but I think this is an oversimplification. My guess is that most big programs end up like yours but my point is that it doesn't have to be this way. Single file libraries are not the problem and can actually help, although the real issue is deeper.
I think the problem is actually dependencies and many times single file libraries go to great lengths to not depend on anything else. To keep it as simple as possible, I think classes / data structures get over used and instead of just making an interface to some data, people stuff everything they are doing into some class they have.
Transformations from one type to another put into data structures means that the class now has dependencies on those other types. If those types have transformations then they have dependencies and so on. The extrapolation is that everything depends on everything else and even compilation units that should be tiny end up pulling in large parts of the entire program as source code. Lots of compilation units can then mean compiling huge chunks of the source hundreds of times.
In addition, although the name of the repo is "single_file_libs", it links to many double-file libraries consisting of a pair of .c and .h files. These libraries won't have the issues you are talking about. Developers are well aware of the potential linking problems.
That said, you are right that single/double-file libraries tend to be small. It is hard to work with a file with >10,000 LOCs anyway.
I never claimed to be writing only about his libraries, but instead about single-file libraries in general. Sorry if I wasn't clear about that.
> For his (and many others') libraries, you "instantiate" the implementation in a .c file and declare functions in other source files. This way you only compile the library implementation once.
I tried to make clear which of the bullet points only applied to header-only libraries (again, apologies if it wasn't clear). For example, my objection to compiling the same library multiple times specifically applied only header only libraries. As you say, with two-file (or single .c file, if you like) libraries you only have to compile the library once per build of your program - but as I made clear in a separate point, that is still potentially a pain if you do a clean build (but it could be mitigated by having a separate CMake target, or whatever, that contains all your library files).
> In addition, although the name of the repo is "single_file_libs", it links to many double-file libraries consisting of a pair of .c and .h files. These libraries won't have the issues you are talking about. Developers are well aware of the potential linking problems.
As I just said, I had tried to distinguish those cases. They're less problematic, but still somewhat problematic.
> That said, you are right that single/double-file libraries tend to be small. It is hard to work with a file with >10,000 LOCs anyway.
Yes, and if you only need a small number of small libraries then all my objections are a lot less severe. To spin the conclusion on its head - you can avoid going down the package manager route if you avoid having any large dependencies. For some projects that's acceptable. Personally, I find the pain of working without basic largish libraries like protobuf to be enormous compared to the relatively simple process of using a package manager.
The owner of this repo is a well known figure on game development. His libraries are somewhat widely used and have probably inspired quite a few single-header libraries which adopt similar strategies. My comments are not specific to his libraries, either.
I know. In my original comment I tried to make it clear that some (but not all) of my points applied to these libraries that have some separate compilation. In my second comment I apologised for not making that clear and tried to stress it even harder. I'm not sure what else I can do to make it clearer.
> The owner of this repo ... His libraries
If you just use this guy's libraries, and that works out for you, then all power to you.
But I'm worried about C++ beginners stumbling across this page on Hacker News and thinking, hmm, using C++ libraries is so hard that I need to restrict myself to libraries like these. That is simply a myth - using vcpkg is really not hard at all, and well worth it for all the other libraries it gives you easy access to. Again, if that doesn't matter to you then that's fine, but I want readers to know that the option is available.
By the way, that the vast majority of libraries linked to from this article are not by the author and do not have the separate compilation model you talked about - they're simply header only with inline functions.
Which is precisely why C++20 has modules support.
Further, C++ does have a central package manager - whatever the system distributes. C/C++ programs constantly link against the operating system and thus have to get those system headers from somewhere. This means a universal package manager (like Conan) requires buy-in from OS vendors or at least someone who can manage those packages with conviction, else the package manager doesn't make a lot of sense to use.
This is contrast to Node, Ruby and Python as they are inherently Cross-Platform in nature and are not coupled tightly to the OS.
This is why C/C++, historically, have not had a strong centralized package manager.
Anymore, I see less and less package management being used anyway. Most C/C++ projects I work with either vendor in dependencies or use git submodules (the latter I prefer very much).
This is because, unlike e.g. the Node community, micro-dependencies are generally not worth the effort to bring in.
The usual exception to this sort of structure are codebases that will be distributed via a package manager - in which case, they generally rely on the package manager supplying correct versions of the dependencies.
These header-only libs work OK usually because they are able to be included with trivial linkage (something else you normally don't need to care about in scripting environments). Linkage is something you have to care about regardless of if you're writing C, C++ or machine code and since these languages give you pretty much free reign on the flexibility of all of these parameters, it's hard to generalize everything into a nice package like scripting languages can.
Usually the functions inside header only libraries will be static inline, and will almost certainly increase compilation time (noticeably so if you use the header in many places).
There is work being done to improve this situation, it's not like we're all just sitting here going "yes, we love the way things are and have no idea how to make things better." That's far from the case. Shit just takes time.
One counterpoint to your proposed cause & effect is that some observers think Javascript NPM's packaging convenience enables the explosion of single file dependencies. Example discussion:
What’s happening in C++ is that you’re getting one-file libraries that are not small, because developers will shoehorn their library into a single file rather rather than deal with a way of managing C++ library dependencies.
You might find in C++, for example, a header file with over 10k lines in it and a bunch of preprocessor conditionals. What you might find in JavaScript is a library that contains one or two functions.
Maybe we should a couple of You Tube videos about linkers and stuff.
I literally wrote a compiler but still spend 15/20% of my development time trying to find the right not user-friendly CMake syntax, understanding while my Conan dependency broke overnight, or writing header files that could be automatically generated by the compiler (à la GHC).
As for the headers, C++20 modules, or just use an IDE like Visual Studio and the respective wizards.
This works for simpler libraries and templates, obviously, but once the function becomes less trivial, it should really be going into a separate source file.
Readability and manageability aspects aside, keeping implementation in a header causes all dependencies of that implementation to be pulled into your code as well. Including the heavy platform-specific stuff. If I need to do some string conversion, I really don't care for the implementation sneaking in <windows.h> into my sources just so that it can call WideCharToMultiByte.
For that to happen you'd need to have a separate .c to include just that .h with IMPLEMENTATION defined. So in the end there's still an .h and a .c, except the .c is now completely superficial.
You can still put multiple implementations into the same source file and compile that into a library, for example:
https://github.com/floooh/sokol-samples/blob/master/libs/sok...
That's what I usually do for projects that don't just consist of a single source file (e.g. simple command line tools).
An actual advantage of this approach is that you can add any configuration defines in that same source file, instead of passing them in from the build system via compiler command line args.
Also for bigger projects you can have dozens or hundreds of header files but only a small number of source files (e.g. one source file per "system" or split by change frequency, or by any other criteria (like whether the implementations include Windows.h or other system headers). A small number of compilation units means fast compile times (since it's essentially a unity build), but at the same time you have enough control over the project's file structure to balance compile times vs "namespace pollution" vs what changes trigger a rebuild.
There’s nothing questionable about this practice, it’s pretty widely used throughout the industry.
I tend to think of single-file libs as a "poor man's module system". The entire "module" is in a single file that you drop into your project, and you're ready to go (and ironically it's a lot more straightforward than the C++ module system).
The real-world differences between single-file and header/source pair aren't such a big issue as many people make it out to be though, either way is fine. The actual problem is libraries that consist of dozens or hundreds of source files and a complex build system setup.
In particular, stuffing non-inline functions into .h is, ultimately, a malpractice. Yes, you get to have a single .h, but you end up creating an instance of each function for every .c that includes this .h.
Example - https://github.com/tronkko/dirent/blob/master/include/dirent...
IMO, if you need a #define like this, you're are not really writing a header-only library, that's basically just a source file in disguise. You could just as well distribute headers + a single source file and just ask the user to add this one source file to their build system (which should really be a trivial task).
EDITED for more clarity about C vs C++.
<grumble>
As a sidenote, I remember my brain hurting when I learned about C++ just redefining `inline`. It would have made much more sense to just define a new keyword.
</grumble>
I would bet they are aware of most of the reasons that you might come up for why single file libs are not good, and yet they authored a bunch of them.
As another poster remarked in this discussion, he has a good overview for how to create a robust single header library: https://github.com/nothings/stb/blob/master/docs/stb_howto.t... If you can offer some good arguments against those, instead of generic dogma, then we can have a discussion. :)
If you can first understand the point of a comment, instead of not understanding it, the we can indeed have a discussion.
EDIT: I see now that this advice is only valid for C++, where the "inline" keyword creates a weak symbol. I forgot that this doesn't work in C.
Anyway, the #define trick is only necessary for C, in C++ you would simply declare all functions as "inline" (which automatically happens for function templates, btw)
https://github.com/nothings/stb/blob/master/stb_truetype.h#L...
The only exception I can think of is accessing lots of indexes in a loop, where getting good performance requires rust programmers to insert awkward asserts or casts to fixed length arrays to get the checks to optimize out[1]. But afaik that's mostly because the bound checks impede other loop optimisations like vectorization, not because the checks are slow themselves.
[1] (e.g. when you access indexes 1 to n randomly, assert n < length before)
https://dropbox.tech/infrastructure/lossless-compression-wit...
Shows a significant improvement in compression code and some examples of checks that can't be elided. I guess this makes sense because it's pretty much the worst case scenario for bounds check impact.
I can still personally say that every time I've blamed rust's bounds checks, I was wrong.
EDIT: neither the unsafe flag nor the macro appear to be present in https://github.com/dropbox/rust-brotli/ anymore today, removed some time in 2017. So it seems they found some other way to deal with it?
Some examples of when this is helpful:
- using gcc's / clang's sanitizer frameworks
- tweaking builds for deep-dive optimization and/or debugging
- letting the compiler target a specific hardware architecture, e.g. `-mavx512f`.
I guess this highlights a limitation in typical Linux systems: a given library can be built with many different configuration options, but there's no (widely known?) way to have all those variants installed at the same time.
I think fixing this would require more than just changing cmake, apt/dpkg, or the C++ standard. A clean solution might also require changes to the ELF standard and/or Linux's compilers, linkers, dynamic loaders, and debuggers.
Later I came across a quote where he said something to the effect that what’s the point of a linker that slower than the compiler.
Sometimes I think what’s the point of this tangle of make files (or cmake/conan or whatever). It’s more difficult to get right than one big single source file.
For all my personal projects, I use a single main.c file which #includes the topologically sorted .c files for each module, one file per module, preceded by a shared #include block for external library headers. If your module dependency graph is acyclic, you don't need any per-module headers; inside a module everything is sorted and you only need forward declarations for mutually recursive functions/types. The only real downside for me is that it breaks 'static' isolation. Even the slower C compilers like gcc, clang and msvc can build 50-100 kloc/sec on my aging laptop with code structured like this.
You get qualify of life benefits like fast, simple builds (one-liner scripts for building on each platform, no need for the compiler to chew through the same monstrous system headers for every .c file, fewer symbols for the linker to resolve) and less boilerplate you have to write (no redundant copies of declarations in headers, no redundant #include blocks at the top of every .c file). In case it's something you care about, structuring your code like this also makes it trivial to automatically amalgamate your entire project into an stb-style single-file header library for distribution purposes: it's already 'static' safe, you've already specified the topological order and you've already eliminated all the redundant boilerplate you wouldn't want in a single-file distribution, so it's mostly just a matter of writing a script that inlines the #include "foo.c" directives in the main.c file.
There is a breakpoint between invoking gcc manually and having a build system like make or cmake that is quite annoying and this would be a nice bridge.
The flip side of this technique is to be careful about blowing up the size of the final executable since everything gets included. You may need a linker pass to remove unused sections from the final executable. One good example to manage this is found in the dinkumware C library where each function is in its own .c (and hence corresponding .o) file. This way only the functions actually used get linked into the final executable.
Yeah, the 1970s linker model is very silly like this. The good news is that link-time optimization will handle this for you nowadays without any special per-symbol mark-up. As an experiment, put an unused 'deadfunc' function definition in a file called ltotest.c and then compare
gcc -O2 ltotest.c -o ltotest && (objdump -D ltotest | grep deadfunc)
gcc -O2 -flto ltotest.c -o ltotest && (objdump -D ltotest | grep deadfunc)There is a point at which multiple files and make systems make sense (no pun intended). Unfortunately, it seems most people overengineer at the start rather than adding complexity as needed.
The IMPLEMENTATION define cleanly separates the public API declarations (which are required in each source file where functions from this library are called), from the implementation code and any implementation-private declarations.
If you put all the implementation code into inline code, this code is parsed over and over again in each source file which would only need to include the declarations.
This is the main reason why C++ projects compile so slowly, and what gives "header-only libraries" such a bad reputation in the C++ world.
"STB-style single file libraries" don't suffer from this problem.
but it doesn't work the same way.
> If you put all the implementation code into inline code, this code is parsed over and over again in each source file
That's true. Good header-only C++ libraries provide "forward" headers, which you can use if you don't need the full definition.
But the best thing is to only write C++ header-only libraries if there is a technical reason (e.g. heavy templated code).
> "STB-style single file libraries" don't suffer from this problem.
That's right, but I also don't see the benefit. To me, telling the user to add a #define to some random source file feels incredibly hacky. On a technical level, it's the same as asking the user to #include a ".c" file.
Why not put the implementation in a single ".c" file, which the user can add to their project? I honestly don't understand this...
Just looking through the list of single file JSON libraries, "nlohmann/json" is buried in the middle (most popular C++ json library) and surrounded by small projects.
Also, following the reasoning line of your comment you can make a parallel conclusion: the C/C++ community and level of adoption are so big that the people of Rust & Go had to put together a "decent & standardised build system" to gain traction and consideration.
With that being said, having a standardized build system would be nice, but it wouldn't solve the underlying problem here which is that portability is often not regarded as important enough to merit thinking about and designing for at the start of a project.
The consequence of this is that it creates a system in which C programmers have very little trust in one another's code.
I write a lot of C for ARM and I am often hesitant to use libraries that I can't inspect in a reasonable amount of time because I have been burned by e.g. liberal use of type punning which results in bus conflicts as it's idiomatic and works just fine on x86, but is ultimately undefined behavior in most circumstances. (Rant: Just use memcpy, please! Argh!)
I really don't like looking at 10k+ line files..
Rax is a radix tree implementation initially written to be used in a specific place of Redis in order to solve a performance problem, but immediately converted into a stand alone project to make it reusable for Redis itself, outside the initial intended application, and for other projects as well.