Hacker News Reader

C++ apps don't pull in that few dependencies. Take something modern that needs to talk over flatbuffs or protobuffs and maybe has a web ui based on emscripten? Then run ldd on it and recoil in horror. C++ dependency trees now often have a hundred or more items. Many of those are packaged with the OS, but that really shouldn't comfort you.

It kind of does comfort me that they are packaged with the OS, because it's an OS problem (and is relatively major) if they are broken.

It often takes months for distros as large as rhel to fix med. Or even high severity CVE. That is assuming your OS is getting updated properly. Maybe it is a container and you are doing that as part of your pipeline. But all that it means having the dependency in an rpm or deb file vs in a github repo pulled by conan or similar, is that there is someone you can sue that is supposed to notice the cve and roll out a new rpm or deb file. They aren't doing massive pipelines full of fuzzers and custom unit tests for all those packages, they are just... packaging... them. Otherwise we wouldn't be seeing things like the curl and openssl vulns. Yes you might have some recourse to bug rhel to fix it, where a crate author might not care, but that really isn't the experience I am having. If anything I end up pulling dependencies into my system to build if I can so I don't have to wait on the OS company. Between corporate IT and the OS companies delay as middle-men, my OS based dependencies are always the most aged and vulnerable of all my dependencies.

When you make a new library for C/C++, you generally just install it locally at first (pkg-config and PKG_CONFIG_PATH make this trivial). It doesn't (need to) get packaged by a distribution until a program using it is packaged. You don't need to do this yourself, unless you yourself use a distribution and want it packaged there.

As for the amount of dependencies, you have to wonder if it's better to have tonnes of small dependencies or whether it's better to have a few bigger ones. When I need something in a library, I don't make a new library for it, I first try contributing to it, and failing that I vendor and patch it. I feel this is less common in the brave new world of everyone just doing whatever and pushing it to a largely unmoderated repository...

I suppose I could manually traverse my whole dependency tree myself and install all those packages myself. Set up cmake and pkg-config for all of them. If anyone else wants to try out my software, they’d have to manually look up all the dependencies in my github (?) and install those too. And hope the versions all line up nicely - which is a big if during development. Maybe I could script all of that dependency tracing - if only there were some standard software to do that. Guess not.

If C/C++ is all you know, you’re used to how painful all of that is. But what you’re proposing is a significantly worse experience in every way compared to what I can do with cargo or npm today. None of the claimed benefits seem remotely worth the cost of wading back into C dependency management hell. And cargo and npm show just how unnecessary all that pain is in C!

Just make good tooling. We can have nice things. I stopped using C not because I decided to write assembler. I stopped when better languages came along. You want me to stop using cargo? Nobody wants to go back to manual dependency management. Make something better.

If you're smart about it you're not gonna have a very deep or wide dependency tree for your own deps. If you need more than maybe two of your own deps, maybe you need to reconsider how you structure your libraries, or you can join multiple libraries into the same project/build system umbrella (e.g. glib/gobject or the whole of QtBase)

Why is this inherently better than better tooling for dependency management?

Why would I want to join unrelated modules together into a kitchen sink library? The only benefit of that is to work around bad dependency management systems that make dependencies a hassle to work with. But otherwise, they’re a bad idea. Packages should do one thing and do it well. If I want a good random number library, I don’t want to import bob’s junk draw of random code. I want a small, reusable random number library. I want one that I chose. And my users want one they chose. And if that choice is bad, I want to be able to replace it without throwing out half my ecosystem or breaking downstream compatibility.

QtBase is a symptom of C++’s bad package management. 8 unrelated utilities should be in 8 small packages that users can pick and choose based on their needs.

It's less a workaround around bad dependency management, and more about the fact that it's simply easier to reason about a smaller set of dependencies regardless of the ecosystem. This includes both vetting, ensuring version compatibility, checking license compatibility and reducing the SBOM, as well as governance over the project and integration between different modules.

Too often have I seen the equivalents of frameworks in other languages being split over sometimes hundreds of packages, that don't always make it clear that they're to be updated in tandem, what their exact relationship is, and that the same organization manages all of them.

As for QtBase, it's a superproject, but that doesn't mean that you can't use its individual modules separately, and depending on the distro (e.g. debian) install them separately as well. A singular project installing multiple related libraries makes a lot of sense.

> This includes both vetting, ensuring version compatibility, checking license compatibility and reducing the SBOM, as well as governance over the project and integration between different modules.

Auditing large code bases takes disproportionately longer than auditing small code bases. So I don’t think that’s a win. License compatibility is trivial to check. SBOM is strictly larger if I pull in a kitchen sink package because I’m probably not using most of the stuff inside. Better to just pick out the components I want.

The one thing I’ll grant you is that shared ownership and visibility means it’s less likely that one rogue person will sneak ransomware into the dependencies.

Personally I’d love a capability based package manager that lets me pass a package exactly the capabilities it needs to do what I downloaded it for. Why does every package need access to my files when I install or run it? That’s ridiculous. Totally unnecessary and a massive security risk. We could solve that C++ style by hobbling package managers and using fewer, jumbo packages. But that doesn’t solve the root problem. I want a package system in a language which lets me pass capabilities to each library I use, when I need to use them. Eg “Please open your web server on the port associated with this capability token.” This needs language support too but it’d be so much better from a security point of view than anything that came before.

This is an example where your tooling limits your solution space. There’s nothing inherently wrong with a deep or wide dep tree.

> or you can join multiple libraries into the same project/build system umbrella (e.g. glib/gobject or the whole of QtBase)

This also seems worse than the status quo. So instead of having 3 or 4 dependencies I now only one that combines 20 different ones I don't need.

> And cargo and npm show just how unnecessary all that pain is in C!

Leftpad would like a word.

Every dependency is a liability. It's another thing to vet, another thing which can disappear. Having few dependencies in c is a feature, not a bug. The approach to dependencies in rust was a major turn off to me. I recall 15 just to get random numbers in the intro tutorial. No thanks.

Just in case you don’t know, packages can no longer be pulled from npm by the developer. At least not without emailing someone. And you need a very good reason - like that version is ransomware.

The particular pulled package version identifier (foo@1.0.1) can also never be reused. If you audit 1.0.1, it will never change out from under you.

The leftpad fiasco was hilarious and embarrassing. But it can’t happen again because of changes in npm policy.

meson actually solves this problem on the build system, their wrap subsystem allows you to use the system provided lib if found, or fetch from a database during the build process if not.

best of both worlds, imo.

Keeping track of C/C++ dependencies is basically the point of Linux distros...

Cargo's main problem is that it's (mostly) rust only. Same problem that pip/npm have, relegating you to (mostly) single-language silos.

...except pip/npm often build C or C++ while packages install, at least in some cases.

And Rust has the same issues, at least when shipping or consuming libraries with C APIs is in the picture.

Yes, sometimes, but it won't resolve C/C++ dependencies in general.

> Keeping track of C/C++ dependencies is basically the point of Linux distros...

Judging by the proliferation of docker I don’t think they’re doing a great job of even that. Docker shouldn’t be needed if apt was up to the task. Nixos is a much better design imo.

> Cargo's main problem is that it's (mostly) rust only.

True. But apt’s main problem is that it only works on a few distributions of Linux. Limiting the users of my rust library to rust developers is usually not a problem. Limiting the users of my rust library to the people who use homebrew or arch or something is not fine. And I don’t want to redo all the packaging work on every operating system and major Linux distribution. Life is too short for this make work BS. Especially given it could be automated away anyway. Rust packages already explicitly define all their dependencies. Why can’t apt just generate .dpkgs from my Cargo.toml?

Apt’s other problems are that packages need to be installed globally. And they make a mess of versions. And there’s no feature flags. Run `apt search vim` some time and see how many packages they have! Or the same for llvm. It’s a dog’s breakfast.

The time that cargo’s rust focus hurts it is with binary packages. It’s not enough for ripgrep to live in cargo since it’s designed to be useful for people who aren’t rust developers. And I don’t have a good answer for how to solve that. I wish I did.

We'll this also works for rust libraries as they're usually useless for other languages, whereas libraries like gdal, opencv, fftw, etc. can be used by many languages. (Though to be fair, until rust gets a stable ABI it's not going to be a good choice for this sort of thing).

I personally consider docker/podman just completely punting on dependency management, and try to avoid them, but I'm probably in the minority...

Rust has a stable C export ABI.

Funnily enough, the most notable rust-implemented with C ABI shared library I'm aware of (librsvg) just became available on crates.io a few months ago...

> It doesn't (need to) get packaged by a distribution until a program using it is packaged.

I think this is a key difference in approach. In languages with their own packaging systems, you routinely package libraries before there are programs using them. Publishing them on Crates/PyPI/npm/whatever is the bare minimum if you expect anyone to use them!

The number of tiny dependencies can go too far - I don't think I need to mention left-pad. But the difficulty of using dependencies in C/C++, and the results of reinventing stuff, vendoring, or using kitchen-sink dependencies like Qt, don't seem optimal either. There must be a happy medium somewhere.

As I said in another thread, I think the happy medium is lots of small packages (people want that). And a capability security model within programming languages so small dependencies are limited to interacting with their parameters (and any resources their parameters provide them) and can’t speak to the OS directly. That would solve 98% of the supply chain problem.

Leftpad has already been solved by a npm policy change forbidding packages from being unpublished.

> As for the amount of dependencies, you have to wonder if it's better to have tonnes of small dependencies or whether it's better to have a few bigger ones.

You don't have to wonder, we've done the experiment. Anyone working in an ecosystem with non-joke package management knows that tonnes of small dependencies are a lot better.

> In this brave new world the author is proposing - what?

In practice, you just vendor your code as part of the source package. It's the distro maintainers' job to factor it out as a separate dependency.

If you do this, remember to provide an option to factor it out. Meson (mesonbuild.org) makes this completely trivial and pitfall-free, and you should use it if you can't get away with just installing your libraries locally.

I went to mesonbuild.org and it doesn't match the description (some sort of betting site? I didn't stick around ...), and a search turned up:

https://mesonbuild.com/

Thanks, I mistyped it.

>I’m sympathetic to the problem. Supply chain attacks are way too easy

This is the crux as to why crate.io is great: you have one official place for packages that can be continuously audited and monitored. This is the way to do security, in my opinion, because you are always able to look at incoming and outgoing packages.

Now, you can argue if we are doing enough with this, and that is definitely up for debate, however, I contend that the premise of having one canonical place for package origination and hosting makes it easier to monitor for security purposes

That’s like saying it’s best to run everything on AWS, because there’s only one place to monitor for outages.

That’s the exact opposite of how supply chain mitigation works.

It's not at all like saying that.

If you want to mitigate downtime in your supply chain then diversity is good. If you want to mitigate security vulnerability then it's bad.

Imagine you have 100 api servers that all have database credentials. If one server goes down you have 99 working servers (ideally). If one is compromised then the attacker has your database credentials, game over.

I - and many others - don’t think this is enough to mitigate supply chain attacks. It’s too easy for a malicious developer to add cryptolocker ransomware in a point release. But I also don’t think having a package in Debian will somehow magically solve this problem either. Debian package maintainers do not - and never have - done security audits on the packages they add. It’s magical thinking to assume using packages in Debian will somehow protect you from supply chain attacks. Asking developers to keep their dependency trees small so we can minimise the risk is a losing battle. Using lots of small, popular dependencies is just too convenient.

There is a solution out there that I wish got more traction. And that is, capability based security. The idea is simple: we partition programs into application code and library code. Library code can never access the filesystem or network (or other protected data) directly. And this is enforced by the compiler or language runtime. Application code can. If you want a library to interact with a file, you first open the file in application code and pass the file handle to the library. Similar wrappers can be made for directories, network sockets and so on. If I want express to listen on a specific port, I pass it that port in a capability object. Not just an integer. (Eg app.listen(net.listenPort(8000))). The amount of code that would need to be changed is tiny since well behaved libraries don’t usually need many capabilities to do their job.

It is ridiculous that my 3rd party json library can invisibly access all the sensitive files on my computer. With capabilities, we take that privilege away. I have no problem trusting a random json library if it literally only has access to the json string I pass it. If we had a capability based security model, I would have no qualms about pulling in hundreds of transitive dependencies. My libraries can still only access exactly what I pass them.

Unfortunately this problem requires language level buy-in. It would be very hard to retrofit rust to work this way and that’ll probably never happen. Raw pointers and inline assembly also make a bit of a mess of things too. (Though pointer provenance might help). But a man can dream… Maybe in rust’s successor.

You don't need the language to be involved to use this strategy, if you can rely on the OS. Lots of programs are structured this way already: open your secure port as root, then daemonize as an untrusted user.

However, I quite like the idea of using the application/library interface to enforce the restrictions structurally, rather than temporally.

> open your secure port as root, then daemonize as an untrusted user.

That works for server applications. Well - so long as bad things don’t happen in a static initializer before the process drops privileges. But I also want it to work for other kinds of applications. I want to be able to fearlessly depend on a library and not need to worry about that library attacking my computer. Then we can have our cake and eat it too with big dependency trees made up of lots of small libraries written by untrusted contributors.

Another nice thing about this is it doesn’t depend on OS level capability support. Once implemented in a language, it should be pretty easy to make it work on every OS.

Interestingly, this is something wasm provides out of the box. And Firefox’s weird “compile deps with wasm as convert the result back to C” also achieves this end.

Creator of Packj [1] here. How do you envision sandboxing/security policies will be specified? Per-lib policies when you've hundreds of dependencies will become overwhelming. Having built an eBPF-based sandbox [2], I anticipate that accuracy will be another challenge here: too restrictive will block functionality, too permissive defeats the purpose.

1. https://github.com/ossillate-inc/packj flags malicious/risky NPM/PyPI/RubyGems/Rust/Maven/PHP packages by carrying out static+dynamic+metadata analysis.

2. Sandboxing file system w/o superuser privileges: https://github.com/sandfs/sandfs.github.io

Cool library!

I’m not imagining something that specifies policies at all. Capability based security systems shouldn’t need them. Instead, the main application can create capabilities programmatically - for example by opening a file for reading. The file handle is a capability. Libraries you call don’t get access to the OS, or anything outside their module scope that hasn’t been passed in as an explicit parameter. A 3rd party library can access the file handle you pass it as a function argument, but it can’t ask the OS to open any other files on your computer. Or open a network socket.

OS APIs aren’t in a scope that any of the libraries you use have access to.

There are a few things capabilities need to be able to do. They need to be able to be tightened. (So for example, if you have a file with RW flags, you can make a copy of the file handle that’s read only). And they need to be unforgeable (file handles can’t be created from an integer file descriptor). We’d also need a handful more “file handle” like objects - and this is where granularity matters. Like we need an equivalent for directories that gives you access to everything recursively in that directory and nothing outside of it.

If the library I call delegates some work to another library, it’s pretty simple. It just passes its file handle through to the other library as a function argument. There’s no policy to maintain or update. Nothing even changes from the point of view of the original caller.

The tricky part is stuff like debug output, environment variables (NODE_ENV=production) or stdout. Do libraries implicitly get access to stdout or not? Maybe they get stderr but not stdout, and they can open new temp files? You would also need to restrict JavaScript code from arbitrarily messing with the global scope or global libraries like String.

But honestly, any answer to these problems would be a massive improvement over what we have now. Being able to know your binary search library can’t make network requests, and that your database can’t interact with files outside its database directory would do so much for supply chain security it’s not funny. A good capability system should just change the APIs that your software can use a bit. Generally in good ways that make dependencies more explicit. Done right, there shouldn’t be any new syntax to learn or deno style flags to pass at the command line or anything like that.

This is similar to how WASI (standard system interfaces) for WASM work.

Yeah I'm really keen for that - though I think compiling a bunch of dependencies into individual wasm modules will be kind of awful to use in practice.

Apparently in firefox they've been compiling some of their dependencies using wasm, then translating the wasm back to C. The end effect of all that is that it gives C code wasm's sandboxing and safety. I wonder what it would look like to do that without needing wasm as a middle step.

You create apt, rpm repos.

And users install from there.

Not saying that's a nice answer, but that is the answer.

Users install from there, so long as they're using Linux, a Debian or Red Hat based distro, and have sudo access. As you say, not such a nice answer.

For certain kinds of software, this approach is getting more viable, thanks to containers. In some contexts, you can tell everyone to run your software in a container with a specific Linux distro inside. But that software is often distributed as container images directly, rather than distro packages to be used in containers.

You create apt, rpm repos. And arch. And gentoo. And FreeBSD ports. And homebrew and macports. And whatever windows does. And you need that to be true, on all those systems, for all your dependencies too. And at a recent, usable version.

Or, I just use cargo. Up to date version. One command install that works on every operating system. Project scoped dependencies - so there’s no need for sudo, and different projects can depend on different versions of their dependencies. And I can publish new versions of my packages with 1 command and they’re available everywhere.

Just in my house I have a Linux mint workstation, a windows gaming pc, a FreeBSD server and a Mac laptop. Cargo lets me publish code that all of them can consume with 1 command. So does npm.

So long as apt is worse than cargo I won’t use it for this stuff. Why does everything have to get installed system wide? Nix doesn’t need that. Why doesn’t it have a programmatic mirror of cargo so we don’t need to mirror rust projects in it by hand? Why do major versions have different package names? And why don’t packages support feature flags? Last time I looked there were about 30 vim packages with random combinations of feature flags set. Why can’t evergreen packages like nodejs or cargo stay up to date in Debian? Old nodejs is useless because it can’t run modern JavaScript.

I really want apt to solve all these problems because I hate docker. If apt and friends did their job well, docker wouldn’t be necessary or popular.

Cargo isn’t pretty either. But it has all the features I need to get my job done. Make apt a compelling tool and I’ll switch to it. The feature list developers want is hardly a mystery.

Correct, there are many operating systems.

That is why each modern language has introduced their own package manager.

I assume you’re being flippant, but it’s kind of true. You support 20 different things by making a 21st thing that papers over all of them. And along the way you can try to fix some of the design problems.

The web. The C programming language. Electron. It’s a tale as old as time.

> as an author of some not very popular packages, what do I do?

Depends. Do you want your software to be used by users? For me as a user, software that is not in distribution repository almost does not exist. It is too much hassle to install, manage and uninstall it compared to installing software from Linux distribution. If i really, really need it, i will make exception, but that are individual cases.

> “It works for C/C++” by making developers working in those languages pay in blood for every dependency they pull in.

I think the industry may have thrown the baby out with the bath water. Now they're is a complete aversion to RYO anything, no matter how simple. Personally, I'm waiting for the pendulum to swing back towards the middle a bit more.

Eh. I love rolling my own lots of things. At some point I really want to make a little OS kernel. How hard can it be?

But then there’s things like rust’s rand crate which implements about 10 different random number generators, all with different properties around speed, whether they’re cryptographically secure and so on. And fed from a variety of entropy sources to make it work everywhere from Linux to windows to wasm. Look at this documentation. It’s an absolute master class.

https://rust-random.github.io/book/guide-rngs.html

Do you need this? Probably not. You can probably get by with some wrapper around ar4random or whatever the recommended entropy source is today on your particular operating system. But I’ll be honest, until the day I decide to care about how rngs work, I’m going to just import the rand package. And serde & friends for json support. Criterion for benchmarking. And so on.

Cargo is full of great libraries that are much higher quality than anything I’d ever throw together in a side project. For most of this stuff, you’d be mad to roll your own.

This thread has reminded me that lots of people don’t know how nice it is to use a package manager like cargo. Here’s my favorite crate browsing website. You can see the most popular packages in every category and search. Look how much great stuff is in here. It’s humbling.

https://lib.rs/

>Yeah. The part that gives me goosebumps with that proposal is: as an author of some not very popular packages, what do I do? Is it my job to package and promote my packages on Debian, redhat, homebrew, and all the others? Do I need usage before they’ll pick it up? Sounds like a chicken and egg problem. And doing all that packaging work sounds like a boring, time consuming job that I don’t want.

Github premium (or similar) will have an army of AI builder bots that spin up VMs and modify your codebase until it works, then send you PRs to approve. People would pay a ton for that service. All your projects, automatically maintained and packaged.