Rust without crates.io
thomask.sdf.org
thomask.sdf.org
This does not reflect my observations at all. It works if you’re an end user and the software you want has been packaged. But if either of those is not the case, you’re left in the mud. Rather, you end up with builds that are difficult to set up or reproduce except on specific platforms, require a lot more maintenance over time because bitrot occurs naturally (your code depends on libexample 1, but Debian only package version 2 now, so good luck dusting off that project you made five years ago and getting it to build again), vendoring upon vendoring, more diverse and generally lower-quality dependencies for a particular task, and other problems like these. You’ve solved one problem, but at a very significant cost that must be acknowledged, even if you decide it’s still worth it.
Take an app written five years ago that depends on majorish C++ libraries, and there’s a fair chance compiling it will be a nightmare and require invasive surgery due to version incompatibilities. It’s not improbable that the most pragmatic solution may be to find and rebuild the transitive package tree for some older version of some package, and that’s generally quite painful (and can be outright incompatible).
Take an app written five years ago in pure Rust, and it’ll almost certainly (p>0.99) still work fine. Take an app written five years ago in Rust that binds to some C/C++ library, and there’s a fair chance you’ll run into trouble, but it will almost certainly be a good deal easier to resolve, much more likely to work with only a version bump and no other changes than in C/C++.
As a developer, I know which general mode of operation I prefer. Debian are welcome to twiddle the dependencies in the way that suits their chosen requirements, but the canonical sources are still crates.io, and I know that when they’ve stopped packaging example-1.0.0, I won’t be beholden to their packaging but can just use `cargo build` and it’ll still work (contingent upon extern bindings).
I’m dubious that any meaningful curation is happening, and curious if there’s even cursory review of crate contents. But most of all, I’d be interested in a statistical comparison of the crates that Debian ships with the most-downloaded crates from crates.io. I suspect the sets would look very similar.
It’s also quite common for me to pull out bits of code I’ve written that I want to share as a dependency between different programs I write. I have dozens of packages in npm along these lines - and it’s quite elegant. I factor out the code, package it independently, write a readme and some tests and publish it. My other packages can then just depend on the brand new package directly. In this brave new world the author is proposing - what? I make a .deb, argue for it on some mailing lists I’m not on, wait who knows how long, eventually gain permission to depend on my own code, then realise I can’t install it on my Mac laptop and do the same dance in homebrew? Oh - there’s a bug. Better manually update all the package repositories? Sounds like a complete nightmare to me.
I’m sympathetic to the problem. Supply chain attacks are way too easy. But maintaining up to date packages on every single Linux distribution, homebrew, ports, and whatever windows does sounds like a full time job. A boring full time job. And it would still not work as well for my users, because when a new feature is released in one of my upstream dependencies I can’t use it until all of those package repositories ship the new version of my upstream dependency too. I’d rather eat sand.
“It works for C/C++” by making developers working in those languages pay in blood for every dependency they pull in. You can tell by looking at just how few dependencies the average C program uses. It’s like that because of how painful it is. Anything under a certain size tends to get either reinvented every time, badly (hash table libraries in C) or vendored (eg json parsers). Common libraries like rust’s rand don’t exist in C because it’s just too much effort to pull in dependencies. So everyone rolls their own crappy version instead. Ask GTA5 how well that went for them. And that’s not the developer ecosystem I want to work in. It’s frankly a terrible proposal.
As for the amount of dependencies, you have to wonder if it's better to have tonnes of small dependencies or whether it's better to have a few bigger ones. When I need something in a library, I don't make a new library for it, I first try contributing to it, and failing that I vendor and patch it. I feel this is less common in the brave new world of everyone just doing whatever and pushing it to a largely unmoderated repository...
If C/C++ is all you know, you’re used to how painful all of that is. But what you’re proposing is a significantly worse experience in every way compared to what I can do with cargo or npm today. None of the claimed benefits seem remotely worth the cost of wading back into C dependency management hell. And cargo and npm show just how unnecessary all that pain is in C!
Just make good tooling. We can have nice things. I stopped using C not because I decided to write assembler. I stopped when better languages came along. You want me to stop using cargo? Nobody wants to go back to manual dependency management. Make something better.
QtBase is a symptom of C++’s bad package management. 8 unrelated utilities should be in 8 small packages that users can pick and choose based on their needs.
Too often have I seen the equivalents of frameworks in other languages being split over sometimes hundreds of packages, that don't always make it clear that they're to be updated in tandem, what their exact relationship is, and that the same organization manages all of them.
As for QtBase, it's a superproject, but that doesn't mean that you can't use its individual modules separately, and depending on the distro (e.g. debian) install them separately as well. A singular project installing multiple related libraries makes a lot of sense.
Auditing large code bases takes disproportionately longer than auditing small code bases. So I don’t think that’s a win. License compatibility is trivial to check. SBOM is strictly larger if I pull in a kitchen sink package because I’m probably not using most of the stuff inside. Better to just pick out the components I want.
The one thing I’ll grant you is that shared ownership and visibility means it’s less likely that one rogue person will sneak ransomware into the dependencies.
Personally I’d love a capability based package manager that lets me pass a package exactly the capabilities it needs to do what I downloaded it for. Why does every package need access to my files when I install or run it? That’s ridiculous. Totally unnecessary and a massive security risk. We could solve that C++ style by hobbling package managers and using fewer, jumbo packages. But that doesn’t solve the root problem. I want a package system in a language which lets me pass capabilities to each library I use, when I need to use them. Eg “Please open your web server on the port associated with this capability token.” This needs language support too but it’d be so much better from a security point of view than anything that came before.
This also seems worse than the status quo. So instead of having 3 or 4 dependencies I now only one that combines 20 different ones I don't need.
Leftpad would like a word.
Every dependency is a liability. It's another thing to vet, another thing which can disappear. Having few dependencies in c is a feature, not a bug. The approach to dependencies in rust was a major turn off to me. I recall 15 just to get random numbers in the intro tutorial. No thanks.
The particular pulled package version identifier (foo@1.0.1) can also never be reused. If you audit 1.0.1, it will never change out from under you.
The leftpad fiasco was hilarious and embarrassing. But it can’t happen again because of changes in npm policy.
best of both worlds, imo.
Cargo's main problem is that it's (mostly) rust only. Same problem that pip/npm have, relegating you to (mostly) single-language silos.
And Rust has the same issues, at least when shipping or consuming libraries with C APIs is in the picture.
Judging by the proliferation of docker I don’t think they’re doing a great job of even that. Docker shouldn’t be needed if apt was up to the task. Nixos is a much better design imo.
> Cargo's main problem is that it's (mostly) rust only.
True. But apt’s main problem is that it only works on a few distributions of Linux. Limiting the users of my rust library to rust developers is usually not a problem. Limiting the users of my rust library to the people who use homebrew or arch or something is not fine. And I don’t want to redo all the packaging work on every operating system and major Linux distribution. Life is too short for this make work BS. Especially given it could be automated away anyway. Rust packages already explicitly define all their dependencies. Why can’t apt just generate .dpkgs from my Cargo.toml?
Apt’s other problems are that packages need to be installed globally. And they make a mess of versions. And there’s no feature flags. Run `apt search vim` some time and see how many packages they have! Or the same for llvm. It’s a dog’s breakfast.
The time that cargo’s rust focus hurts it is with binary packages. It’s not enough for ripgrep to live in cargo since it’s designed to be useful for people who aren’t rust developers. And I don’t have a good answer for how to solve that. I wish I did.
I personally consider docker/podman just completely punting on dependency management, and try to avoid them, but I'm probably in the minority...
I think this is a key difference in approach. In languages with their own packaging systems, you routinely package libraries before there are programs using them. Publishing them on Crates/PyPI/npm/whatever is the bare minimum if you expect anyone to use them!
The number of tiny dependencies can go too far - I don't think I need to mention left-pad. But the difficulty of using dependencies in C/C++, and the results of reinventing stuff, vendoring, or using kitchen-sink dependencies like Qt, don't seem optimal either. There must be a happy medium somewhere.
Leftpad has already been solved by a npm policy change forbidding packages from being unpublished.
You don't have to wonder, we've done the experiment. Anyone working in an ecosystem with non-joke package management knows that tonnes of small dependencies are a lot better.
In practice, you just vendor your code as part of the source package. It's the distro maintainers' job to factor it out as a separate dependency.
This is the crux as to why crate.io is great: you have one official place for packages that can be continuously audited and monitored. This is the way to do security, in my opinion, because you are always able to look at incoming and outgoing packages.
Now, you can argue if we are doing enough with this, and that is definitely up for debate, however, I contend that the premise of having one canonical place for package origination and hosting makes it easier to monitor for security purposes
That’s the exact opposite of how supply chain mitigation works.
If you want to mitigate downtime in your supply chain then diversity is good. If you want to mitigate security vulnerability then it's bad.
Imagine you have 100 api servers that all have database credentials. If one server goes down you have 99 working servers (ideally). If one is compromised then the attacker has your database credentials, game over.
There is a solution out there that I wish got more traction. And that is, capability based security. The idea is simple: we partition programs into application code and library code. Library code can never access the filesystem or network (or other protected data) directly. And this is enforced by the compiler or language runtime. Application code can. If you want a library to interact with a file, you first open the file in application code and pass the file handle to the library. Similar wrappers can be made for directories, network sockets and so on. If I want express to listen on a specific port, I pass it that port in a capability object. Not just an integer. (Eg app.listen(net.listenPort(8000))). The amount of code that would need to be changed is tiny since well behaved libraries don’t usually need many capabilities to do their job.
It is ridiculous that my 3rd party json library can invisibly access all the sensitive files on my computer. With capabilities, we take that privilege away. I have no problem trusting a random json library if it literally only has access to the json string I pass it. If we had a capability based security model, I would have no qualms about pulling in hundreds of transitive dependencies. My libraries can still only access exactly what I pass them.
Unfortunately this problem requires language level buy-in. It would be very hard to retrofit rust to work this way and that’ll probably never happen. Raw pointers and inline assembly also make a bit of a mess of things too. (Though pointer provenance might help). But a man can dream… Maybe in rust’s successor.
However, I quite like the idea of using the application/library interface to enforce the restrictions structurally, rather than temporally.
That works for server applications. Well - so long as bad things don’t happen in a static initializer before the process drops privileges. But I also want it to work for other kinds of applications. I want to be able to fearlessly depend on a library and not need to worry about that library attacking my computer. Then we can have our cake and eat it too with big dependency trees made up of lots of small libraries written by untrusted contributors.
Another nice thing about this is it doesn’t depend on OS level capability support. Once implemented in a language, it should be pretty easy to make it work on every OS.
Interestingly, this is something wasm provides out of the box. And Firefox’s weird “compile deps with wasm as convert the result back to C” also achieves this end.
1. https://github.com/ossillate-inc/packj flags malicious/risky NPM/PyPI/RubyGems/Rust/Maven/PHP packages by carrying out static+dynamic+metadata analysis.
2. Sandboxing file system w/o superuser privileges: https://github.com/sandfs/sandfs.github.io
I’m not imagining something that specifies policies at all. Capability based security systems shouldn’t need them. Instead, the main application can create capabilities programmatically - for example by opening a file for reading. The file handle is a capability. Libraries you call don’t get access to the OS, or anything outside their module scope that hasn’t been passed in as an explicit parameter. A 3rd party library can access the file handle you pass it as a function argument, but it can’t ask the OS to open any other files on your computer. Or open a network socket.
OS APIs aren’t in a scope that any of the libraries you use have access to.
There are a few things capabilities need to be able to do. They need to be able to be tightened. (So for example, if you have a file with RW flags, you can make a copy of the file handle that’s read only). And they need to be unforgeable (file handles can’t be created from an integer file descriptor). We’d also need a handful more “file handle” like objects - and this is where granularity matters. Like we need an equivalent for directories that gives you access to everything recursively in that directory and nothing outside of it.
If the library I call delegates some work to another library, it’s pretty simple. It just passes its file handle through to the other library as a function argument. There’s no policy to maintain or update. Nothing even changes from the point of view of the original caller.
The tricky part is stuff like debug output, environment variables (NODE_ENV=production) or stdout. Do libraries implicitly get access to stdout or not? Maybe they get stderr but not stdout, and they can open new temp files? You would also need to restrict JavaScript code from arbitrarily messing with the global scope or global libraries like String.
But honestly, any answer to these problems would be a massive improvement over what we have now. Being able to know your binary search library can’t make network requests, and that your database can’t interact with files outside its database directory would do so much for supply chain security it’s not funny. A good capability system should just change the APIs that your software can use a bit. Generally in good ways that make dependencies more explicit. Done right, there shouldn’t be any new syntax to learn or deno style flags to pass at the command line or anything like that.
Apparently in firefox they've been compiling some of their dependencies using wasm, then translating the wasm back to C. The end effect of all that is that it gives C code wasm's sandboxing and safety. I wonder what it would look like to do that without needing wasm as a middle step.
And users install from there.
Not saying that's a nice answer, but that is the answer.
For certain kinds of software, this approach is getting more viable, thanks to containers. In some contexts, you can tell everyone to run your software in a container with a specific Linux distro inside. But that software is often distributed as container images directly, rather than distro packages to be used in containers.
Or, I just use cargo. Up to date version. One command install that works on every operating system. Project scoped dependencies - so there’s no need for sudo, and different projects can depend on different versions of their dependencies. And I can publish new versions of my packages with 1 command and they’re available everywhere.
Just in my house I have a Linux mint workstation, a windows gaming pc, a FreeBSD server and a Mac laptop. Cargo lets me publish code that all of them can consume with 1 command. So does npm.
So long as apt is worse than cargo I won’t use it for this stuff. Why does everything have to get installed system wide? Nix doesn’t need that. Why doesn’t it have a programmatic mirror of cargo so we don’t need to mirror rust projects in it by hand? Why do major versions have different package names? And why don’t packages support feature flags? Last time I looked there were about 30 vim packages with random combinations of feature flags set. Why can’t evergreen packages like nodejs or cargo stay up to date in Debian? Old nodejs is useless because it can’t run modern JavaScript.
I really want apt to solve all these problems because I hate docker. If apt and friends did their job well, docker wouldn’t be necessary or popular.
Cargo isn’t pretty either. But it has all the features I need to get my job done. Make apt a compelling tool and I’ll switch to it. The feature list developers want is hardly a mystery.
That is why each modern language has introduced their own package manager.
The web. The C programming language. Electron. It’s a tale as old as time.
Depends. Do you want your software to be used by users? For me as a user, software that is not in distribution repository almost does not exist. It is too much hassle to install, manage and uninstall it compared to installing software from Linux distribution. If i really, really need it, i will make exception, but that are individual cases.
I think the industry may have thrown the baby out with the bath water. Now they're is a complete aversion to RYO anything, no matter how simple. Personally, I'm waiting for the pendulum to swing back towards the middle a bit more.
But then there’s things like rust’s rand crate which implements about 10 different random number generators, all with different properties around speed, whether they’re cryptographically secure and so on. And fed from a variety of entropy sources to make it work everywhere from Linux to windows to wasm. Look at this documentation. It’s an absolute master class.
https://rust-random.github.io/book/guide-rngs.html
Do you need this? Probably not. You can probably get by with some wrapper around ar4random or whatever the recommended entropy source is today on your particular operating system. But I’ll be honest, until the day I decide to care about how rngs work, I’m going to just import the rand package. And serde & friends for json support. Criterion for benchmarking. And so on.
Cargo is full of great libraries that are much higher quality than anything I’d ever throw together in a side project. For most of this stuff, you’d be mad to roll your own.
This thread has reminded me that lots of people don’t know how nice it is to use a package manager like cargo. Here’s my favorite crate browsing website. You can see the most popular packages in every category and search. Look how much great stuff is in here. It’s humbling.
Github premium (or similar) will have an army of AI builder bots that spin up VMs and modify your codebase until it works, then send you PRs to approve. People would pay a ton for that service. All your projects, automatically maintained and packaged.
If I really need an old library, it's generally not hard to install that in a temporary prefix and set the relevant PATH variables for it, though I do wish this was easier sometimes. It gets better as more projects just use pkg-config.
That all said, Meson[1] solves all of these issues in a way that keeps both developers and distributions/users happy. But as is with all things C, getting everyone and especially windows users to adopt it is gonna take a good while.
(My >4000 comments range in score from −4 to +76 plus one outlier at +287 because people really hate stale bots on issue trackers. See https://news.ycombinator.com/item?id=32541391 for methodology. This comment has also clearly evoked a strong emotional reaction.)
All I can say is that I’ve had a lot of trouble over the years building older C/C++ software, often seeking to consume it from Python but sometimes just from C/C++. Maybe if I lived and breathed those languages I’d cope better, but at the very least they consistently take extra effort that requires at least mildly specialised knowledge, compared with my experiences of pure-Rust or even mostly-Rust software.
When I started using Rust it truly felt like magic. I could any app by anyone and essentially `cargo build` would always just work. So effortlessly.
My best comment by far is one on an article about linux support for laptops being great, talking about all the problems I've experienced in recent years, even though I do agree with the article's core statement.
If someone had written approximately the comment I wrote first, I’d have upvoted it, because it matches my experience.
That's because you're in the very small group of people that knows how your distro's packaging process works in detail. The majority of developers don't use Linux at all, and those that do typically know just enough to install a few blessed packages, not enough to hunt down and install the correct shim in a temporary directly.
Multiply that lack of training by the dozens of Linux distros being used by users of a language, and it's no wonder most modern languages have a standard language-specific package manager. The alternative is for them to be constantly fielding impossible-to-answer questions from developers who (unlike you) don't understand their distro well enough to make compilation work.
It's far easier for the language to have a single cross-platform package management system where the answer is usually as simple as "X, or Y if you're on Windows".
Something in rust is always just `cargo run` and you're done.
That's the theory. In practise, it just means small app developers putting up with:
1) #ifdefs for different versions of same libraries in different distros
2) distro specific bug reports for an operating system with less than 2% user base
3) binary releases that target just 'Linux' being out of question for most proprietary applications / apps that can't/won't be included in distro repositories for various reasons.
It's not as if distro maintainers will test your random little application every time they update ffmpeg anyway. It's the users, for whom the application breaks.
Interesting. My experience doesn't back this up at all. I may have issues with 20+ year old code, but I don't remember ever having anything but minor issues otherwise.
I wonder what the difference between us is?
If you believe that you won't run into a compiler bug in a large enough sampling of C++ code to build against you are misrepresenting reality or haven't ever looked at the gcc or clang bug trackers.
Rust actually does a really nice job along these lines by continuously building and running test for some subset of crates in crates.io
The only thing maybe even close to that with gcc/clang involved I've seen is nixos's hydra.
Note that this is not all Rust code that exists, but it does help a lot.
I never made anything like that claim. Nor was I making any comment about the utility of something like crates.io.
I was just relating my personal real-world experience working with code. Interesting that I got downvoted, as if what I've experienced in my career was somehow wrong. I certainly wouldn't claim that devs reporting the opposite experience from mine are wrong. It's just that the difference is stark, and I was curious as to why.
#!/usr/bin/env nix-shell
#! nix-shell -i fish --pure
#! nix-shell -I https://github.com/NixOS/nixpkgs/archive/4ecab3273592f27479a583fb6d975d4aba3486fe.tar.gz
#! nix-shell --packages fish gcc gnumake gmp git cacert
...build script goes here...
--pure: Environment is as pure as it can reasonably get.
--packages: Packages from nixpkgs to be made available in the environment.
-I <github_commit_hash>: pinned to a specific git commit of nixpkgs. This script will give you the exact same environment with the exact same packages with exact same dependencies forever.
You can drop into a shell with the exact same configuration by using the cli command nix-shell with the same options.
Admittedly this is not as declarative as using flakes, since it's a script, but hey, I'm lazy, still didn't sit down to learn them once and for all.
Reference: https://nixos.org/manual/nix/stable/command-ref/nix-shell
As researchers using nix derivations to preserve usability of code meant to be archived with our papers, we found that it was not particularly effective for this task. Even with nixpkgs pinning, our code failed to compile after only a few years. Outside of nixos itself, it does not appear that the build environment is the same; aspects of compilation change, and we ended up needing to change compiler flags a few years later, while expecting that we might need to again at some point.
Overall nix has been quite disappointing for us from an archival standpoint.
If the sandbox is disabled (as it is by default outside of nixos I believe) then yes, you need to be very careful that there is no outside state that leaks into your build (usually through environment variables, PATH but not only - GCC is a big offender here), which sounds like what happened in your case? For archival purposes you really should be enabling the sandbox.
Our approach was to make a script for nix-shell that would put the readers in an environment where they could run the code, compiling specific (old) versions of two research software packages. But I think the problem here is that in writing the derivations here, we didn't put gcc in the build inputs. So the system's compiler ended up being used, and with the old, finicky code, compiler flags needed to change. A frustration here is that it appears sandboxing can't be enabled for nix-shell, a problem that has been known for close to a decade [2]. But in our case, if we can get the build inputs to be reasonably complete on our own, it should improve the situation for our readers in the future.
[1]: https://github.com/DNA-and-Natural-Algorithms-Group/SST-sequ... [2]: https://github.com/NixOS/nix/issues/903
And it seems to be popular in science circles.
Was this using flakes or no?
Since you mention pinning, I'm guessing not.
An experience report, even if brief, would be very valuable to a Nix/NixOS community working seriously on reproducibility.
Archive of system repo + mock + SRPM does the same thing, but offline.
[0]: https://fedoraproject.org/wiki/Using_Mock_to_test_package_bu...
[1]: https://en.opensuse.org/openSUSE:Build_Service_Tutorial
What if you need different versions of the same package to support end applications?
If we add loading binary libs built for an older system, then there is no question. Who wants to deal with incompatible versions of the GLIBC?
As someone who has worked in a big corporate their entire career this statement is bizarre. This has been solved for Java for over a decade.
Builds don't access the internet directly. You host a proxy on-premise (Nexus/Artifactory/etc) and access that. If the internet location is down, you continue as normal.
> You need only one author in your maybe-hundreds-of-dependencies tree to be hacked, coerced or in a malicious mood for you to have a really bad day.
There are products (Nexus Firewall) that can check dependencies for vulnerabilities and either block them from entering the network or fail CI pipelines.
There's no technical solution to that, really. Do you think Nexus Firewall can pick up every exploit, or even most? How confident of that are you, and what data do you have to back that up? I don't have any myself, but would not be surprised at all if "hackers" can easily work around their scanning.
However, I don't have a better approach than using scanning tools like Nexus, or as the author proposes, use a curated library repository like Debian is doing (which hopefully gets enough eyeballs to remain secure) or the https://github.com/crev-dev/cargo-crev project (manually reviewed code) also mentioned. It's interesting that they mention C/C++ just rely on distros providing dynamic libs instead which means you don't even control your dependencies versions, some distro does (how reliable is the distro?)... I wonder if that could work for other languages or if it's just as painful as it looks in the C world.
Also, nix can do arbitrary building (arbitrary commands). You don’t need a specialized builder. You can write a specialized builder for your language too, if it doesn’t exist.
1. Git clone https://git.savannah.gnu.org/git/guix.git
2. Nicolò Balzarotti already has a work in progress patchset adding Dart itself--see https://issues.guix.gnu.org/44926 . Patch your guix checkout with the files you can download on that page via the downarrow buttons on the right.
3. Create a file guix/build-system/dart.scm (I'd copy it from guix/build-system/dub.scm and then s/dub/dart/g and s/ldc/dartcc/g) or whatever it is. This file is the "front end" and is support for dart for the guix package manager script.
4. Create a file guix/build/dart-build-system.scm (I'd copy it from guix/build/dub-build-system.scm and then s/dub/dart/g). This file is the "back end" and is responsible for the build of your package inside a new container.
5. Add references to your new files to gnu/local.mk
6. In your user package script (whatever you want to compile in dart), add #:use-module (guix build-system dart) near the top and then add (build-system dart-build-system) to your package.
7. guix environment --pure guix
8. ./bootstrap
9. ./configure --localstatedir=/var
10. make -j5
11. exit
12. ./pre-inst-env guix build -f yourpackage.scm or whatever
13. Fiddle with the "invoke" parts in guix/build/dart-build-system.scm until your user package works.
14. Submit your results to 44926@debbugs.gnu.org or guix-devel@gnu.org (no need to be subscribed).
As you can see, adding a new programming environment is quite some work. Because every programming environment has its own idiosyncracies, I don't think it can be avoided even in principle.
I mean the working copy (the local directory) created by git clone.
>Looks like it's a huge effort.
Probably.
I've never used Dart in my life, so I don't have so much Dart-specific information, sorry.
> That Pull Request is adding Dart 2.8 while the current version is 3.1.5 (and it's released often, at least once a month). Most pub packages don't even work on Dart 2.x anymore.
Then as a next step you (or someone) would probably add another dart package for version 3, building it with Dart version 2 (is that how the dart build works? Probably).
>as I am using Guix OS (in a VM) and as far as I know there's no sources in it?
Just do the git clone. The guix package manager is a regular program (with source on git) like any other.
I always use a git clone of the guix master branch (for like 6 years now) and it was never a problem. But if you instead want to use the version used by your OS, do guix build -S guix . It will unpack the source of the current guix and tell you where it is. I wouldn't bother doing the latter though.
Also, basically there's a guix daemon that does the build container management (think of it as something like dockerd) and then there's the guix frontend. I'd never do "make install" in the guix working copy--so the guix daemon will not be fucked with. The ./pre-inst-env allows you to run the local directory guix frontend without installing it.
This is NOT standard practice in the JS/Python/PHP/whatever world because these products do not exist or are not widely available, everyone "drinks" directly from the open internet source.
Yes, again, solutions exist, technically, but how many companies do you know, especially smaller ones, that do this?
For Java almost every company bigger than 5 devs tends to set up their own Maven repo at some point, many do it on day 1 because at least 1 dev is familiar with this practice from their former BigCo and it's easy to set up Nexus/Artifactory.
The average Java dev will try to reach for Artifactory/Nexus much sooner than the average Python/Ruby/PHP/Javascript/... dev, in my experience.
Devs might do whatever they feel locally, however if those files are going through CI/CD their dependencies better be in Nexus.
There is. Theoretically speaking, you could confine libraries to their own sandbox.
But Rust can't do that.
There is at least one language that tries to do that (someone posted it here in a comment some time ago, but I can't find it).
Unfortunately it is deprecated and scheduled for removal because too much security is annoying and not enough people care to use it.
EDIT: Has a proxy for packages.
Mirroring all of crates.io takes about a terabyte of storage, not much bandwidth and is really easy to set up, but judging by the download numbers of obscure packages there are only about 20 places in the world doing that. Maybe people haven't bothered because crates.io is so stable, maybe everyone goes the vendoring route instead.
Of course if your development is already set up in a way that prevents you from easily adding, experimenting with or updating dependencies, maybe because of how the vetting process works, that might not be an issue for you. And many places are completely fine with just being able to build
It's unlikely that I'll need a new dependency at the exact moment that the source is down, but fairly likely that if the source is down then I'll want to build something. For personal projects I'll have stuff cached locally, and I don't mind so much if I need to wait for CI. For work stuff I need to be able to fetch my dependencies in CI.
Also, hammering upstream with requests for every single build is both slow and also sort of rude.
I had no idea \o/ Thanks!
Products can only do best effort scanning for known patterns of vulnerabilites. Those are only added after someone (or something) discovers it and verifies that it's not a false alarm. In between the time gap, scanners are ineffective and anything in your infrastructure can run the malicious code and get hacked. (Google supply chain incidents on npm or pypi.)
In general, there is always a way to bypass it, since the language is turing complete and unsandboxed. Anything going beyond that is a lie, or simply impractical. Systems with latest software and antivirus get hacked all the time. Nothing can really stop them. Why?
- https://web.archive.org/web/20160424133617/http://www.pcworl...
- https://security.stackexchange.com/questions/201992/has-it-b...
- https://en.wikipedia.org/wiki/Rice%27s_theorem
> and either block them from entering the network or fail CI pipelines.
Also, relying solely on an endpoint security product for protection is dangerous since antiviruses themselves get hacked all the time. Sonatype for an example:
- https://nvd.nist.gov/vuln/search/results?form_type=Basic&res...
My approach is basically: train a model on a large obfuscated dataset seeded with "known" examples. While you can't say with certainty what an unknown sample contains, you can determine how similar it is to a known sample, so you can spend more of your time analyzing the really weird stuff.
The hardest part in my opinion is generating the training data. You need a good source code obfuscator for your language. I've seen a lot of papers that use obfuscator-llvm[1] to obfuscate the IR during compilation. I use Tigress[2] to obfuscate the source code because it provides more diversity, but it only supports C.
[1]: https://github.com/obfuscator-llvm/obfuscator/wiki/Installat...
[2]: https://tigress.wtf/
One question: how do you detect when a binary is intentionally made to statically look similar to one binary, while its behavior actually mimics another?
In my work I'm not looking for intentional spoofing, but the obfuscations I do use [3,4,5,6,7] end up building very similar control flow structures for different functions. Maybe that fits the spirit of your question... Let me know if not.
So far I'm doing purely static analysis and control flow, but the broader field of reverse engineering includes dynamic/symbolic analysis where you track values through a running/simulated program. Great results but very costly to run.
I've been focusing on making cheap/static analysis better, so I haven't explored the dynamic/symbolic side at all yet.
[1]: https://tigress.wtf/virtualize.html
[2]: https://tigress.wtf/jitter.html
[3]: https://tigress.wtf/flatten.html
[4]: https://tigress.wtf/split.html
[5]: https://tigress.wtf/merge.html
Once they're known. I suppose nexus firewall let log4j pass.
1. https://github.com/ossillate-inc/packj 2. https://github.com/ossillate-inc/packj-github-action
Isn't that how life works? Unknown unknowns? :-)
> I suppose nexus firewall let log4j pass.
It should be reasonable to assume that a product dedicated to something will, on average, do this job better/faster than a random developer, though. Since that's their job.
So Nexus Firewall presumably blocks these vulnerabilities faster than the average app/system developer notices them. There's value in reducing the delta between the vulnerability exploit day and an entire ecosystem being patched/fixed.
Once log4shell was discovered it was trivial to eliminate it from the organisation. We knew exactly what projects were using it and updated them accordingly. Access logs showed us that it was no longer used and we delete it.
> Builds don't access the internet directly. You host a proxy on-premise (Nexus/Artifactory/etc) and access that. If the internet location is down, you continue as normal.
Right now I'm using Sonatype Nexus (the free version) for my personal needs: https://www.sonatype.com/products/sonatype-nexus-repository
And I do mean most things: apt packages, Docker images, Helm charts, as well as packages for Go, Java, .NET, Node, Python and Ruby, maybe even more in the future. I'm very glad that it is possible to do something like that, but honestly it's hard to do and I can easily see why most wouldn't bother: even though the configuration UI is okay and you can create users with appropriate permissions, you'll still need to have plenty of storage, manage updates and backups, setup automated cleanup tasks (especially for Docker images), in addition to the whole process on the developer's side sometimes being a nuisance.
With npm you get --registry which is one of the simpler approaches, Ruby Gems needed some CLI commands along the lines of "bundle config mirror" and other stuff, Maven needs an XML file with the configuration, in addition to publishing your own packages to those also needing more configuration. Sadly, it seems like this fragmentation of approaches is unavoidable and complexity increases with the amount of languages you use.
On the bright side, I keep control over my data, as well as have really good download/upload speeds, even on a cheap VPS. Plus you learn some vaguely useful things the more you explore if you want to go further, like how I also start with Debian/Ubuntu base container images and provision whatever tools or runtimes I need for more specialized images (e.g. my own JDK, .NET, Node etc. images).
There are actually multiple different kinds of blob store backends: https://help.sonatype.com/repomanager3/nexus-repository-admi... though so far I've only used the File one, because my servers have incremental backups that can be restored in a few clicks (that I've sometimes done in the past).
I agree with the pain that some languages/package managers add as well. Its a shame package managers aren't necessarily designed to easily support writing your own mirror or switching repository.
Fair play for running your own Nexus for personal use!
Does anyone have a proxy for Rust? I'd love to run it. Please. Someone... help
edit: just learned Panamax exists for this... will try it
A Rust program `p` can depend on `q` and `v` which depend on `u`, but `q` requires `u` @ 1.0.0 and `v` requires `u` @ 2.0.0. In the Debian world, you would have to change `p` and `v` to support a mutually compatible version of `u`. If those changes introduce bugs, they're Debian's bugs, but the author of `p` is the one who gets the reports.
Cargo naturally handles this case, it installs multiple incompatible versions at unique locations in its cache, and correctly invokes the system linker to link the compiled objects correctly. If `p` was a C program, we could not handle this case, because `u` would be a shared library and you cannot link two different versions of the same shared object into a single address space at runtime (if they were static libraries, a sufficiently smart build system could deal with it, just like cargo - but there is no unified build system and package manager for C, so good luck).
Further, Cargo creates lock files for your projects, and authors commit and push them for executables. If you build against a lockfile. This prevents most of the cases that the author is bemoaning. People don't run Cargo update willy nilly.
Frankly, the "problems" the author is claiming don't exist in practice and the "solution" makes life worse as a Rust developer.
I'm getting tired of hearing people praise broken package managers like Debian. They make life worse for everyone for no benefit. How many times do people need to read testimony that "C/C++ package management sucked and Rust is great just for Cargo alone!" before they get it?
"People don't run Cargo update willy nilly."
We don't know the same people then. Most devs like to keep their stuff up to date. Auditing the dependencies tree before updating a lock file is like reading a ToS. Most people don't even if they should.The problem of securing the chain of production is not solved with a simple lock file.
That's not quite correct. Even if a library version is yanked (which hides it in crates.io and makes it so cargo will skip that version during resolution), if it appears in a lock file, cargo will dutifully find it and use it. This is explicitly to avoid the leftpad behavior of a library disappearing breaking your build.
> The problem of securing the chain of production is not solved with a simple lock file.
I didn't say it was. I said the issues the author describes almost all are. Relying on distro maintainers doesn't solve them either.
As far as I can tell, this is more about the feeling of security because of familiarity. I can completely understand why they would feel more comfortable using something they've successfully used for many years, but that feeling doesn't actually translate to improvements in security. There is a feeble attempt to suggest that slowing down the rate of change improves security, but I would prefer to have the latest versions of the language toolchain and libraries available immediately, so I can make a decision on what to use myself.
If security actually needs to be addressed, it's going to be by paying maintainers and paying people to review dependencies. And that would likely be in the framework of PyPI/crates.io/npm, because none of them place constraints on where the code can be used, unlike using a single Linux distro's package manager.
They do not make this assumption, and in fact state the opposite.
> They might decide to give the diff at least a cursory look, which is better than nothing.
So yes, they do value the cursory review, which isn’t that useful. And they talk up the security benefit of only pulling in a subset of published updates, which is even more dubious.
Contrast this with the approach that Google is taking here - https://opensource.googleblog.com/2023/05/open-sourcing-our-.... Actual, careful review of the code that anyone can take advantage of. And no limitation of having to choose one flavour of one OS. This review helps you audit your supply chain regardless of OS you’re developing and deploying on.
> A simple time delay will allow egregious malware like malicious build.rs scripts to be caught, whether that’s the super-long Debian stable cycle or even the several days required to migrate from unstable to testing.
Normal cargo + crates.io usage has the same property via the existence of lockfiles. For most people that care about stability, this will provide a natural delay (which you can also see in crates.io download graphs) to consuming the latest version.
-----
The other points brought up in the article are definitely valid, but if that's the main strategy of defense you are looking for, you might already have it.
mostly, there can be a small attack gap when adding new dependencies to the project and at the specific point in time when you run `update`
through you can also pin versions in `Cargo.toml` and then review any updates, maybe except for a few highly trusted sources (it's a bit annoying and costly (time wise) but viable)
Through trying to vendor things, especially with Debian, seems like a horrible solution. And there is a lot of precedence for this causing tons of headaches, wrong bug reports and similar for developer (== time loss == time is money so we could probably be speaking about multiple millions of monetary damages).
Through I have been thinking for a while that it could be interesting to have a partial crates.io "proxy" which only contains and updates crates which have gotten a "basic screening" in all of their code and dependencies probably with some tool assistance this might not find bugs but it should find many forms of supply chain attacks. Through given that this is costly to provide it probably would not be a free service.
Most package managers will keep the versions of transitive dependencies as unchanged as possible when adding a new direct dependency.
Of course if the only solution to satisfy the dependencies of the new direct dependency is to upgrade a transitive dependency, that will be done.
(I've seen a lot of people treat dependency additions as completely unpredictable operations that regenerate the whole lockfile in the past, which is why I wanted to clear this up.)
Be aware however that lock files are ignored by default. You need to specify "--locked" if you want Cargo to use the "locked" versions from Cargo.lock.
> By default, the Cargo.lock file that is included with the package will be ignored. This means that Cargo will recompute which versions of dependencies to use, possibly using newer versions that have been released since the package was published.
https://doc.rust-lang.org/cargo/commands/cargo-install.html#...
If you work in a repository on your own project and use basically any other cargo command (e.g. `cargo build`, `cargo run`), the lock file is of course taken into account (otherwise it would be near-useless).
Cargo does respect the package lock file when building a project from source (i.e. when you checkout a project, and run `cargo build` in that project directory). This happens, for example, when running builds in CI, or preparing a new release for each individual package manager.
And I completely disagree that that someone else than the authors should distribute or manage the package. They know the best how and what there is to do in certain situation (security fixes and so on). Of course it is good to have independent reviews, but these independent people should not be in charge of updating, maintaining and updating these packages/libraries.
The risk when it is too easy is that you suddenly pull half of the Internet into your project. In C++, the cost of having a dependency makes me choose carefully if I want a dependency or not, and if I do, if I want this particular dependency (Who develops it? Is it maintained? If it stops being maintained, can I easily move to an alternative, or could I take over maintenance? ...).
> About the best option we found was to treat everything as build from source and then git submodule our dependencies in.
If you mean having a git submodule and doing something like `add_subdirectory(dependencyA)` in CMake, please don't do that. Building from source is fine, but then you can totally install the dependency locally and have your build system (be it CMake, Meson, ...) fetch it. At least I started doing it the way it is described here [1] and it works well for me.
You can totally say "here is my code, it's open source, and the build system will look for those dependencies on the system" and let the users (who are developers) handle the dependencies the way they want. This way the user gets to choose if they trust the dependencies or not. Ideally the distro ships with those dependencies, and the user can just install them (better: write a package that depends on those dependencies and install that).
It seems like developers don't really know how to do that anymore, and language package managers allow them to not learn about it. But at the cost of control: they don't know anymore what dependencies they are pulling, or even how many they are pulling.
The modern way seems to be mostly about productivity, and therefore devs don't want to learn anything about dependencies, they want everything to "just work". But that is arguably bad for security.
> Ideally the distro ships with those dependencies
This is all good and well in the world where there is one single Linux distro, but usually you want to target all mainstream distros and macOS and Windows and what now. Depending on system packages becomes a brittle solution in these cases, since who knows which version of the necessary lib is packaged on this ancient Debian installation. If you depend on a newer version, then you are just forcing you users-developers to either spend time packaging it properly or just running `make install` and littering the system with untracked files. Honestly, stuff like `cargo` fixes this elegantly so I never have to think about it again.
I am convinced that you are more the exception than the rule. For node projects that pull hundreds packages transitively, I can't believe for one second that the devs even read the list (and even less that they would start considering reviewing the code).
> but there is not much difference between a dependency from the distro repository and the dependency from crates.io
Who reviews what goes into crates.io? I know for a fact that other package repositories don't have any check at all: anyone can push anything. Whereas the distro repository is reviewed by the distro maintainers. Big distros have a security team even.
I think that is a noteworthy difference.
> but usually you want to target all mainstream distros and macOS and Windows and what now.
Of course, usually you want everything, for free, and for yesterday. But let's be realistic: the vast majority of projects don't have users on all the mainstream distros, macOS and Windows. I would start by maintaining a package for my preferred distro, and maybe one for Ubuntu. But maintaining a package should not mean "building a .deb": ideally you should use the program on that system so that you actually test it.
If someone wants it in another distro, they can write and maintain a package for that distro. And again ideally they use it themselves.
I believe that distro maintainers are responsible for distributing software for their distros. And for that they can rely on contributions and community repos, of course.
But projects that target 50 platforms and offer 50 binaries to download even though nobody has ever even installed 48 of them are missing the point, IMO. It is great if cargo allows you to say "builds for 50 different OSes", but if nobody ever tested them, to me it's just marketing.
I don't think it globally matters and it certainly does not scale. Are you saying that Debian devs review all code in their repos? I doubt that, and they're certainly let log4j into their repos.
We already have pretty good automated tooling for detecting known vulnerabilities in deps, and updating and rebuilding is not hard. I don't see the added value of Debian's managing build-time deps of my apps.
It should be the devs' responsibility to audit code they use in their projects (including the toolchain), as well as it should be their responsibility to package, maintain and support the final product. I wouldn't want to support copies of my projects which were somehow modified by the 3rd-party maintainers before being provided to end users.
> the vast majority of projects don't have users on all the mainstream distros, macOS and Windows
I don't need to go far, if we're talking for example about some internal corporate tools, then the vast majority of projects have the majority of their users on macOS and Windows. In rare cases some deb-based usage is supported on best effort, but any other distro -- good luck!
The situation is not much different with some public projects, almost all devs don't go further than "supply a .deb and be done", and it's lucky if they support even that.
> It is great if cargo allows you to say "builds for 50 different OSes", but if nobody ever tested them, to me it's just marketing.
Cargo is not an end-user package management tool, it's a build tool for devs, which is then used for building deb, yum, whatever packages you need. It tracks mostly just build-time deps, and for everything else such as glibc, sqlite, any other .so I have no choice other than apt on Debian. This is fine with me.
Well they certainly do patch a fair amount of vulnerabilities faster than I would. Everytime I checked because I was concerned about a vulnerability, it had already been patched.
> I don't see the added value of Debian's managing build-time deps of my apps.
I can trust the Debian security team to do a better job than I would. I definitely do not trust arbitrary developers about security. If there is one thing I have learned from the software industry, it is that almost nobody cares about security. Turns out that the people in distro security teams generally do care about security.
> I wouldn't want to support copies of my projects which were somehow modified by the 3rd-party maintainers before being provided to end users.
Well maintainers can and sometimes do that. But of course if you don't want them to distribute your project, that's your right. Maintainers usually don't run after devs to work for them for free :-).
Also note that some distributions are not the typical general-purpose Ubuntu. Maybe an embedded IoT distribution wants to use your library, and maybe they want to harden it, or something. If you have an open source license, it is totally their right to do whatever they want.
Is my computer more secure for it? Am I in more control? Absolutely not. I have weird 3rd party programs installed in weird places all over my system. I have a computer that is almost impossible to rebuild if my hard drive dies. Finally I can make fragile builds that might have bugs that nobody else can reproduce because of the idiosyncratic combination of package versions I’m using.
This is so unbelievably worse than cargo or npm. With cargo I can take any package on GitHub and reasonably install and run it on any computer I own (running any OS) with one command. The only time this breaks down is when part of the dependency tree is C code. It’s miraculous.
Give me that experience in apt and I’m there. But the experience apt currently provides is unbelievably bad for developers compared to cargo/npm/etc. I can only assume nobody who works on Debian has seriously tried npm or cargo.
Why not start by building the version of that dependency supported by the project you want to build? It's not like you ask cargo to take the latest versions of all the dependencies, is it?
> This is so unbelievably worse than cargo or npm.
I agree that in many cases it is less convenient. But in your example, 6 out of the 8 dependencies come from distro packages, which I think is better. For the last two, you could actually contribute community packages.
Have you ever tried Arch Linux, for instance? There is the Arch AUR where the community can contribute packages. I don't remember having had to compile a dependency from source, because everything I need is either in the official repo, or on the AUR. It is explicit when I need to get it from the AUR, so I know that I may want to pay more attention to what I am pulling. And usually I do and have a quick look at the package recipe (and its dependencies). In this case I would have to check 2 of them, and not 8, which is a big win.
That is of course less convenient, but I believe it is more secure. I see it as a tradeoff.
Because I don’t know which version that is half the time. Does the readme say to use version 2.2 or 2.0.1? Are they meaningfully different? Maybe 2.2 is needed to be compatible with something else on my system. (Nvidia has entered the chat). I have no idea what is in the changelog of some random dependency that I’ve never heard of before - and I just remembered, I don’t care. I never care.
> For the last two, you could actually contribute community packages. Have you ever tried Arch Linux, for instance?
That sounds like an even more effective way to waste an entire evening. Maybe multiple evenings! When I see a cool program someone’s written that I want to play around with, what better use of my time could there possibly be than figuring out how to submit community made packages to arch Linux for the random dependencies of some random software someone linked me? All this before I’ve even built the program I want to try out? No thanks.
And how is it more secure? Do programs magically get security audits as part of their addition to arch Linux community packages?
Your comment reminds me of that infamous response in Dropbox’s “Show HN” thread. “Well, the problem Dropbox solves sounds like something easily done with rsync which is already part of arch Linux. Have you tried arch Linux? Instead of starting a billion dollar company, you could submit a simple bash script as a community contribution to arch. Using a bash script in arch Linux is of course less convenient. But I believe it is a trade off.”
And to answer the unspoken question, no. Arch Linux doesn’t have the tens of thousands of up to date packages that are already in cargo. And are already available on every operating system under the sun. Manually adding them to arch sounds like a pointless task that would only serve to make updating my dependencies more difficult and make my software less compatible on all the other systems it already, effortlessly works on. (Like Debian, FreeBSD, macOS and windows.)
If the library is done right, then 2.2.0 should work if it requires 2.0.1, and the reverse may not work (if the program you want uses a feature that was added after 2.0.1).
> and I just remembered, I don’t care. I never care.
Yeah, I think it is a big reason why language package managers are so popular: most don't care.
> When I see a cool program someone’s written that I want to play around with, what better use of my time could there possibly be than figuring out how to submit community
First, probably those packages were already contributed by someone else. Someone who cared.
Then... for the rare packages that may not already be there, I would hope that you could consider spending a couple hours contributing something back to the community you are most likely using for free and complaining about. For most libraries it should not even take two hours, except maybe the first time ever you do it.
> And how is it more secure? Do programs magically get security audits as part of their addition to arch Linux community packages?
As I said above, the ones that are shipped by the Arch official repo seem more secure. For the community ones, it's less clear, but I would argue that AUR users are at least not less likely to review a package than the cargo users are to review a transitive dependency.
> Using a bash script in arch Linux is of course less convenient. But I believe it is a trade off.”
Well if you want to use this as an argument, you should say why it is a tradeoff. Why is it a tradeoff? Are you saying that rsync is more secure than dropbox, but less convenient?
Is me volunteering my time and expertise to write and publish opensource code not virtuous enough? “If you really loved opensource, you’d also do this other task constantly that doesn’t even take 2 hours each time”.
Why does arch even need rust packages to be added by hand? I’ve already programmatically expressed the contents and dependencies of my rust package. And so have all my dependencies. Why not just mirror that programmatically if you want my dependency tree in arch?
> the ones that are shipped by the Arch official repo seem more secure. For the community ones, it's less clear, but I would argue that AUR users are at least not less likely to review a package than the cargo users are to review a transitive dependency.
It seems more secure? You’re making security assessments based on vibes?
I can’t speak for others, but I’m personally much more likely to review my dependencies in rust or npm than in Debian or whatever because the source code is available and linked from the cargo package page. And I can control+click in to my dependencies (or debug into them) and read their code. And with npm they all live in node_modules. Source and all. I dive in there all the time. Does arch do that too? I have no idea where to look for the source code of a package I installed using apt. I’d probably need to Google the package and hope apt hasn’t patched it in any significant ways that make the version on GitHub out of date.
> It seems more secure? You’re making security assessments based on vibes?
I shared an opinion, I am not publishing a security paper. You also say a ton of apparently uninformed stuff that expresses your "feeling". Like "with Arch I would constantly have to contribute packages myself" (Did you try it? Do you have statistics and proofs?). You are entitled to disagree with my opinion, just as I am entitled to have one.
I think that we won't agree here, and for the record I was not saying that language package managers were fundamentally bad. I was merely noting that I see pros and cons on both approaches. I tend to defend the "distro maintainers" side more often, because in my experience, most developers don't know about it.
> I have no idea where to look for the source code of a package I installed using apt. I’d probably need to Google the package and hope apt hasn’t patched it in any significant ways that make the version on GitHub out of date.
Exactly my point. You don't like the distro way because you don't know anything about it. Not saying it is not a valid point: you are entitled to your opinion.
My opinion is that there are pros with distro package managers, like the fact that maintainers put together a set of packages that they ship as a distribution (that is the whole point of a distribution).
Re: security, specifics matter. AUR “feeling” more secure than cargo doesn’t mean it is. If the principle reason to use it is security, tell that story. How is it better?
> You don't like the distro way because you don't know anything about it.
I’ve been using Linux since a version of Slackware I installed off floppy disks. I remember trying apt for the first time and thinking it was an absolute revolution. The best thing since sliced bread. I haven’t tried arch but for what it’s worth, gentoo gets the source thing right. You can see the source of anything on your system with 1 command. And better yet, it all builds. There’s no stupid -dev version of packages like there is in apt (that drives me nuts). I’ve still never submitted a package to any package manager - and maybe you’re right. Maybe I should.
I’m upset by all of this because I want apt and arch and all the rest to be better. But as far as I can tell, apt hasn’t improved in decades. And now docker (which I hate even more) has come along and solved a bunch of apt’s problems for it by making something even uglier on top. What I want is a package manager with these features:
- Reproducible, hermetic environments (nix)
- Local dependency installation (nix, docker, cargo, npm)
- Cross-platform packages (docker, cargo, npm. Nix is trying but not there yet)
- Cross language packages (nix, apt, docker)
- Feature flags (cargo)
- Support for the same package to be depended on at multiple different versions in the dependency tree (cargo, npm)
- Semver API compatibility checks (nobody does this, but Cargo is working on it.)
- Simple, cross platform publishing. None of this “here’s how you install it on a few Linux flavours we got working manually”. I don’t want to have to play a guessing game of which distributions package which dependencies and what they call them. (Cargo and docker do a good job of this)
- Support for binaries and libraries (apt, cargo, npm, nix. Basically everything but docker.)
- I can just take a random project on github and run it reliably without spending hours pissfarting around first. (Cargo, npm. Apt gets negative points because software packaged with apt breaks so often on newer Ubuntu / Debian distributions.)
I can’t speak for arch but apt doesn’t cut the mustard any more. I wish it did. But it doesn’t. And no amount of “just work around it by manually submitting more packages to this one distribution specific package manager” will satisfy me. I change computer and Linux distribution all the time. I want something that works reliably everywhere. The other day I ran some rust code I wrote on FreeBSD. There are about 100 transitive dependencies, and I bet almost none of them test on FreeBSD. But it doesn’t matter. Once I had rust installed, I checked out the project and ran cargo test. Everything built and worked perfectly. That is how good life is with cargo. Or docker or npm. Or even Python once you get your head around conda or venv.
It’s 2023 and that’s table stakes for a package manager. Distribution specific package managers need to keep up because they don’t pass muster any more.
I specifically mentioned the official ones, not the AUR. The story is that the maintainers of a distro check (on a best-effort basis) the packages they ship, and the distros that have a security team patch them when CVEs are announced. Not sure how that would not seem more secure than e.g. pip, where I don't think there is any kind of check (before or after the package has been published).
> Or even Python once you get your head around conda or venv.
For what it's worth, I have this third-party project that used to work and now always fails to install because of some dependency issues in pip. For me it's been broken for years now, I always need to go fiddle with it, `pip install <package>` just doesn't work (it installs something that then complains about a dependency missing, and I can't install the dependency because of some incompatibility reason). I have never had an issue with Arch or Alpine.
I am not a huge fan of writing apt recipes, I like the pacman/apk ones a lot more.
> It’s 2023 and that’s table stakes for a package manager. Distribution specific package managers need to keep up because they don’t pass muster any more.
I do agree with you that there are many issues and I would like it to work better. But one thing I would like to see is devs learning how the tools work and trying to "do things properly". Too many people throw their project in a docker container just because they have no clue how to use their build system correctly, or how to install a dependency properly.
I see a lot of "I use this because it just works and I don't need to understand how", and it feels like this brought us stuff like docker-for-absolutely-everything and ElectronJS. Nix seems nice, but if people can't even be bothered to learn how to use SemVer properly, I don't see how they would ever get the motivation to even consider Nix.
When you install from AUR you are compiling from source though. Unless you're using chaotic aur, which only packages a subset of what's available.
There is no such risk, as you're not playing a roulette. No one is pulling a gun to your head and forcing you to pull in hundreds of dependencies.
You can do exactly the same in Rust as you do in C++ and be conservative in what you use for dependencies. This is what I do, and it works great.
Now, that said, I agree that for the ecosystem as a whole this is a problem and people do tend to be too trigger happy when pulling in dependencies.
Even if it were solved on Linux (and it isn't, at least not for "our" dependencies), this doesn't help MacOS and Windows users. And no, Homebrew and Chocolatey (or whatever) don't háve all needed packages either.
Crates.io does have problems, the main one being non-existent namespaces.
For me the alternative to Crates.io/npm/PyPi is a "platform release" like Android where a bunch of stuff is signed off by a corporation.
Whereas the developer of a popular Rust library most likely just added some dependencies that were convenient, and doesn't know the full story of the transitive dependencies.
To me it is much more likely that somebody at Debian _saw_ the dependency being shipped than an arbitrary Rust developer, who probably does not even know how many dependencies are being shipped with their program.
DDs review the libraries they package. All uploads are personally signed to keep people accountable.
Instead opening a github account and developing a library can be done anonymously and there's been supply chain attacks done this way.
All this is to say: a large amount of debian users likely are using rust, and an even larger amount of non-debian users is likely to be using rust.
those few narrow components likely have good support.
True that, but that is only because we got used to not having to download dependencies manually (like with C). The option of "download all of your dependencies by hand and slap them in a directory" is always there, only instead of invoking ./configure with --with-libxyz=PATH or setting XYZ_PATH you modify a Cargo.toml.
> this problem is largely solved for C and C++: Linux distributions like Debian package such a wide range of libraries that for many things that you want to develop or install, you don’t need any third-party libraries at all.
That's rich. If that was the case, there would be no Docker. What about Windows? On Windows installing C/C++ libraries is a massive PITA. What about if your distro packages the wrong library version, or it's been built with a set of flags you can't use? The whole point is moot IMHO. A small part of why we ship libraries through package managers is because disk space was scarce years ago and it's arguably "safer" to build shared objects (at least for C libraries, C++ has templates so shared objects is a recipe for disaster and requires rebuilding everything anyway). The critical reason why we have a `libxyz` package in the average distro is because _C/C++ build system is a massive mess_, and that's the only sane way to distribute and allow people to build packages without wasting several hours to find and gather all dependencies.
This approach has considerable downsides, so that's why we now have Flatpak and the like.
This way you would only need one service for all your needs. Everyone could host their own proxy/cache/mirror/repository. BOMs would be simpler. CVEs easy to attach and query. Tools could handle multiple different package types. Auth could be shared and simplified as well. Less reliance on a single provider. Projects with multiple languages would be easier. New tools that need packages would not need to invent everything from scratch.
This of course only slightly helps with the trust problem, but at least offers a common way to attach things like CVEs or if someone has reviewed a package/version and share some effort between different platforms. In the end you still need people to put in the effort to actually review the packages.
RPM? You'll never get everyone actually on a single format, but even then between LSB and just its actual popularity it has support on most distros (even if only by way of converters like alien)
Agreeing on one format is kind of the point. I know it is difficult, but I think it is possible if at least a few implementations agree on something. Maybe some kind of "backend" like LLVM which could be used by cargo, npm, maven, etc. to configure their resolution strategy. Probably only a dream though.
Switching to VMs and Docker containers helped but really this was a band-aid to deal with the fact that you couldn't depend on an OS package manager to handle this well.
Packaging up Rust libraries with the OS package manager is just going to have the same problems.
dynamic linked libraries are a way to save tiny amounts of RAM and disk space while wasting millions of hours of people's time.
If you can’t wait for recovery, I’m pretty sure cargo can pull source from any accessible git repo.
> There is no mediation of any kind between when a new library/version is published and when it is consumed. You need only one author in your maybe-hundreds-of-dependencies tree to be hacked, coerced or in a malicious mood for you to have a really bad day.
How is this prevented on Linux distros? Surely not every piece of code is being audited to prevent supply chain attacks?
But what makes this design especially nice, in my opinion, is that it means that you can host Go modules anywhere, so as long as you follow some conventions. And because of the GOPROXY mechanism, even if your site is a little unreliable, it shouldn't generally be much of an issue. The sumdb and lockfiles additionally helps ensure that the integrity of existing published versions of packages can't be compromised even if e.g. somebody acquires the domain later on.
I'm not saying this is perfect, but it's a really nice blend of properties. If you're going to have to have a centralized entity that provides package services, making it work like this prevents a hard-dependency on a centralized party to handle package management while reducing some of the security and reliability concerns of self-hosting package repositories. Admittedly, this would lose some of its luster if a lot of packages became unavailable because the GOPROXY stopped caching them and they were long removed from the Internet; but OTOH, things eventually disappearing is always going to be a concern on the Internet, as the act of keeping things online is an active effort that requires time and money. As of today, I think this remains a sensible compromise.
This doesn't solve curation at all, but that's kind of the thing; if programming language package management is the wrong place to gate insecure or potentially even malicious packages packages, then they may as well not colocate packages to specific central locations, either.
https://doc.rust-lang.org/nightly/cargo/reference/unstable.h...
`--locked` might not prevent index updates, but prevent any changes in used code, i.e. only what is in the lockfile will be used
`--frozen` like locked but also requires the (download?) cache to be up to date
`--offline` prevent index updates and downloads
So if you have everything in the download cache `--offline` is like `-Z no-index-update` and `--locked` is most times good enough because since partial indices where added the index updates don't consume much time at all.
Would be even better if the dependency tree for a basic package was more modest, about 157 times smaller.
On the other hand, compiling Rust projects is easy. Works great. There are very good reasons for language toolchains to remain independent at all levels from OS toolchains.
Try telling this to GitHub customers. Half the world was unable to work in the last GitHub outage. Not to mention NPM.
All up, this makes it genuinely practicable. I’m still not convinced it’s worthwhile (and it does make life a bit harder in other ways, especially in ways that I’m sure will become more apparent over time as the system gets used increasingly), but it’s not as unreasonable as it may initially sound.
Really, what’s being presented is Rust without crates.io (the domain name and content host), but with crates.io (the packages).
It's really two worlds, and it seems that most people belonging to one don't fully understand the problems faced in the other, even though they think they do.
A problem for such repositories like the Debian one is that it can only contain vetted versions of libraries. Many Rust libraries are still 0.x and have changing API and frequent updates, which makes it hard to vet all dependencies, especially when an update implies updates of the dependents.
So, there are solutions already.
> There is no mediation of any kind between when a new library/version is published and when it is consumed.
No one is forcing you to stay automatically up-to-date. I personally on changes in critical libraries before updating.
> Any tampering with crates.io itself (espionage, disgruntlement, national security) could have an incredibly wide blast radius, or a incredibly wide set of targets from which to choose.
Repeat of point 1.
> Since crates.io is the source for crates, it is normal for both developers and CI machines to be hitting this web service all the time. Opportunities for mischief are exacerbated when clients are phoning home so frequently.
Again, point 1.
> So what’s the alternative?
Given there's only 2 problematic points in this list and you have pointed to solutions already, I don't see any need for an alternative.
How is this not a third-party library, and how is this not exactly the same problem as with Crates.io?
Especially when it comes to flat downloads with no underlying package manager like Yocto/Bitbake and some are just terrible to setup like NugGets. The fact is, most languages or frameworks were just not designed with private repositories in mind. This is made worse when you have to make these private repositories integrate with non-developer environments like a CI worker node or a BitBake recipe.
BitBake has a flat cache for downloads and binary builds and you can setup remote caches online for example and it works great. There is a dire need for some kind of universal caching protocol. Downloading files addressed by content through a proxy should be trivial and universal.
Most people use one of a handful of Git hosts anyway (GitHub, GitLab, ...) so Git dependencies are still like a huge centralized repo.
If git was the only option, most people would probably just use GitHub, which is merely changing one big host for another.
When it's actually decentralized, when developers use their own git URLs on their own domain, suddenly you have to worry about things like the domains expiring. You have to worry about security practices of every single host. crates.io is a big target, but they know it and act accordingly.
crates.io doesn't allow deleting of packages, but an arbitrary git URL can disappear causing a "left-pad" incident.
Some of these problems could be solved with another protocol better suited for decentralization (where the data is immutable and content addressable, and identities are private keys rather than domains), but that's way more complex than "just use git".
In the corporate world this solved by using an intermediary like Nexus or JFrog.
The current situation with the language ecosystems is basically "we must build this very lax system with as few friction moments as possible, or else our language would be outcompeted by a language with laxer requirements" and "We must make our ecosystem grow, grow, grow! Externalities down the line be damned!", i.e. it's a capitalism, careerism influenced situation imho. So we have "You must vet all the 300 modules and their updates yourself" as the result. Requiring module authors to unify into meta projects to review each other's commits is perceived as laughable, unneeded self-hindrance. Same goes for separating the ecosystem into vetted(stable)/unvetted(unstable).
I would love to have a paid curated package repository, but it would be really hard to sell.
I know a handful of companies who care that much about their security not to depend on OSS libraries. Even FANG don't care.
If you can raise some money and get enough sales firepower to close big contracts with the few security conscious actors, this might become reality.
I'd rather have a system devoid of emotional biases, transparent and introspectable and rely on automated tooling to constantly monitor my dependencies.
Problems of Swift: Swift can add dependencies using git URLs, but there is no central package registry.
Both languages should have a central package registry and the ability to add dependencies using git URLs, as it would make flexible dependency management much easier. For a central package registry, it makes it easier to find popular packages, but it has a problem of centralization.
https://doc.rust-lang.org/cargo/reference/specifying-depende...
There is absolutely no lack of that in cargo; this is literally the first sentence in the relevant section in the doc (https://doc.rust-lang.org/cargo/reference/specifying-depende...):
Your crates can depend on other libraries from crates.io or other registries, *git repositories*, or subdirectories on your local file system.