The Rust community's crate host
crates.io
crates.io
"Take care when publishing a crate, because a publish is permanent. The version can never be overwritten, and the code cannot be deleted. There is no limit to the number of versions which can be published, however."
That. Is. Awesome.
But once published, isn't such information already compromised for all practical purposes ?
I can understand the legal reason behind "pulling it down" (using someone else's IP, etc), but practically speaking the private information once published cannot really be "undone" (at best you can make it hard to find, and disassociate from yourself).
If it is only up for 1 hour, the chances that anyone malicious sees it is low.
If it is there forever, the chances that someone malicious eventually sees it approaches 100%.
CPAN packages can't really be uninstalled (http://stackoverflow.com/questions/2626449/how-can-i-de-inst...) and seem to be unnaturally prone to dependency hell.
The concern about running the entire test suite on module installation really isnt a problem if you think about. Additionally you are able to force the install if it fails on account of a test.
While that behavior feeds back into improved quality for responsive maintainers, it takes additional time during installation. When combined with point #1, it can make dependency chains seem more fragile.
Generally, if you've published a package, it's possible someone has downloaded it. If you take it down, that's going to be strange and unexpected behaviour for those people.
Someone has accidentally posted a package that contains a 2TB file, and now all mirrors have to sync it.
Someone accidentally puts their personal information in a package.
2TB file: That's .. nonsense. I assume guards are in place to prevent the oldest form of DOS attacks. If not, the guys at cargo.io will learn and .. make that go away?
Personal information: That looks like the only case where I sympathize with the guy uploading stuff. That said, this is how the net works? Publishing sensitive stuff to Github means that it might be out there forever (force pushing a new history doesn't mean that no one cloned the stuff before or just grabbed a zip of the current head).
For me its a win. I certainly can imagine some scenarios that might be painful, but .. that usually boils down to your third example, a developer error. The usual issue with 'removing packages' is that the user suffers. My gut feeling is that there are far more users that get 404s than developers that share their API keys.
* you push a revision which introduces a bug
* you push a release which introduces involuntary API breakage
* the new release has a glaring security issue
* release X relies on a third party which has changed (think: some web service) and therefore doesn't work anymore
Sure, you can push a newer release but you don't want _anyone_ to be using the old one.
I'm not saying yanking is good, but maybe a notification system "this package should not be used, upgrade to XXX" would be useful.
"NuGet.org does not support permanent deletion of packages, because that would break anyone who is depending on it remaining available. This is particularly true when using the workflow that restores packages on build.
Instead, NuGet.org supports a way to 'unlist' a package, which can be done in the package management page on the web site. When a package is unlisted, it no longer shows up in search and in any package listing, both on NuGet.org and from the NuGet Visual Studio extension (or nuget.exe). However, it remains downloadable by specifying its exact version, which is what allows the Restore workflow to continue working.
If you run into an exceptional situation where you think one of your packages must be deleted, this can be handled manually by the NuGet team. e.g. if there is a copyright infringement issue, or potentially harmful content, that could be a valid reason to delete it."
(Source: http://docs.nuget.org/docs/Creating-Packages/Creating-and-Pu...)
As you may be aware, there is limited support for versioning APIs using "go get" (Go's nearest equivalent of this). The developers of Go argue that the maintenance of your dependencies is your responsibility, and you should clone everything you need to your own environment. That gives you ultimate control, allowing you to pull and update them as you require, modify and bugfix as you see fit, and makes it hard, if not impossible, to end up relying on two packages with different versions.
If a project B depends on project A, then they must collaborate in some way to make sure that there are no breaking changes introduced between (minor) versions. If another project C also depends on A, then they must both use the same version of A if they are to be used together. The maintainers of A, B and C all have a duty to ensure that their code remains up-to-date and backwards compatible.
It seems that Rust values a different freedom. By specifying the exact version you require, there is no need to keep separate projects in sync. The flexibility comes at a cost; you may find you have different versions of many dependencies. Because they are not required to, maintainers may neglect the tedious task of keeping their projects' dependencies up-to-date, requiring bugfixes in multiple places in your own codebase.
Clearly, some of the smartest minds in our industry disagree what the best tradeoff is. It is interesting to see the range of opinion though.
> By specifying the exact version you require,
This is only half true. It comes in two steps: you specify a _kind_ of version that you require, like
foo = "^1.2.0"
This also matters because...
> maintainers may neglect the tedious task of keeping their projects' dependencies up-to-date,
You just run 'cargo update' and it will re-calculate that dependency graph, and re-write out the lock file. You're as up to date as you can be. Of course, if that fix is only available in a range outside you've specified, that's different.
Subtle things in dynamic linking may break applications when dependency versions are changed.
Static linking lets you bundle whatever version is known to work, including the most up to date versions.
I should also state I fall on the extreme end of the spectrum. My belief is that the moment you depend on upstream package X it is wholly your responsibility to track changes and adapt your code accordingly. I don't believe software versioning does anything but encourage poor security, performance and general fragmentation.
I believe that before pushing your changes to a remote repo, you locally check that everything is stable, meaning there is no dev branch or stable releases just one true master branch!
If you don't want the bleeding edge then don't 'git pull'. simple!
The world goes boom! exponential overload!
... at least it get's really frustrating after a while, twice the space, twice the compilation time. And you'll know at least one of the libs has bugs lurking in it that have been fixed elsewhere except in that lib:(
That's the last time where I heard about this wonderful idea that vendoring everything is a solution, because versioning is hard: https://www.kickstarter.com/projects/2066438441/haunts-the-m...
I'll take the solution where I can send a PR to the maintainer with updated dependencies over this sort of thing any day.
Well, that's a problem waiting to happen. Infringing source? Malicious material? It's a publishing facility and so will be abused.
Of course, I can verify the contents of the script, but I still don't know that the script I've verified on their website is the script that's being run in my shell. Is it OK because the script is hosted over https, and therefore can't be modified on transmission?
There are various issues filed in the repository if you want to track when this changes.
This is something to take seriously; but not something to dismiss as entirely unacceptable. It is generally a manageable, and somewhat quantifiable, risk. This is no different than if the Rust developers provided an RPM or deb package for you to install; the same privileges are being granted to them. Though it is the same thing, it doesn't seem to generate the same concern as the "pipe this file into a shell" method of installation. In fact, one of the folks who responded to you explained that this problem will be gone once rust is packaged for distribution...but that merely means the commands are different, not that rust developers aren't being granted root access to your system.
It is possibly even more suspect, in the case of binary packages, since in the case of the shell script, you could download it, look at it, and then run it. A binary package might have modifications that aren't included in the source distribution, and it would be difficult for a non-expert to spot those differences (and an expert would need to go looking for them, and might miss them).
This is why signed packages are a big deal in the free OS world. At least you know who signed off on your package.
In short, you're right: It's kind of crazy that people grant root access to arbitrary people every time they install software. But, it's the way things are done, for now, so we deal with it and only install stuff from people we perceive to be trustworthy. (i.e. be careful what apt and yum repositories you enable on your systems, and what scripts you pipe into a sudo shell.)
This is just a bad habit from the OS X world that needs to die. Really there should be some part of the step that allows a typical user to at least try to manually authenticate the root of trust, via a published hash on a bootstrap package maybe (that's what third party RPM/deb archives do, for example).
If I don't trust the hash value I got from the Fedora web site I can ask Google. I can check it vs. the mirrors. I can ask my friend who installed it already to see if it matches her file. I can check the signature on the hash to make sure it matches the key I have stored from my existing install (or the one before that, etc...). That's not perfect but at least it's something.
Or if it's there somewhere I don't see it. Where it it?
I'm aware of no situations where linux distro mirrors have been compromised with code that will install on live systems. Again, the key is set up on the clients at install time, you can't drop compromised software on them via package install without stealing the signing keys.
The whole point here is that the "run the shell script off hte web" skips the whole root-of-trust issue, and suffers for it. It must stop, guys.
In the end is the same level of security, this one at least you can first download the script, read it and then pipe it to bash much more easily than a rpm/deb.
Having the Debian approach is far better here!
But that just pushes the problem on the user: now the user needs to know affirmatively that "rust-lang.org" is the correct source. And they need to know the same thing about every piece of software they install using this insanity.
That's a rather different situation than simply trusting the install-time keys shipped on your system, which you can authenticate manually (via a public key check against a well-published, historically attestable key) with comparative simplicity.
To get a reasonably fresh version, you would need to use something like Ubuntu's ppa where you need to trust the ppa's author and TLS, to get you signed packages and the corresponding keys.
"keys + signed package" via TLS from a known (via google) site is more secure than "human readable sh script" via TLS but it is not by leaps and bounds.
Systems need to be made more secure and curl+pipe is a step in the wrong direction. It's kind of shocking that attacks are becoming more sophisticated and prevalent but security is becoming more complacent.
> only install stuff from people we perceive to be trustworthy.
Except you can't authenticate the script with curl+pipe, so if someone compromised the server or the communication link it's game over.
If you install packages from a third party, and if they are signed, you are generally getting the signing key from the same location. i.e. when the rust developers build packages (as has been suggested will be the solution to this problem), you'll also need to install their signing key. Presumably from the same webserver that is serving you the script that is being judged insecure in this thread.
"Except you can't authenticate the script with curl+pipe, so if someone compromised the server or the communication link it's game over."
s/package and signing key/script with curl+pipe/
The two are not dramatically different if an attacker has taken over the website in question, or if the build/distribution infrastructure has otherwise been compromised, and that was my point.
You could do an sha checksum of the script, and you could obtain the package signing key by meeting up with the developer in person and checking their ID, and you could download the sources, check them thoroughly, and build your own packages, etc., etc. Very few people do that.
There are levels of risk at every stage. It is a difficult problem to distribute software in a safe fashion.
All of that said: I install everything from a package. If it is not from the OS standard repository, I download the source package and build it myself (sometimes checking the source tarball for anomalies and checking the patches to be sure nothing is obviously amiss). I do not enable third party repositories unless I trust them a lot. If no package is available, I make one, and sign it with my own key. If I can't make one, I don't use the software in a deployment (the cost of maintaining a piece of software built from source is way too high).
I was not suggesting the curl+pipe is a good practice. Merely that we live in a world full of compromises. And, insuring you are getting what you think you're getting, and that the person who packaged it had no malicious intent, is a lot harder than merely installing from a package (particularly a package from a source that doesn't have stringent identity requirements...Debian and Fedora packages have a pretty clear ownership trail...most third party packages do not).
Moreover, there are things you can build on top of signed packages (distributed consensus of signatures and keys) that can't be built on top of curl+pipe.
curl+pipe is mental.
But, lots of people add third party software repositories without out-of-channel confirmation of signing keys, and without any awareness of the risks involved in that. I see it more than most (I support server-side software products with a million or so installations), and I'm amazed at how often third party repos are enabled on people's systems, seemingly without any reason other than "more software is better". They often don't even know why/when they enabled those other repos or who operates them.
That's no different than this curl+pipe scenario.
Systems need to be safe by default and they should largely service the needs of users. Distros are no longer doing that.
Most likely because they either blindly followed instructions from a website or blindly ran an install script from a website, giving it root access. In which case it is basically equivalent to curl + pipe.
What if it dies and the last thing it got was "rm - fr /" but not the rest after the / ?
You can try this equivalent, killing the server when it tells you to - https://gist.github.com/aidanhs/e40417381e9aecb87b35. It prints '/', i.e. would remove your root directory if it was an `rm` command.
Won't be doing that again then.
What alternative do you propose? You can check the script whether it does something malicious, but then it downloads and installs some binaries.
All it does is bullshit of 100 lines or so, but the main part of what it does is
download a tarball, make some temporary directory, and run install.sh
How difficult would those instructions be? Here take a tarball, and you know, unpack it and run install.sh
I guess, too difficult for the audience that Rust developers are targeting with this curl this | sudo bash bullshit.
However, when I first went to the site, all I could think was "Whats a crate in relation to programming or more specifically Rust? Is it Rust's idea of a package (eg like ruby gems)."
I just think it would be good to have the site say Cargo: The Rust language package manager.
The site also says this:
"The Rust community's crate host". Is it not a cargo host for distribution of crates where crates are Rust packages? Is there non-community crate hosts?
I just don't feel comfortable working on someone's code if they aren't willingly consenting to it. just not cool at all! Human consent is just not a permanent thing.
Besides if the file disappears from the remote source, you can still build for the time being. It's not the end of your project, but it may be the end of someone else's career - you never know what reasons people had. Just respect peoples rights over their data.
The constant argument, "it's already on the internet" doesn't absolve any of us, from respecting peoples reservations. And if it's "already on the internet" then you can trust that someone will illegally copy and repost the repo on-line but where does that leave us ethically?
not the way I roll.
But fortunately "thanks" aren't zero-sum, so I'll leave your ill-gotten gains alone. :p
Edit: There are plenty of crates not categorized, and that must be how I didn't see this feature.
And couldn't the same crates end up occupying all the "good" crate names?
I assume Rust will be around for a while (I certainly hope so - I've just started learning it!) so it might be wise to plan for growth now before it's too late to implement.
While it's true that sometimes collisions happen, in practice, you just choose a new name. Everything is significantly simpler, which in software is very often a Good Thing.
Just from my own experience, I know that it makes me less likely to make packages public if I have to give them odd, collision-avoiding names first - especially as I then have to use those names myself. It's particularly annoying when the incumbent package is 10 lines of poorly-written abandonware that hasn't been touched in years, which often seems to be the case.
I know this isn't an issue for Rust now but I'm thinking about ten years down the road when you have 250,000 crates on the site, 99.9% of them junk, clogging up that flat namespace. (Python's repository has more than 50,000 packages and seems to be growing at an accelerating rate now that it finally ships with a decent package installer by default.)
Nothing is impossible, of course, but the language already has a flat crate namespace. So you could either implement some kind of layer on top of that, or change the language itself to have some sort of crate hierarchy somehow. All just to have two crates named 'http' rather than one named 'http' and one named 'requests' (for example.)
Here's another way of looking at it: shipping a flat namespace is a proven, well-worn solution. Adding in a hierarchical one, on top of the language, which has a flat one, provides dubious benefit, for significant implementation complexity.
Seriously, I do get that there are trade-offs. I'm just interested in how you've been weighing up the alternatives as I do often find myself wishing that other projects I've been involved in had taken the hierarchical approach at the beginning when it was an option.
Developers who really like a name could always add something such as using "moz-foo" instead of just "foo".
I've been surprised with how few name collision problems I've had on Python's Pypi. If "z-foo" module is more usable than an unmaintained "foo" module, everybody recommends and uses "z-foo" without a second thought.
I'm somewhat uncomfortable with the idea, but I'll gladly defer to people with more experience on this matter than me.
Besides, the only alternative I can come up with, is some form of procedure for voting out crates which doesn't play nice with the community...
https://www.npmjs.org/doc/misc/npm-disputes.html
Not an urgent need, but a similar process will need to be implemented.
Naming conventions could be nice too.
Naming conventions are currently "the same as Rust identifiers".
BUT.
1) Why store a package, wouldn't an index rather than another repository be more efficient and flexible.
Cargo already has the capability to get packages from git urls...
I had some other nicks but I see this is on github so I'll try and contribute there.
I like the site, and can't wait to test it out. Thanks to all who have been working hard on this!
I suppose the authors experienced the "guy deleted his Github repo" effect one time too many. It's a very sensible decision. Getting packages for git urls is fine when you're hacking something together, but there is absolutely no way I'd put anything in production relying on something that fragile.
Besides git is hardly fragile especially if we are talking github. I happily clone and pull projects all the time without a single hitch. Also if you think about it github/attlassian are all dedicated to task with datacenters around the globe, while for the rust team the repo might be somewhat of a side gig.
And why do you think depending on a repo is worse than depending on Github? If the package is not maintained anymore, license willing, you can always fork it and bump the dependencies yourself, even if, in the worst of cases, you need to keep the fork local. However, if it turns out you have to maintain a lot of third-party libraries yourself, it's a sign either of a bad ecosystem, or that you picked the wrong libraries.
Let's agree on one thing we shouldn't be maintaining other peoples abandoned/illegal libs, however thanks to them being posted on the crates-repo permanently, we'll continuously keep running into them...
Maybe. But at least, if you pick your or somebody else's unmaintained code from five years ago, you have a fighting chance of making it compile and run.
Awesome work!
Why don't we port something that's already got an eye for security, like Minix?
For other install methods see: https://github.com/rust-lang/rust/issues/16456