Hacker News Reader

Comment by PragmaticPulp | Hacker News Reader

I use “cargo … —-locked” to install things using the dependencies from the Cargo.lock file, which includes specific commit hashes for dependencies. Avoids things like the 0.0.1 problem or even replaced crates. Need to be careful to watch for actual security updates, though.

I really wish crates.io would have at least launched with a name spacing feature. This wouldn’t solve every spoofing or typosquatting issue, but it would go a long way toward improving the situation.

There’s a separate issue of crates.io squatting. One person famously registered hundreds (or thousands? Tens of thousands) of common words as crate names on crates.io and has been squatting them ever since. Those names are effectively unavailable for use but also completely useless because they don’t contain anything.

It’s also becoming a huge problem for abandoned crates. New forks have to choose a completely different, less intuitive name because they can’t just namespace their alternative. As old crates get abandoned, this leads to weird situations where the newest, best maintained crate has the least obvious crate name. It takes work to find the good crate some times because the best named crates might just be the oldest, most abandoned ones

View on HN

I feel like "single shared namespace" works best when the ecosystem is managed collaboratively, with a clear leadership to make calls on who gets what name— for example, something like a Linux distro, where there are Replaces/Provides metadata specifically to facilitate these kinds of transitions and avoid being stuck forever with crappy legacy nonsense.

But this doesn't work at all in a free-for-all environment like PyPI, NPM, or Crates, where anyone can just grab a name and then have it in perpetuity.

IMO the Docker ecosystem got this right, with baking in a domain name as part of the container, and insisting that everyone on docker.io use a vendor/product convention. This meant that the toplevel namespace was reserved for them to offer (or delegate the offering of) specific blessed container images, much more in line with how distro packaging might work.

    # Get the "official" image, whatever that means (but you trust Docker Inc, so yay).
    docker pull nginx

    # Get an image supplied by a specific vendor.
    docker pull bitnami/nginx

    # Get an image from a different server altogether; maybe it's your company, or you don't trust Docker Inc after all?
    docker pull quay.io/jitesoft/nginx

Maybe the big-flat-namespace thing is still a years-later reaction against huge and unnecessary hierarchies in Java land? I think the ideal is not to permit infinite depth, but perhaps to insist on 2-3 levels.

Similarly, Perl has CPAN and system where if a module is unmaintained and the author unresponsive, new maintainers can be granted permission by admins to take over. This ensures the namespace is preserved, peoples dependencies keep getting updated.

> peoples dependencies keep getting updated.

Updated maybe with malware sometimes? Is it unbelievable that the admins sometimes would hand over an inactive module to the wrong person?

Maybe Perl is a less interesting target than npm so it works (so far) for them?

It's worth noting that this behavior is also a sticking point with some users and distros, and I believe was at least part of the reasoning behind podman being developed by Red Hat.

Sometimes you don't want some third party controlling and blessing what's okay, you want to explicitly control that by only allowing sources you control so you can't even accidentally use that third party.

Different groups have different use cases and it's hard to support them all well.

I definitely see both of sides of it— on the one hand, it's nice that "docker pull nginx" is always shorthand for "docker pull docker.io/_/nginx" and whether or not it is isn't dependent on some config file somewhere telling you what your primary registry is. We solved this for ourselves internally by just always using fully qualified names (containers.corp.com/foo) and not worrying too much about it.

But I can also see how this is absolutely a power grab by Docker Inc. For a company which has given away almost everything and has basically no moat in terms of technology, ecosystem, or thought-leadership, I can understand them wanting to at least retain the final say on which containers are which.

And of course, given that perspective, it's clear why a company like Red Hat would chafe against this, particularly when it's clear from the architectural differences in Podman (daemonless, rootless) that they were looking for an excuse to do it over anyway.

You mean you want to override or replace the global name registry?

I can't imagine that getting any harder by the global vs. scoped name distinction. It's just an extra feature.

It's a feature docker refused to implement, and is one of the reasons Red Hat created podman, which is a drop in docker replacement (you can alias it to docker for the most part), but it allows full control over what repos are checked.

For docker, as I understand it to get something functionally similar you have to define a mirror for docker.io that's not really a mirror and then prefer that mirror and/or disallow traffic to docker.io in some fashion.

I've heard this argument a million times, but I really don't get it. If you want to control which images are deployed, you want to be running your own image registry anyway. So you already have full control: just limit images to ones in your private registry with its fully qualified name. Why do you also insist on being able to deploy with 'short' names? Arguably it's better because users can't confuse "enterprise customized ubuntu:20.04" with the public one and bother open source maintainers with internal issues.

Nobody is insisting on being able to deploy with short names (or at least not the people that actually care about this). They are insisting that someone screwing up and using a short name when they shouldn't doesn't go out to a third party that's not explicitly trusted. Making this about short names is completely missing the point.

Requiring docker to be run with a "mirror" which isn't actually a mirror of docker.io and making it use your non-mirror mirror so docker.io can't be reached and everything still works is just a hacky way to trick docker into working in what many people think is a very valid way - where no external resources can be used as a container not just as a matter of policy, but as a matter of capability.

Put simply, it's not about replacing the default repo, it's about excluding all external repos. Requiring they all use long names would be a perfectly acceptable trade off for most orgs I think, if that was an option offered.

Ah thanks for clarifying. I think this issue gets muddled by people who do want to override the default repo to get short names, to which I roll my eyes. But it seems there is a real issue in that this:

> just limit images to ones in your private registry

... is actually not possible for a basic docker install. (Every time I use the word "just" it turns on me, and this is not an exception.) There are some implementations that have this feature (k8s, podman, etc), but the fact that it's not possible in the industry default container solution is an issue.

Github generally has two levels (organization and repository) which works pretty well.

It works pretty well, but might not if making org2/repo55 a copy of org1/repo55 (i.e., forking) weren't free for the user (namely, org2) and maybe a venture-backed startup can afford to keep it free for the user, but a non-profit cannot.

Forking doesn't cost github anything it's a git branch of parent repo. That's why you could see people accessing youtube-dl via a commit hash of github dmca repo.

I feel like namespaces came much later than Docker Hub, but I can’t really find a good historical record of docker registry features (or at least easily searchable ones). The global namespace was either forcibly cleaned or delegated to who was currently publishing them and projects raised issues if that wasn’t them. I vaguely remember annoyance that I had to publish my toy containers to the global namespace. I could be wrong though.

So per [1] it looks like Docker Hub launched in mid-2014. Admittedly my earliest pushes to there are more like the late-2015 timeframe, but definitely by that point I was pushing to namespace under my username. My impression had always been that the structure of Docker Hub was meant to mirror Github in terms of an org/repo scheme.

[1]: https://www.docker.com/blog/announcing-docker-hub-and-offici...

This one is frustrating because this is an issue that has been solved many times before and I hate seeing it repeated in every new package manager. A vendor name should always be required and the top level should be reserved for official/standard packages.

I want all of the following from a package manager:

1. Required vendor/namespace for third party packages

2. No multiple package versions. If there is a version conflict between transitive dependencies of a package because of semver, you should not be able to install that package.

3. Lock file and a separate 'install' command for installing the locked versions and an 'upgrade' command for updating versions via semver

4. Upgrade command should support a --dry-run option that lists the packages and versions that are to be updated and a --diff that lets you preview the code changes.

> 2. No multiple package versions. If there is a version conflict between transitive dependencies of a package because of semver, you should not be able to install that package.

I am grateful Rust allows this unlike C++ or Python. While I ideally minimize repeat dependencies, it is a big help to not be constrained to only on version. We've already had cases in Rust where some people were overly restrictive on dependency declarations (since Rust does block some versions as too similar) and it has caused a bit of pain.

For all the animosity that PHP gets these days, every single item on your list (granted, of very basic demands) aligns with PHP's composer. I am surprised that Rust is that much worse off than PHP in this regard.

I don't think composer has a diff option to dump the actual code differences before you update yet but yes most of this list comes from my past experience with composer. My current company doesn't use PHP but I look back fondly at how easy it was to audit my dependencies manually and be explicit about upgrades and transitive dependencies.

It does offer a diff option when you have local edits in the /vendor (for whatever insane reason). Always assumed it could be triggered manually as well. TIL.

I also love how easy it is to declare conflicts [1]. Some sub sub sub dependency down the tree had a bad 0.0.1 release? Just declare a conflict and have the tool do the work.

[1] https://getcomposer.org/doc/04-schema.md#conflict

> 2. No multiple package versions. If there is a version conflict between transitive dependencies of a package because of semver, you should not be able to install that package.

I think this depends highly on your environment. In npm land, where a typical project has hundreds or thousands of dependencies of dubious quality, this would be a nightmare. It guarantees that every single deployment will be different and risky.

This model works much better in linux, where packages are maintained by maintainers and there is not an explosion in the dependency network. Especially on a Debian or Centos box, you can be confident that upgrading packages won't break stuff.

If you are in node land I highly recommend using 'yarn install --flat' and I desperately wish this was the default in npm from the beginning. It would have radically altered the package development culture in a good way.

The way npm currently handles version conflicts is one of the primary reasons why using npm is currently a nightmare. The average node project will have dozens of abandoned or ancient package versions precisely because allowing multiple versions to exist means that these packages never get forked or updated. Each one of those packages is a ticking time bomb waiting to be taken over by a malicious actor. Forking is a better solution than pulling in unmaintained packages with out of date dependencies.

Why are multiple package versions a problem? In Rust they Just Work.

It can mess up when you are dealing with global resources. ie. two different versions of the same package both attempting to initialize global state (memory, filesystem, databases, whatever) will usually conflict with itself. Often in ways that will crash or even cause data loss.

I've never had that happen in Rust because multiple versions of a crate effectively live in completely separate worlds: they have their own copies of globals, for example. Can you name an example of multiple versions conflicting at runtime in Rust?

By globals, I mean global resources outside of the codes namespace. It may not even be a resource in the process, such as a log file or temporary directory or a database. If you have two versions of a crate in their completely separate worlds, and call both of their init_logging() functions to log to a file specified by an environment variable, things are likely to go pear shaped when they stomp over each others log file.

I'm a Rust novice, but the example I tripped over in Go was https://github.com/golang/glog. It has a module level init() initialization routine that makes calls to the stdlib flags package, manipulating the default command line flags (a global resource). If you ended up with multiple versions of glog via transient dependencies, your program would panic on startup as the second version's init() would make calls only allowed to be called once. Rust thankfully avoids this particular one by requiring initialization to be called by main() (apart from the hack described in the article).

First of all, in Rust global are highly discouraged and if there are shared resources then they should be defined or initialized by binary writer not the library writer Otherwise the library is not of good quality.

Regarding go I can't say as I have never programmed go.

I think if somebody decides to act as a vandal I can too?

https://crates.io/users/swmon

Edit: Lots of love for this user in the thread....

https://github.com/swmon/Charles-Crack/pull/1

While I can agree it's a dickhead thing to do, it seems to be entirely within the rules of crates.io, so I think the hate should go towards them, not someone who's just playing by the rules.

Domain squatting has been a thing for decades, it's not like no one expected this kind of thing would happen.

Just use org/author namespaces for crying out loud.

I am a mild proponent of namespaces. But adding them means you are now managing an identity layer, which is not something that a volunteer organization wants to deal with. In a flat namespace, users don't expect that a google-foo library is actually from Google. But with namespaces, people expect that everything under the google/ namespace actually is officially supported by Google. So when the initial rush to register names happens, who verifies that Google owns the google/ namespace? And not just for Google, but for every litigous trademark-defending organization under the sun? And since the package repository is supposed to be immutable, you don't want to wantonly delete crates from namespaces that they originally registered. Likewise, you can't go transferring ownership of a namespace from one entity to another, because that violates the security assumptions of your users who originally trusted the original owner of the namespace and not the new owner.

Identity layers are hard. It's a problem worth solving, but it's not going to happen on a volunteer basis. It needs full-time paid employees to deal with this. Perhaps the Rust Foundation will make such a thing a priority.

These problems you raise have been solved problems for a long time.

Maven Central uses verified reversed domain names, so to own `com.google` you need to prove you own the domain `google.com`. It's not hard to do it.

Dart's pub does the same thing with verified publishers.

crates.io already requires user accounts from publishers, so it already manages identity.

The only reason they don't require namespaces is to make crate names look cooler and the barrier to entry non-existen, so they can amass large numbers of packages in a short time (then deal with the fallout only when it becomes an emergency), I can't see any other reason anywhere, including from the ones you've raised.

> crates.io already requires user accounts from publishers, so it already manages identity.

It currently requires a Github account, but the registry is not tightly coupled to Github. But once you introduce Github identities as something that code is explicitly depending on, now you're at the mercy of Github for the rest of time. Furthermore, immutability is a desirable property of package registries, and Github identities are not immutable; Github allows people to change their usernames and then allows any random person to snap up the old username.

> The only reason they don't require namespaces is to make crate names look cooler and the barrier to entry non-existen, so they can amass large numbers of packages in a short time

No, this is needlessly conspiratorial. Please exercise some perspective. crates.io is run by volunteers, and managing an immutable identity layer is not something that the current crop of volunteers wants to commit to.

When (not if) a domain name expires or changes ownership, how is this detected and the connection re-verified?

It's interesting people like to bring this up because this is also a non-issue as a publisher needs to provide PGP keys... if the domain expires and someone else takes it over, their PGP keys will be different and it's up to users to decide whether to trust those...

When a domain changes hands, it's also up to users to decide whether to trust the new owner's TLS certificate... I am saying this because this is forms the basics of Internet security, when people bring this argument up, they show a certain ignorance of how things work on the Internet in general.

Saying it’s “up to users to decide whether to trust those” is simply saying that the problem is currently not solved. I mean, either the package is tethered to a domain name or a PGP key. Pick one.

You can, of course, have both, if you use DNSSEC and DNS “PGP” records; the PGP key is then validated by the DNS.

> Saying it’s “up to users to decide whether to trust those” is simply saying that the problem is currently not solved.

Again, Internet security is mostly based on trust and there's no way around that: when you enter a URL on a browser, you need to trust that the domain belongs to who you think it belongs, and the only way you have to do that is to trust its TLS certificate... which you do by explicitly trusting your browser to only recognize reliable TLS certificates signed by a legit root certificate authority... unless you decide to whitelist TLS certificates yourself by obtaining the domain's root certificates out-of-band, something nobody seems to think necessary to do, even the most security-conscious government agencies probably go to that kidn of trouble.

Basing package ownership on the same kind of mechanism is the best you can ever do... if you consider that as not being a good enough solution, then you should never use the internet with consumer-grade browsers at all, let alone use package managers in any form.

Describing the current CA system for TLS certificates as the peak of what is currently possible is simply untrue. As it happens, there is a solution, and I described it. Domain names can be tied to a strict key hierarchy using DNSSEC, and DNS for domains can in turn contain PGP keys (or any other kind of verification data). This way, a package could be directly verified as being issued by a current owner of a domain name.

The fact that we all use and trust CA-issued certificates today is not an argument for accepting that system as the best we could ever do in the future, or even today. And the fact that one wishes to improve the current situation does not make it wrong to use the current system as it is – to claim otherwise is to be Mister Gotcha (https://thenib.com/mister-gotcha/).

What a weird line of reasoning. That something is allowed by "the rules" doesn't make it morally justified. Assholes should be called out on their behavior, rather than hating on the creators of a system that believe people to act in good faith. Sure, those creators can take steps to mitigate malicious behavior, but the onus is not on them.

You must be from a very nice place, where people not only follow the rules, but also restrain from abusing them even when not morally justified to do so.

I live in "the real world" where this is not how things go, and abuse is absolutely expected.

that doesn't mean abuse isn't an asshole move

My understanding is that validating identity for package authors is a hard problem thus expensive to solve robustly, and the crates.io folks have hitherto deferred tackling it in earnest. That is arguably a responsible approach up to a point, in that they haven't committed prematurely to something half-baked.

As described elsethread, there is prior art — Maven's identity verification is substantially better: https://news.ycombinator.com/item?id=29266591

Validating ownership of a namespace reliably enough that it is difficult to spoof is tough. It's possible for PGP creds to be stolen. But then at least the keys can be revoked, and old packages signed with a new key.

I believe Debian Maintainer process and its keyring is one, if not only, project that gets this as good as it can get.

https://wiki.debian.org/DebianMaintainer

They don’t need to validate anything, though. Just treat the namespace as something that can be claimed exactly the way that a crate name can be claimed.

Only owners of the namespace can add crates to that namespace, in the same way that only people who own a crate name can publish to that crate.

If you're not validating anything, then that violates the assumptions of users who see an amazon/ namespace that isn't officially associated with Amazon. We don't even need to presume malicious or squatting intent; the entire left-pad fiasco was precipitated by a person who innocently registered "kik", and then later a company named Kik demanded that NPM transfer ownership to them. This happened with a package, but it would just as easily happen with a namespace. Once you introduce an identity layer, people start to expect identity validation. It's a problem worth solving, but crates.io would need to move beyond being merely staffed by part-time volunteers to solve it.

> This happened with a package, but it would just as easily happen with a namespace.

But that’s exactly the point. This is equally a problem for packages and namespaces, so it should not be considered a reason to avoid adding namespaces to a system that already supports packages.

And namespaces do help. Yes, maybe the user has to externally validate that the namespace is registered to the owner they expect, just like they would for an individual package. But the difference is that, with a namespace, they’ve now validated the ownership of every package inside it and don’t need to repeat this process for each package.

> But that’s exactly the point. This is equally a problem for packages and namespaces, so it should not be considered a reason to avoid adding namespaces to a system that already supports packages.

I think this is missing that crates.io doesn't transfer packages between owners. That NPM decided that it was acceptable to unilaterally transfer ownership of the kik package from the original maintainer was an lapse of judgment on their part. It would be unprecedented for crates.io to begin doing so.

> And namespaces do help.

I agree with this entire paragraph and have made these same arguments in favor of namespaces before. As I mentioned, I am mildly a proponent of namespaces. But we can't delude ourselves into thinking that an identity layer is simple to maintain, regardless of how much we want namespaces. It's a messy social problem.

Robustly verifying identity isn't enough, because a developer with an outstanding reputation could lose their credentials to a spearphishing attack, and the attacker could then modify crates using stolen credentials.

I agree that authentication is not sufficient on its own, but I argue that it's necessary as part of the solution.

A source package should meet the following criteria:

• Package contents match exactly the source tree at a PGP signed commit at a public repository URL. This must be verified before the package is made available through the package manager.

Now, if I am whitelisting PGP keys, an attacker needs to steal creds to get something by me.

Ideally you want multiple signatures by trusted keys prior to publication. Each additional signature makes it significantly less likely that a package is provided by a malicious attacker.

EDIT: Hmm, how about package signing parties held over videochat? If we already know each other and you tell me that a particular package was created by you, I sign it.

The lack of namespacing in crates.io has always been a strange decision to me. I hope at some point they decide that it's worth it to introduce namespaces. The could use some an aliasing mechanism to avoid breaking any existing package references.

From previous times this has come up, I was led to believe that it's not so much that the maintainers don't want namespaces as it's a thankless and overwhelming task just maintaining the status quo, and so they don't have the time and resources to implement something.

Hopefully this is one of those issues that the Rust Foundation can direct some funding toward as the entire Rust community would be all the better for it.

This is passing the buck in the worst possible way.

Not implementing namespaces is fantastically lazy, to the point of outright professional negligence.

If you can't be bothered to lift a finger to type in the few hundred lines of code it takes to implement namespaces, then by definition you're not responsible enough to run a public package repository.

Do it properly or don't do it at all.

There's a lot more to it than a few lines of code, see: other comments on this post

There really isn't, and everyone else is just making excuses.

Whatever the effort level is, it is certainly much lower than the thousands of Rust users navigating a tangled mess of typo squatting, name conflicts, crate spam, and difficulty in identification of official or related crates.

Again: If the crates.io team doesn't have even just this much time, then by definition they don't have the time to manage the site responsibly. They should shut it down, seek funding, or hand it over to someone else.

Doing anything else is irresponsible.

Problem isn't creating namespaces.

Problem is what happens when someone claims google, amazon, and then stars doing shady stuff with it.

And maybe that somebody, was working for such organization and then left on bad terms and doesnt want to give login code or whatever.

It's not that easy problem to solve, and you need to have people on standby to resolve such issues, some of which might include layers.

Implementing namespaces in code are not even 10% of effort for this to work.

Maybe "seek funding" (like jiggawatts wrote) is a good idea? Maybe MS and other companies would want to pay for a namespaced and identity verified Crates.io -- it's in their self interest?

https://users.rust-lang.org/t/name-squatting-on-the-crates-i...

This a very old discussion if you keep digging through the links.

I hope that the Rust Foundation (which finally exists) can put in some time / resources to help fix this.

I totally get that maintaining crates.io is a time suck for all involved, and mediating disputes would increase that even more.

But at some point, this is going to need to be addressed.

It's heartbreaking to see crates.io repeat nearly every mistake which npm has committed. I did not expect that from the Rust crowd which usually tends to design things carefully and with correctness and corner cases in mind.

Reply on news.ycombinator.com