The lack of namespaces on crates.io is a feature
samsieber.tech
samsieber.tech
Let's say you have a crate (Rust's terminology for a package) named "foo". Inside "foo" you have a few subcrates (named "foo_bar", "foo_baz", "foo_qux"). There are a number of situations where a crate will have subcrates. Sometimes these subcrates are intended to be used publicly (e.g., you want people to be able to use just the "foo_bar" subcrate if they don't need the whole "foo" crate) Sometimes these subcrates aren't intended to be used publicly (e.g., Cargo forces you to put your proc macro in a separate crate, but that's an implementation detail of "foo" and isn't something other people should use directly).
So now you have "foo" on crates.io, as well as a bunch of "foo_*" subcrates on crates.io. Currently there is nothing stopping some random developer from publishing their own "foo_haha" crate. This new crate isn't related to your "foo" crates and might even be squatting on a name you would have liked to use for a subcrate.
Or multiple people might want to create different crates for similar things. e.g., there are multiple regular expression crates. But only one of these crates got the "regex" name. Everyone else has to use a different name, and sometimes people use creative (unintuitive) crate names.
So for various reasons, some people want namespacing on crates.io that would allow someone to reserve an entire namespace. This would allow you to prevent others from using a name that might be confusingly similar to yours. It also allows you to use conflicting crate names if multiple people want to publish different crates with the same name.
TL;DR some people want something like npm's scopes for crates.io. And other people don't want it. This has led to much debate and arguing and whatnot.
It's funny, though, because the team that owns crates.io almost never weighs in on the discussion, and on the rare occasion that one of the team members does say something they always do it in a personal capacity and not in a way that represents the team. This has created its own set of frustrations.
A better title may be "No Namespaces in Rust's Package System Is a Feature", although that is long.
There's two big things here:
1. "The discussion" has often happened outside of the contexts that the teams are supposed to evaluate; that is, RFCs are the way in which policies are changed, and the vast majority of the discussions here haven't been on RFCs, but on forum threads. Team members cannot be everywhere, and comments outside of the decision making process are kinda inherently non-normative.
2. Team members used to comment more in these threads but they've gone back and forth and around and around for years, and people are frustrated by it, so they've just stopped commenting in these threads. See #1. When it's optional, and it isn't pleasant, it shouldn't be a surprise that people don't participate.
(I am not on the relevant team anymore, to be clear.)
If you community is clamoring about something, but no one has stepped up to make an RFC for the situation, you have to do two things:
1. Make an RFC for the clamor
2. Figure out why no one has done this already
Also, characterizing it as "no response" is not accurate, as I said, people have not come up with proposals that addressed concerns. It hasn't been ignored. Just not every person has the ability to comment on every thread in every forum.
And there have been some people who have put forward proposals. There's an open RFC I linked downthread about a related problem, "crate name transfer," and there was an RFC that was opened two weeks ago, but the person closed it rather than respond to the team's feedback. That there haven't been successful proposals is different than "there have been no proposals."
Personally, I think that we haven't had movement because it is an incredibly complicated space, with a ton of overlapping desires, and a lot of complexity, and a lot of angry, low-signal discourse. That nobody wants to wade through all of it isn't surprising. I'm glad this blog post was created; it's the exact kind of work that needs to be done to get things moving in a positive direction.
That directly contradicts your previous comment (that RFCs are the contexts that the teams are supposed to evaluate). Either:
1: non-comprehensive discussions are appropriate for a RFC, and random forum threads are the wrong place for that, in which case your previous comment is absolutely correct that it's unnecessary and likely counterproductive for the crates.io team to weigh in on said threads, but that still leaves the question of why they [edit: the crates.io team] haven't made such a RFC [edit: and told people commenting on the forum threads to instead comment on the RFC, where the crates.io team can evaluate] yet.
Or:
2: non-comprehensive discussions are not appropriate for a RFC, and should take place elsewhere (ie random forum threads), in which case it's completely reasonable to expect normative (but obviously not binding) feedback on what would be necessary to produce a RFC-quality proposal in the first place.
Personally I weakly favor the former view, on the basis that RFC literally stands for "Requests For Comment", but either one would be a sensible position, as long as one doesn't try to assert the negation simultaneously.
> that still leaves the question of why they haven't made such a RFC yet.
Yes. I speculated a bit in my last paragraph.
> Yes. I speculated a bit in my last paragraph.
Sorry, I phrased that poorly; see edit.
Because as I said before, coming up with an RFC means putting together a viable plan. The team has not come up with a viable plan yet. Therefore, there's no RFC from them.
Anyone who feels that they have a viable plan is free to submit an RFC, and then the team will discuss it and give them feedback.
Okay, so it's actually number 2[2], then? Which is a prefectly resonable position, except that it's the opposite of what you said before.
> "The discussion" has often happened outside of the contexts that the teams are supposed to evaluate;
(Also:
> 1 is correct, sorta. Comments on an RFC are absolutely fine. That is their purpose. Random forum threads aren't the wrong place for that, like, you can discuss ideas for RFCs on forum threads, but you can't expect the team to always weigh in on threads.
but I'm not clear on how much of that was attributable to my poor phrasing of 1.)
2: "non-comprehensive discussions [eg, ones lacking a viable plan] are not appropriate for a RFC" https://news.ycombinator.com/item?id=24449289
1. You write up a proposal. We have a template so that you get the structure right and some general prompts to help you ask the right questions.
2. You can post it on the forums, as a "pre-RFC." This is like a vague temperature check from anyone and everyone. Nobody is required to pay attention to these, but it can be a valuable step if they do. This step is optional for everyone involved, you can go straight to step 3.
3. You take your proposal and submit it as an actual RFC. The teams are required to pay attention to these, and they either accept the proposal, reject the proposal, postpone the proposal, or give feedback designed to drive the consensus of the team towards one of these states.
When people say "the team is ignoring threads," they are referring to threads that are in step 2. Most discussions aren't full proposals yet. And they generate a lot of discussion. And then people never move to step 3. Those that do are guaranteed feedback from the team in some fashion.
Your initial edited comment:
> that still leaves the question of why they [edit: the crates.io team] haven't made such a RFC
Nobody is required to make an RFC for anything. If the team doesn't want to write one up, they don't have to. It's on the advocates of the feature to start the process, no matter if they're on the team or not.
> [edit: and told people commenting on the forum threads to instead comment on the RFC, where the crates.io team can evaluate] yet.
They didn't say "comment on the RFC," because there is no RFC at that stage. They said "please make an RFC." This is the same process every single major change to Rust goes through. It's pretty standard. The vast majority of people who desire this feature never actually do it though, because they tend to post on the forums, get a ton of comments from people, and then burn out, it seems.
I hope this helps. To be honest this thread is so old I'm likely to lose track of it and not reply further.
The process is open to anyone. I don't recognize the author of this post, but this is a valuable contribution to the process, for example.
Not at all! In fact, npm's implementation is why some are pro, but also why some say that it's not worth it; see https://news.ycombinator.com/item?id=24444852 for some npm specific sentiment.
(Incidentally, npm added scoped packages in 2015, after these decisions were made for crates.io. However, crates.io is still smaller than npm was then...)
There isn't really a TL;DR because this space is complex, and there's a lot of details, and they matter, and different people have different reasons for not wanting them. I think it's good that github has namespaces, but that doesn't mean I think that Cargo should have them, because they're fundamentally different services.
The short reason we gave back in 2014 was:
> Because namespaces are strictly more complicated in a number of ways,and because they can be added compatibly in the future should they become necessary, we’re going to stick with a single shared namespace.
from https://internals.rust-lang.org/t/crates-io-package-policies... which is linked in the post.
The npm's namespace is basically just a preserved prefix, mention scoped package inside another scoped package still requires you to write full name (scope + package) or it didn't work.
I don't really see a reason that you can't implement it in a flat name package manager? Something like "you can't publish a package named as 'USERNAME_private_WHATEVER' unless you are USERNAME"
1. the crates.io team is understaffed (unlike NPM it's not a VC funded entity), and it has not been a priority for them (or for anyone deeply involved in Rust/Cargo/crates.io development, AFAIK)
2. there are some core design decisions around the feature for which no consensus has been reached yet
The developers have convinced themselves that they're right on this, having, I suppose, not learned from the hundreds of package managers that have come before them. They're wrong; full stop, no qualifications, no asterisk, no footnote, they are totally wrong on this and it is inevitable that their opinion will change.
But, this is the nature of open source (and even startups). New projects start under the (oftentimes correct) belief that they can do something better, which experiences linear extrapolation into believing everything you do is better through virtue of being unlike what everyone else is doing. We see the end-state of Maven, NPM, etc; we don't see the war that those projects went through to reach this state. History is elided, forgotten, then repeated.
Comments like this are why this discussion has devolved over the years. It's "all heat and no light." It does not help convince anyone that you're right, and if you want this to change, I would suggest that you change your approach.
The team has stated that they are open to implementing namespaces, but that the people who advocate for them need to address the concerns. This hasn't been done successfully yet, and is part of what this post is trying to help out with, by cataloging some of the large amounts of previous discussion.
What are the other possible concerns? The only comments I've seen on this is "It's not any different, people will just squat namespaces" to which I reply: reserve namespaces up front for Rust nursery and bespoke crates.
That has always been the biggest benefit of namespaces, when i type "std.rand" I never have to worry about an attacker squatting "std.rnd". But I could just mistype the namespace! Fine, I could, but all the obvious mistypings could _also_ be reserved!
It seems from your comment (and I may be wrong!) that you're purely talking about the "use namespace to solve squatting" issue, which is only one dimension here. Some people want namespaces and don't care about squatting. Some people want to solve squatting through other means. Some people do see namespaces as a solution to squatting, while others disagree.
The reasons may be unclear, but it seems likely that eventually the crates.io team will receive the same reasons that the other package managers got and implement namespaces.
This seems like human nature. We continue to do the same thing with client side / Server side rendering. When it comes to computing, we oscillate a lot. In all these cases there is the desired system (lack of namespaces in a way they doesn't absolutely kill you later), we pursue it, we inevitably fail, get burned, revert, forget, repeat.
In the case of namespaces, this is a tale told a thousand times. To the point where I think people don't even want to try to fight for them. What's the point? They shall forge forward, namespace free, until it hurts enough decision makers. Then we get namespaces.
Now, back to Ruby for me. Where anything anywhere can just yolo reach into your namespace and mess your whole life up.
This sentence takes the place of a reason. You could have offered a reason why you were right or a rational argument about the matter. Instead, you basically yelled "I'm right, they're wrong! I'm right, they're wrong!".
Let's let the future determine whether he was right or wrong, no need for squabbling.
It's not that evidence doesn't exist (it may or may not, and people may or may not agree that something is evidence), it's that none was provided. Without that, all you get is people yelling that they're right and someone else is wrong. Evidence lets people move the argument to useful territory.
Since not every package manager that came before it is consistent with the assertion, no.
If you'd like to: Go back and re-read my comment, but re-read it in the mindframe that I'm against the change. It still holds. I didn't provide reasons why they should make the change, because I don't have a stance on whether they should or shouldn't. I am not qualified to say whether its the "Right" decision; just that Rust is on the wrong side of history with this one.
A thousand times this.
Why are so many things in software three-steps-forward-two-steps-back?
This is a complete misrepresentation of the issue. Adding namespaces to crates.io is a social problem, not a technical one, because adding them means you now must do the work of arbitrating disputes over an identity layer, which is a job unfit for part-time volunteers, which is all that crates.io has. The only alternative to arbitrating your own identity layer is to tie yourself to some other preexisting identity layer (e.g. GitHub, DNS), which raises questions about the notion of project ownership, the potential transfer of ownership, the immutability of the package registry, and all the while just pushes the "race to register" to a different sphere.
If crates.io had full-time employees rather than only volunteers, I'd be all for paying someone to handle identity problems in exchange for the existence of namespacing. But who's going to pay for it?
This is why the conflation in the title of the OP here is especially annoying; package management is a concern that the language doesn't necessarily benefit from being aware of, hence the separation of the compiler and the package manager. :)
All I am trying to say is that this is a technical problem, in addition to the larger problems, which you correctly identified are social.
Nit: rustc gets passed each crate available to be used or extern crate via an --extern flag that specifies the name of the crate as well as the full path. The path is optional and if not specified rustc performs a search, but the name is mandatory. Cargo only uses the mode with both path and name specified. This also isn't an edition 2018 thing, it exists on older cargo versions too. It's possible to specify the same file under multiple names, then it's available to Rust programs under multiple names. It's also possible to specify multiple paths for the same name, but then only one will be taken, the first one which is available.
Also, note that Cargo works in terms of packages, and rustc in terms of crates. A package can contain up to one library crate, and the names of the two don't have to match.
That said, I'm having a hard time finding the Rust Internals discussion thread, so I can't tell if I'm misremembering somehow.
Edit: Actually, I think this was in context of re-assigning crate names that may have been "squatted", and how making decisions around what amounted to team member's arbitrary decisions around re-assignment could cause issues. Scratch this, please.
Devolving identity down to DNS has been a really good decision for Go. I'm not sure why everyone doesn't do it. It's like XML namespaces, but actually useful.
If it works for go, the because go is not old enough. Domain registrations lapse quickly and then someone else can buy them. That's very close to the worst that can happen for trust in packages.
Golang's package system depends on the current DNS records.
With maven, there are plenty of old projects that do not own the domain names for their group ID, and that doesn't cause any problems - they are well-known, established projects, and there's no reason to introduce breaking changes to everyone just because they want to go through some DNS naming changes.
Because of this, there's little reason to fear that if you publish your project today as "io.github.my-project:my-module" you may find yourself having to break all your users if github blocks your account tomorrow (or just changes their URL schema) and you move to gitlab.
But if you lose your domain spreading bad software is the top of my concerns.
How many domains lapse while the owner is active and responsive in the community?
OTOH, it's probably higher on the list of downstream code users’ lost of concerns.
> They can reset all email passwords and access every service you’ve ever used
Not if you dealt with all that before letting the domain lapse. And if you don't, more to the point, that's your problem, but not the package manager user communities problem. And that it is a bigger concern of yours than the impacts on the package manager user community is exactly why that community might want to protect themselves from you neglecting their concerns..
https://blog.golang.org/protobuf-apiv2
> The github.com/golang/protobuf module is APIv1.
> The google.golang.org/protobuf module is APIv2. We have taken advantage of the need to change the import path to switch to one that is not tied to a specific hosting provider.
Basically, Google realized that it's not a good idea to tie your package's identity to GitHub's URL scheme. This is easy for Google to fix, but the majority of Go packages suffer from the same problem, and it is much harder for a small project to start maintaining a DNS record that they control to point that to GitHub.
Not to mention, if GitHub decided to change its URL scheme for some reason, almost all Go builds on the planet would have to change.
And of course, the whole problem stems from relying on DNS to name your packages. No one wants to bother with DNS, so they go for the low effort option and let some hosting provider handle it for them, without caring what happens in 10 years time.
I don’t see how namespaces would be different?
To be honest, that's an impressive oratory monstrosity you've produced there. No argument, no nothing, and nevertheless it sounds completely brain-shutting-down conclusive.
I don't think the rust people are deserving of it. They've done impressive work with minimal resources, and they have shown the ability to create consensus that's better than any of the individual parts that contribute to it. So I'm going to trust them in this for now, and maybe, they'll evolve in the future.
Nevertheless, I have to admit respect for that sentence, in the same sense I'll admit respect for an earth-shattering nuclear explosion. You might want to run for US president if you produce more of them, though I'm not sure if that's praise or damnation.
One of these years I'll understand this place! I just know it!
There is, sort of! Assuming you choose a semi-reasonable timeframe (5? 10 years?), I’d be happy to take the other side on Longbets.
I believe there’s even a reasonable argument to be made that your bet is societally important, as the rules [1] require, based on your final paragraph. Roughly your take is that over time, efforts realize that the naïveté of youth misread the battle scars of what came before. crates.io having namespaces or not is just the mechanism for this debate :)
Since clojure uses maven as well, the same applies, but clojure tools like leiningen decided to create a shortcut: if the group and artifact name are the same, like `iglu/iglu`, they can be collapsed into one name: `iglu`
Well, that just encouraged everyone to choose collapsible names. In retrospect, this didn't buy us much. Who cares about saving a few characters of typing? Most now seem to agree it wasn't a good idea.
When the "collapsed" name falls out of maintenance, the forks will all seem somehow less "official", even if they are much higher quality. Forks are inevitable; why would you want to discourage them?
I finally decided to start using the reverse of my personal domain for my future libraries. The java folks were right all along.
The conclusion that we don't want to discourage forks may be valid, but this doesn't seem to be good reasoning. Lots of things are inevitable that we want to discourage or delay.
The epitome of this mindset I think is "hostile fork" -- the entire notion is nonsensical. The whole point of being FOSS/OSS is the freedom to fork -- by all damn rights, you should fork as you please, and be pleased to fork!
The actual problem is not forking.. its community fragmentation, and more importantly loss of a "source of truth". Of course, maintaining that source of truth is otherwise known as centralization, with all the problems that brings, but there's nothing inherently wrong with forking.. that's just the natural specialization and evolutionary processes at work.
The solution is to make it easier to find those top-tier libraries, and this is orthogonal to forking; mainly handled by blog posts and "official" library listings/recommendations, and things like This Week in Rust.... namespacing or not doesn't really get you anything there.
It is not. It is based in experience. See the xMule/aMule fork. A hostile fork is when the fork project starts bad-mouthing the original project and its maintainers.
The notion that forking is by itself hostile is non-sense.
I think it's in there somewhere that he compares the right to fork with the right to bear arms: Good to have, but the situation must have gotten really shitty if a fork is a good solution.
https://firstmonday.org/ojs/index.php/fm/article/download/14...
And also by the experience of going through GitHub hosting a RubyGems server and all the fallout that happened there.
1. It was introduced very late, meaning the community had already formed patterns of contribution around a flawed flat system. This is a problem of the flat system, not of the namespaced one.
2. It is still to this day entirely optional (for understandable backward compat. reasons). This gives namespaceless packages a misplaced position of authority over namespaced ones, which erodes the value of namespacing.
These are tough problems to get around if you start with a flat structure, but they really just outline the urgency of switching to namespaces for a relatively young project.
I wasn't arguing that npm's namespacing system is worse than their initial system, nor that their switch to namespacing was a mistake.
The current npm namespaced system, with flaws, is head-and-shoulders better than the previous flat system.
You're saying you did "look and learn". If by that you mean you looked at the end product (npm's is seriously flawed) without looking at the journey to that product (npm's is still a huge improvement over what they started with), then you're not going to learn much from that kind of "looking".
I highlighted Composer/Packagist in a sibling comment as a system you should look and learn from (w.r.t. namespaces).
Choosing to only look at flawed systems that started flat seems like you're just being selective to support your own thesis.
Not effortless, but not necessarily very costly either.
Don't those package registries all suffer from not having namespaces? RubyGems in particular [1].
[1] https://thehackernews.com/2020/04/rubygem-typosquatting-malw...
However, typosquatting is an orthogonal problem to namespacing, you can still typosquat a namespace.
This is not the first time typosquatting attacks of this kind have been uncovered.
Popular repository platforms such as Python Package Index (PyPi) and GitHub-owned Node.js package manager npm have emerged as effective attack vectors to distribute malware.
"Orthogonal" suggests no connection but what I see above is a list of package managers that don't have namespacing.
I stated my reasoning in my comment: you can typo squat a namespace, just as easily you can any identifier. I don't see any inherent difference between the two.
correlation does not equal causation.
how is it not apparent that typosquatting is possible regardless of whether namespacing is in play?
for example, URLs are namespaced, and are the classic example of typosquatting: https://en.wikipedia.org/wiki/Typosquatting
People can just have maliciously typeo-d namespaces
Actually you could argue it's worse since people tend to pay more attention to package names rather then namespace names
The new tools.deps in Clojure actually is moving to disallow collapsed names for similar reason and will force iglu/iglu.
Here's a rationale from them:
> The groupId exists to disambiguate library names so our ecosystem is not just a race to grab unqualified names. The naming convention in Maven for groupId is to follow Java’s package naming rules and start with a reversed domain name or a trademark, something you control. Maven itself does not enforce this but the Maven Central repository does all new projects.
> In cases where you have a lib with no domain name or trademark, you can use a third party source of identity (like github) in combination with an id you control on that site, so your lib id would be github-yourname/yourlib. Using a dashed name is preferred over a dotted name as that could imply a library owned by github.
That said, ya this is best intentions unfortunately. I'm guessing if you own a real trademark, you could actually sue people using your trademark as their group-id.
Otherwise in general they recommend using a registered web domain name. Someone else could take over your domain name as theirs, but I think the registry owner, like maven-central, if you contacted them and could show you own that domain, they might be able to take actions against the impersonator. Same for a github user.
Actually, thinking about this, I feel it be great if the repository owner like maven-central required a form of proof of ownership of the domain or the github id. That could add a lot of trust to the whole process.
They do. There's even a manual review procedure.
This appears to be using evidence to prove the opposite conclusion; if everyone voluntarily chose to use shorter names, then it means that everyone cares about having shorter names. If there is a more substantial argument for why people have decided that the collapsing was mistake, I'd like to read it.
Another point i didn't mention is that maven was designed from the start to be decentralized; many companies run their own private maven repos, but also pull artifacts from maven central. Having group names reduces the chances of collisions between their private servers and a public maven server.
Companies change names, they merge. Sometimes they go out of business but stick around as a foundation stewarding their old projects, and you might be going to example.org for years for documentation on a com.example module.
And the namespaces weren't enforced (who is going to stop me from publishing a com.example.foo module?), so it expected much and delivered little.
No namespaces is bad. Five level namespaces are better, but still bad for different reasons. Two might be good. Some might prefer three. But zero is right out.
On the other hand I really like how C# and dotnet in general handle the matter. Package namespace are separated from logical (in-code) namespace. Package namespace are usually two/three dotted term, making ownership clear while not bloating the names.
Well, that's more a bad thing about the DNS though. "toplevel.domainname.subdomain/path" should really be how it should be structured. SUN improved this and made the hierarchy proper.
I recall an interesting precursor in Solaris/ System V - the OS package guidelines recommended using your _stock ticker_ as part of the short name.
(For the record, I do think that namespaces on crates.io would be a marginal improvement, but the benefits are often overstated and it would add organizational costs that the crates.io maintainers can't currently bear. The amount of ire over this topic is disproportionate to its impact.)
Hard agree but this has nothing to do with namespaces. This is equally applicable to names in a flat registry. I don't see the point of bringing this up in this article.
> there are also real benefits. [...] discourages forks
What. How? Why?
I guess there's possibly some valid argument for disliking forks (though it's certainly not put forward in this article), but discouraging them via package registry structure is... delusional.
Forks are a reality of software and they neither happen as a result of registry structures, nor can the be discouraged by their designs. They happen for entirely external, human reasons, and a well-designed registry structure (i.e. a non-flat one) will be set up to be resilient to forking.
Identity
Continuity
Stability
Squatting
None of these are features, which are incompatible with namespaces. I can't find a compelling reason that Crate works this way in this entire article. The article reads like a list of "why Crate is poorly implemented". Of course, it works when the community is rather young and small. Give it time.
Your... first name? So "rene" (if I was to assume a few things about your username)?
I mean, it's probably not a big deal in this case. But if I had a github account that someone wanted and github just gave out my (personal?) info, I might feel a bit betrayed by them. I know, I shouldn't give them any info I really want private, but that doesn't mean I can't expect them to act responsibly with what I have given them. I just didn't want to state all that up front, because it assumes a lot about what happened when there wasn't a lot of detail provided.
It is nice when people can just make simple stuff happen though, I agree. It's just a hard line to walk, because you allow that and then you get stuff like the recent Twitter hack. It's hard for us to all have nice things and nice people that can help us because there really are a bunch of people out there to get you (or someone in a similar situation, at least). :/
Of course, if they just let you login to the other one that would be crazy business. You'd know the other guy's email for starters. But they don't.
It's all perfectly kosher. You just get the namespace.
That person is definitely not helping the perception that a significant fraction of the cryptofinance community is scammers, and that their presence has a negative effect on the community.
That certainly puts his participation in the namespacing debates into a new light.
They're not out there hawking popular names for sale and some of their placeholders are even "namespaced" (the abscissa or signatory subcomponents).
There's nothing you can do to prevent this except devise a system where it's inconsequential to register all of these names: namespaces.
If it bothers you that one person can do this with many packages, then presumably it bothers you that 100,000 people can do it benignly with a few package names which is what goes on in npm/ruby/cpan. The solution is to simply making them worthless.
Subreddits have the same problem.
The case above is clear but a whole load of not so clear cases would follow and then there would be a need for an appeals process.
My guess is that the developers have better things to do.
Instead of just coming up with a good name for your project, you have to come up with a unique one even though the project with that name isn't even active. And since most names are taken, you have to come up with a clever one for no real reason. Or go down the route of naming strats you see on npm/gems: better-lib, lib2, lib2-better, lib3, better-lib3, etc.
It really does nobody good except for the person who was able to clutch the original name, and that's a weird incentive structure. Subreddits have this same problem where you're instantly gimped when named anything other than r/bitcoin, r/politics, r/news, r/atlanta, etc.
Instead, there should be no preciousness to names nor dictionary words. That's the correct incentive structure: none.
People keep linking https://crates.io/users/swmon and it probably seems like an easy fix to just revoke those names. But swmon isn't the problem, the system is. It's no bigger problem when swmon does it overtly vs thousands of people doing it incidentally when they register a package they don't do anything with.
The problem isn't to calibrate a system that can detect swmon but build a system where you don't care what swmon does.
What you want is something like this: https://package.elm-lang.org/
Watching Crates get this wrong was like watching Swift not implement qualified imports.
Bugsplaining?
Do you think Sam is being ironic with that title, or sincere? His intro is telling us how he's going to calmly explain to us dumb hicks and newbies that we wouldn't know a feature if it bit us on the ass (which it most definitely will, at some point).
The rest of the article doesn't (quite) follow up on that warning, but if you start out by putting everyone on guard nobody is going to listen to you.
The rest of the article follows through on the intro's promise.
"You don't actually have that problem. The system doing that is expressly a feature for you."
Not sure about recognition, or the vaguely defined requirement to avoid Poe's Law.
The rest of Rust is extremely pleasant to work with; a lot of thought has been put into ergonomics and I love it as a result.
But with crates.io it feels like the opposite. I have to deal with name-squatting and a much more difficult naming process for crates as pretty much every short, meaningful name in English is taken at this point. For what gains? Less forks, less fracturing -in theory-?
I feel like a lot of the arguments for no namespaces are strawmans or make big stretches. For example, figuring out which fork of a project to use is really not a problem. And a lack of namespaces does not fix that anyways, because it's not a code problem... Importing two crates with the same name would be fairly rare, and rust easily solves that with `use crate::... as ...`. Etc.
Anyway, this discussion has been trod to death. There's no point having it, as I'll just be told why I'm wrong and why namespaces are just -impossible-.
As mentioned elsewhere in here, this is a misrepresentation of the issue. The problem is that crates.io is a free, volunteer-run project with zero full-time employees who could be tasked with the drudgery of intervening in naming disputes or managing an identity layer. This is unlike GitHub and unlike "Java" (by which you probably mean Maven Central, which is operated by Sonatype). Solve the funding issue first, and then you can start solving the rest.
What do these have to do with the issue of namespaces?
Having namespaces would lessen naming disputes since users would be able to name their crate whatever. Any system will have disputes, so I'm not sure what the argument is here.
I understand why crates uses github for identity management, but it's not ideal that people are locked into using one specific git platform. Github is already an unavoidable entity, much like facebook, but this is a bit of a harder issue to solve without a lot more resources.
That's why there are proposals that delegate the naming to outside systems with their own established dispute resolution rules.
This is due to the crates team requirement of "namespaces are not allowed to create additional work for the team".
So any proposal that makes them the naming authority is out from the get-go.
Sell namespaces? Or verified namespaces I guess? Checkmark and all.
> I have to deal with name-squatting and a much more difficult naming process for crates as pretty much every short, meaningful name in English is taken at this point
Squatting is not a major problem just name your package yourname-xml or randomname instead of yourname.xml
The obvious one is people can put packages under their own namespace and not worry about naming collisions. If this were really such an issue, things like github,maven, or just about any other library manager would be impossible to work with. But that isn't the case.
"The cost is that package paths are longer and hard to remember. Also confusing because multiple packages can have the same name"
In 5+ years of programming I have not once had an issue due to namespaces. In both systems you still need to remember the author + repo name. Both systems would have multiple competing projects for a problem space. Etc. It's up to you (the engineer) to do your research before adding a library to your project. Namespaces or the lack of them will never fix that.
Name length? Really? Is ~5-15 chars vs ~20-25 so important? It's not like we're memorizing these things. This is not worth removing namespaces for.
Both systems have issues due to things being named the same. One example in rust is `http` and its related types, which get referred to by multiple crates. I ran into that recently when I pulled in various networking crates into the same project. Removing namespaces doesn't fix this.
"Squatting is not a major problem just name your package yourname-xml or randomname instead of yourname.xml"
I'm sorry but I cannot agree with this. Naming should not be a compromise. There's a reason we don't use random names or just random numbers.
In my experience, the thorny problem with namespaces isn't bad actors because you can fix that with content moderation. Yes, that takes time and effort which can be scarce in an open-source community, but it's doable.
No, the thorny problem is good actors with bad execution: people who publish a package with a nice name that does what it says it does, more or less, but does it poorly. It doesn't cover all the edges case or it is poorly optimized or it has limited features or it has some fatal design flaw in its architecture... Maybe it was fine at some point but just hasn't kept up with the times.
So what do you do if a package like that grabs the name "json" or "http" or something? You don't want to kick packages off just for not being great, but you also don't want it to seem like the de facto package for a task in your language. I do a lot of Haskell, and it's not the best developer experience when you have to tell people "don't use json, use aeson instead".
Namespaces would solve this problem. You could have a "blessed" namespace (like rust/) and, when functionality becomes important enough that the quality of the "default" package matters use the namespace to provide that package (rust/json, rust/http... etc). You don't have to kick off the existing package (even if it's squatting a "nice" namespace like json/json) but you also don't get into the point where packages with nice names look fine but are actually flawed and a bad choice for real work.
If crates wants a policy that stores crates forever then just store crates forever. What I'm talking about just affects how new crates and new versions of existing crates get "uploaded".
Imagine you call your crate foo.com/bar. In order to push your crate the crates server needs to verify that you actually own the foo.com/bar URL.
There are many ways to do that. A simple one is to just have the crates server pull code from that URL instead of pushing it. If you want to push (e.g. to work around reachability issues such as firewalls) it's just the matter of devising one or more ways to prove that you "own" that handle. For example by serving some challenge response in a html meta tag
All of the existing services it would make sense to delegate to are more mutable than we wanted for our system.
What I wanted to highlight was:
1. I like a package repository that has "namespaces". In fact that's what I like about Go approach to package repository from day zero, despite other rough edges.
2. Classic approach to "namespacing" like maven's is susceptible to name squatting or just "split authority": who can claim the "org.apache" org? First comes, first served? If that doesn't match who owns the apache.org "brand" on the wider internet, it's not technically a problem of the package repository but surely it's a practical one.
3. Name squatting problems exist for DNS too; it's a know problem, with an existing administration and dispute resolution body, a marketplace and whatnot. If you cannot get hold of foo.com you'll settle on getfoo.com or (some abomination like that) and that will be your brand as far as online presence is concerned. It's only natural for artifacts you want to associate to that owner to just piggy back on that more prominent identity rather than have to fight it independently for getfoo.com and perhaps land on gotfoo.com because getfoo.com has been taken by somebody else.
4. Now, if you decided that you want namespacing and given the observed consequences of allowing your users to compete for namespace names independently of their main online presence, I claim there is a technical approach that can achieve your stated goal of reproducible builds. The goproxy is almost what you want. The main different is that the go team aims at a best effort service and thus want to call it a caching mirror. They could have chosen to offer a more strict retention guarantee. Go packages are indeed immutable. The sumdb transparent log is there to enforce that invariant. It's really just about somebody willing to take the burden of storing the package bits forever, which is what I understood you choose to do with crates.
(I do remember maven before that check was implemented and it was chaotic)
I'd like to understand more about the arguments for and against a URL-based module naming / resolution approach. Could you point me to any past discussions on this for further reading?
I am not 100% sure exactly what you're asking for, so if you could say a teeny bit more than that, that would help me understand what part of the space you're trying to explore. It would also be handy to know what problem you are trying to solve, there are several here.
I think the way Go and ES Modules (Deno / browsers) represent module naming and resolution by URLs has some nice benefits:
* decentralized module hosting
* could be extended beyond HTTP / Git (ie. import modules via "ipfs://")
* re-uses existing web infrastructure for namespace ownership (ie. via DNS)
* possibility of hosting backend and frontend modules for multiple languages in a single HTTP registry
* modules may become unresolvable from their original URL (link rot)
* cannot guarantee immutability at the original URL
* semver support becomes a concern of the registry, and some registries may not support it (ie. the logic to resolve "https://example.com/my-module/1.x.x/lib.rs")
* more difficult to discover (browse and search) all available modules
* proxies and dep lockfiles of file hashes to help prevent link rot and immutability
* possibility for open source registry implementations with built-in semver support
* central index of modules which can be searched / browsed, if modules are submitted to it
I'd be very interested in learning more about the costs / benefits of the various approaches and the Rust team's discussions on modules. I really appreciate the team's thorough approach to design decisions, as it's manifested into a really cleanly-designed language.
Thanks!
1. the only part of this that's been in recent discussions is using the URL as a namespace. This has led to questions around how those are represented in code, given that URLs are not identifiers. Also, using DNS for namespaces is mutable and costs money, this is a huge barrier to adoption that we did not want. Publishing should be free and easy.
2. I think you've already identified some of the tradeoffs that made this be rejected: for reproducible build reasons, referring to external services is considered unacceptable. This is due to stuff like downtime of services we don't control, but also the mutability of those services. In theory, you could proxy them, but it's not as clear what this actually buys you, because to get those advantages back, you'd want to always refer to the proxy, and now you've re-centralized everything.
Another way to look at this is that all of the solutions you've proposed are significantly more complicated than what we've built. That complexity is for good reasons, but given 2, it's not clear that they're good enough to justify the costs. The current situation is significantly simpler for all parties, and the end experience ends up the same anyway.
I hope that helps!
So what would Rusts equivalent be... Rustic? Namespaces are not Rustic.
why do you think that?
Why does everybody always have to reinvent the wheel? DNS has been around for decades. Using DNS records to verify ownership has been around since, what, the early days of the ACME protocol in 2014/2015? Earlier?
If you only develop one or two packages, you don't really care about the squatting issue. Especially if a non-DNS-namespace is free. Light users are generally unaffected by the problem, so they continue to fit within a "free" offering.
Squatting is a non-issue with DNS. Nobody can take your domain from you; effectively, using DNS outsources the control issue to DNS control. The only issue is to guarantee consumers that their builds won't break just because their packages went through forced deprecation when DNS control transferred. The people who suffer from the issue badly enough to need DNS namespacing are either heavy users (who are therefore much more likely to pay) or corporate entities that actually desire a paid option.
> Squatting is a non-issue with DNS.
We must be living on a different web :)
dependencies: regex
Vs
dependencies: com.cooladtech/regex
But the world doesn't let us have it elegantly, because the library with the name "regex" will almost never be the regex library you want to use. It will almost never happen to be the best, most maintained, best documented, highest performing, with all the regex features you need library.
So instead, the regex library you want will be called "potato". Like a hot potato, potato doesn't let you down when parsing regex.
And then there's the issue of groups of packages that generally make sense together, and no good way to define those groups. So you might have: regex_core regex_engine and regex_api, but it turns out regex_api is for another package and not really part of the group.
Anyhow, back to the article...
Identity? So I read and re-read the section on identity, and I'm not sure what they're saying, it seems they're saying that not having namespaces makes identity worse for discoverability? So basically it argues for namespaces? Now I'm confused...
> A flat registry makes identity management (naming a crate) harder
Than for continuity, it seems to say that if you have a namespace it's bad, because you could confuse two packages of the same name, but in different namespaces?
Okay, I did see this sometimes, mostly in tutorials where they don't list their dependencies, but only their import statements (and sometimes they don't even list the imports). So the reader is kind of left figuring which "regex" lib they are supposed to choose since it exists in many places.
But then the post also says that the lib name can differ from the package.name already? Which means you can already have the same lib name for two different packages in Crates? So I'm confused again?
> Updating to a new package version with different content under different ownership is a real security risk
I have no idea how this would happen unless I manually changed the dependency to point to the one from somewhere else? So I really don't get the risk here.
> Doubly so if you don’t ban new minor versions on the last major version after an unintentional ownership change
Okay, hold on a second. I don't think this article is debating what I think having namespaces mean. Here it seems like the article imagined a world where you depend on the package name and do not specify a namespace, and then in crates.io someone could transfer the ownership of the package to someone else. Or something like that? This makes zero sense to me, I need someone to explain it to me.
> This is a strike against schemes that encourage multiple distinct crates to have the same default name in code. I don’t think any proposal that encourages this could be approved.
Okay, now we're talking. This is a GREAT point. Obviously, the import statement must include the namespace as well. Otherwise you can depend on two packages and have import conflicts where who knows which module you're getting? But is this about namespaces? Maybe the argument here is that the way Rust work, this would not be the case, thus risky? If true, I think this is a defeating argument honestly, something has to be done about this. Also it sounds like this is already a problem, because lib name can differ from package name?
Anyhow, I'd direct people to this amazing post from the Go world, and I think they got everything right when it comes to their view of package management: https://research.swtch.com/vgo-principles
I prefer the Go approach which lets you host or import packages via a HTTP or git URI. If you have to have packaging built-in to the standard toolchain these days, then please use already well-known and existing protocols like DNS and git. Java also figured this out years ago (com.sun... etc).
That said, an alternative registry that enforces namespacing would not work, just because of cargo's limitations on package names that prevent characters like /
It could be done by making every "namespace" a different registry, and for those registries to allow packages with dependencies from other registries (something that crates.io doesn't allow, for a reasonably good reason). Would be quite verbose for the user though.
For the most part, unless a lawyer says that it needs to be taken down, things aren't taken down. There is a small amount of wiggle room around registering a large number of packages in an automated fashion, but generally speaking, the principle is non-intervention.
Alternative to just removing them, would the crates team be more open to transferring ownership of these packages?
Plus, that's only 100 packages. There are people who legitimately maintain more crates than that.
> would the crates team be more open to transferring ownership of these packages?
fwiw I also don't think namespacing necessarily solves this problem, it just shoves the squatting problem up a level.
In fact this scheme is already supported with cargo since you can specify the registry per dep. This means the default registry, if nothing is added is 'crates.io/' but you can specify any other registry, eg 'gitlab.local/looks/like/namespace' for personal CI, or even 'crates.io/{username}'
Let's give each user their own registry at 'crates.io/{username}'. This works in a backwards compatible way cause registry:'crates.io/' + crate:'mypackage' is different than registry:'crates.io/mypackage' + crate:'mypackage', so nothing changes with the current crates and it's all an incremental implementation detail for crates.io
Now we have the npm user style namespace ids that lessens the namesquatting problem, but increases the security issues if the registry changes hands. Honestly, I've never really understood the security argument here cause there's no reason an owner can't hand over a single package. As well, there's no reason a currently trusted owner can't patch update a package with bad code. This is a social problem that has no solution and is the same for both naming systems, so we need to come up with another way of handling things. Personally I think all published crates should be immutable and all deps should be pegged, not even allowing patches. But this leads to the argument that sometimes publishers really do want to take down a crate. I'm on the fence with this one, so let's try to solve it in another way.
So the problem is we need to potentially remap any (transitive) dependency to something else, and since we still have a flat hierarchy we really just need to remap the registry for that dependency. So let's add a registry remapping section for the cargo deps ... wait cargo already has something for that (https://doc.rust-lang.org/cargo/reference/source-replacement...). (Reading through the docs here, it does seems like it might be a little strict on whether we can use it for this case, but it's pretty close.)
OK wrapping up, it looks like we could have backwards compatible namespaces right now if crates.io would just implement user specific registries, ie it's an implementation detail. And the security issues are really no different than before. Thus I've got to ask, what am I missing?