Russ Cox on Go dependency management
threadreaderapp.com
threadreaderapp.com
Sure, some communities (e.g. the Rust one) show that a sole person need not reside at the top. But that often comes with slow implementation/decision costs.
And this is exactly why the BDFL model is not sustainable in a longer term. I do agree that a consistent tie-breaker (a role of GvR, a stereotypical BDFL, was indeed much like this) is good to have, but a BDFL eventually becomes a venerable monarch whose everything become controversial enough to make one a political entity regardless of one's intention. As GvR shows this also puts a lot of burden to a single person.
It also comes with feature bloat.
Same with green threads which at one point they were hell bent on adding and then one fine day decided to dump. Also leading to ban someone in Rust community who was against green thread.
I think what people do not want to talk about or do not get is unlike Oracle, Google, MS or Apple Mozilla can't just add few dozen highly paid people for language development. So this big community led approach is more of practical matter than some enlightened way to develop software.
But with green threads, Rust had them for quite a while, on the order of years. I think the person you're talking about was in favor of the removal of green threads, which we were initially against, but it did end up happening. That person also left Rust but wasn't banned; we've only ever banned one person, and that was fairly recently, and it was for an extended harassment campaign, not over some sort of technical disagreement.
> So this big community led approach is more of practical matter than some enlightened way to develop software.
We believe that it produces better technical outcomes. Rust has more paid developers than many, many languages. Regardless, when making decisions, it's important to be informed, and there's no better way to be informed than to get a broad description of the problem space and other approaches to solving them. One person is never going to be able to be an expert in everything that a language needs.
There's also sustainability concerns with BDFLs, but that's a larger topic.
> And even in Rust earlier module system written by someone else and was dumped wholesale without enough justification to author who finally left frustrated.
I don't know what module system you're talking about. The first Cargo was unmaintained for a long time before being removed; the original author was long gone. The second rustpkg was largely Graydon's work, not "someone else".
> Same with green threads which at one point they were hell bent on adding and then one fine day decided to dump. Also leading to ban someone in Rust community who was against green thread.
We dumped them in no small part because the community didn't want them! And nobody was banned.
> I think what people do not want to talk about or do not get is unlike Oracle, Google, MS or Apple Mozilla can't just add few dozen highly paid people for language development. So this big community led approach is more of practical matter than some enlightened way to develop software.
In other words, you're saying we treat the community as just free labor. That's an incredibly cynical accusation, and one that bears no resemblance to reality.
I'm pretty confident in predicting that if Rust had adopted a BDFL style of governance, it would have been dead long ago. Community input was critical in decisions like removal of green threads and I/O reform that have helped Rust find its niche.
Additionally, Rust development frankly moves faster than any other language that I know of. You need only look at the release notes for that. The reason is that people can truly be invested in the project, able to make technical decisions as well as contribute code.
I think the ideal situation is when the community develops a solution that works well for them and then leadership steps in to shepherd it through the integration process. Leadership isn't there to force their ideas onto the community. When done properly, everyone is empowered and nobody comes out of the process frustrated. This is something we have had to learn over time, but I truly think it is the right way to develop open projects like languages.
There needs to be some thought leadership and gatekeeping, otherwise you get evaporative cooling, chaos, and devolution to the Least Common Denominator. All "leaderless movements" are subject to this.
https://www.lesswrong.com/posts/ZQG9cwKbct2LtmL3p/evaporativ...
The problem with BDFLs, is that when they're wrong, there is sometimes no pressure relief valve.
Sure, some communities (e.g. the Rust one) show that a sole person need not reside at the top. But that often comes with slow implementation/decision costs.
What you're describing is a republic. There is no dictator. There is no absolute ruler. However, there are democratically chosen leaders who need to make decisions that won't make everyone happy. Most importantly, there is no tyranny of the majority.
At this point it would be great to no longer be talking about whether go modules should be the thing but working together towards making them work for everyone. (which I don't think is far off actually, Russ and co have done a good job post-reveal of addressing use cases that break with modules)
One thing that I find extremely troubling about Go's dependency handling though (and which at least partly contributes to people's frustration I think) is the mixing of the concepts of a namespace and a resource identifier in Go's package names. It's absolutely reasonable that package names should be unique, making the package name double as a storage location is not always a good idea though (IMHO): For example, we recently switched from using gitlab.com to our own private Gitlab instance, which runs under a custom (non-publicly resolvable) domain. Since all our packages were named like "gitlab.com/.../..." (to make them "go get"-table) we would have to rename all of them to make them "go get"-table from the new location. In the end we solved it by defining a git URL rewriting rule that rewrites the old to the new URL when fetching the repository. This ensured that we could work primarily on our new Gitlab server while maintaining the possiblity to build packages on the gitlab.com server. While that works I think it would have been a better idea to keep package names and storage locations separate, or at least give users a way to override/modify the resolution of packages into locations using a more standardized approach: Such a mechanism could then allow for different name resolution strategies (like a package file, a list of standard repositories) while falling back to the standard way of resolving packages into URLs for "go get". Granted, this would add another layer of complexity, but as we already need a separate file for proper vendoring (to e.g. specify versions) it wouldn't hurt too much to also put (optional) information about storage locations into that file.
Just my 2 cents, apart from this minor inconvenience I'm absolutely happy with Go and its ecosystem!
https://golang.org/cmd/go/#hdr-Remote_import_paths (read until the end)
or TL;DR: https://blog.bramp.net/post/2017/10/02/vanity-go-import-path...
If I have a canonical import path of "mydomain.io/foo", and I host html there pointing to github or whatever, I now have a different single point of failure: the domain. If I lose control of the domain or the
Most package management systems have one additional layer of resolution: a centralized pluggable repository location which can be configured to some default, and then lookups are made against that instead of against a url specified in the source code.
For example, with cargo, I can open my ".cargo/config" file and specify "registry.index = whatever-url.com", and then for package "foo" it'll look it up on that registry.
If I forget to renew the domain, I don't have to change a bunch of import comments in my source code to update the registry location; I have to update one cargo config file.
> Most package management systems have one additional layer of resolution: a centralized pluggable repository location which can be configured to some default, and then lookups are made against that instead of against a url specified in the source code.
TBH, these two seems to contradict each other. You don't want to use your own domain, because that's a SPOF that you could lose control over. Whereas the centralized repository is a SPOF that you never had any control over to begin with. Seems strictly worse to me. And the nice thing about the chosen approach is that it can coexist with what you seem to be wanting: gopkg.in exists and is exactly that. A centralized, hosted (and free!) registry for go packages with its own registration and authorization process.
What I find so beautiful about the go-get solution is a) that you can build any other mechanism I know of on top of it - i.e. the user can decide on their own tradeoffs of convenience/scalability/security/future-proofedness… and b) that domain names already provide an elaborate, globally agreed upon namespace that can separate ownership from administration and hosting, with its own takedown-semantics… It's decentralized, existing, robust infrastructure. It gives full control to the user. And it solves a whole bunch of technical challenges already, for free. It means the Go team doesn't have to manage user accounts of package maintainers and it doesn't have to have policies of what code to accept or not accept - as it can't enforce them.
It's really quite brilliant, IMO.
I can use "foo.registry.com" with no changes to my rust source code.
If I want to switch go from "old.com/x/package" to "foo.com/x/package" I have to rewrite all the source code first.
That's the weird coupling which makes go not so configurable; code has no place dictating where it's downloaded from, yet go packages do that. I can't use the go package management ecosystem if my canonical import path doesn't resolve (e.g. dep won't work, go get won't work, vgo won't work).
I never have to worry about my rust code having a magic comment in it which makes the package manager incapable of fetching it from some arbitrary differently named mirror. That's simply not possible in a well designed package management system.
git config --global url.https://our-new-domain.com/.insteadOf https://gitlab.com/
git config --global url.https://gitlab-ci-token:${CI_JOB_TOKEN}@${CI_HOSTNAME}/.insteadOf https://gitlab.com/
I vastly prefer the Rust way of doing things though, as I don't think using domains as a namespace is a good way. For example, most code hosting sites will shut down or be deprecated eventually (possible even Github at some point), which for Go will lead to a lot of headaches since there's no way to know where a given package will end up when it's no longer available at its original location.
Apparently it's a matter of opinion, but I consider that a feature, not a bug. It means the import path, as the identifier of the package, identifies the package. If two source files refer to the same name, they mean the same code.
> code has no place dictating where it's downloaded from, yet go packages do that.
That is still not actually correct. The webserver listening on the domain is dictating where the code is downloaded from. i.e. the owner of the identifier of the code you are downloading.
The basic assumption is, that if you take some code and put it somewhere else, you are assuming ownership, so you should actually own the identifier of that code. If someone wants Library X, they should be sure to actually get Library X, not your fork Y.
> I can't use the go package management ecosystem if my canonical import path doesn't resolve (e.g. dep won't work, go get won't work, vgo won't work).
To me, this seems to be a weird and unfair comparison.
gopkg.in exists. There is nothing in the Go ecosystem that is preventing what you want to have. The equivalent to Rust's cargo+crates.io isn't go-get+custom-domain, it's go-get+registry. And the equivalent to the import path "github.com/user/package" isn't (import of the "package" crate plus a mirror-directive in your cargo config), it's just the string "package". Because that's the identifier of the crate.
You could make the argument that Go still doesn't allow you to have mirrors, which is fair on the one hand, but will be solved with modules on the other.
Really, IMO it's just a fallacy to think of the import path as a download location. It's an identifier and the only reason the domain must resolve, is because go-get must consult the owner of that identifier what code to fetch. And DNS is the canonical tool to do that. Just like you need to resolve a hostname to get a Let's Encrypt certificate or use the Google Webmaster tools - you have to prove ownership of the name.
But I think on the plus side it was very easy for you to reason about the tooling to get your simple solution going.
Thinking more of: It is great that you can kind of apply “Dependency inversion of the tooling” without jumping to too many hoops to solve your problem.
Then why didn't you? Go had that ability since the inception of go-get. You could have used your own domain to name your packages from the beginning and have avoided that.
Mealy mouthed walls-of-texts are just as unhelpful as one sentence dismissals. Sooner or later, the silent miscommunication will manifest itself and impact the product directly.
Needless to say, the health of a project relies upon the health of its politics.
I don’t know if we can avoid this. What I think we can avoid are blow-ups, or at the least calm them back down again. We have to use our hearts, not just our heads, to say and mean, ‘yes, I acknowledge your feelings, which are valid.’ In the case of Go & dep, I think Sam Boyer feels let down because he’d been so proud of the idea that his tool dep would become the official tool. Rationally he acknowledged that it was the official experiment (maybe he even acknowledged that was an official experiment — I forget), but emotionally I believe that he thought that it’d become the official tool. To see it replaced is a colossal letdown.
We can honour him and his investment of time & energy while at the same time acknowledging that the tool he created isn’t the right tool long-term. This is something the net is terrible for: it’d be far better to break the news, in person, over dinner or drinks. There’s no good way to do it, of course, but sharing physical space helps.
It's easy to end up in a position where you're making that mistake if you're a small team used to shipping smaller products or features. Suddenly you're on a bigger project, maybe a bunch of the developers work at another company, maybe you've got a few million users instead of a smaller set like a hundred thousand, and the same planning and team collaboration techniques you've used are subtly harmful and produce bad outcomes a year or two down the line. I observed the WebAssembly design process suffering from same of the same problems early on because it was a group of engineers from multiple companies who were generally used to just working in tiny silos on experiments or features of their own, not A Collaborative Effort For Millions Of Users, much like what Go has turned into now that it's scaled out and not just a prototype language made for Google internal use.
Or just ignore it and apologize afterwards for rendering the bike shedding useless.
Meanwhile millions of people use it for 10 or so years for all kinds of deployments.
If only all languages had that unimportant problems.
This seems to be a major sticking point but I'm having trouble understanding why. I'm not familiar with any other package management system which supports simultaneous major versions in a single project. If two things require different versions of a package, those seem like separate projects to me. Is this to support dependencies which rely on the same package at different versions? If so, why not just call it a version conflict and fail until it's resolved?
Because its unnecessary and it doesn't scale. In Rust, versions become part of the symbol name, and the whole thing just becomes a non-issue. Go did the right thing here.
You may be thinking of packages on the scale of frameworks. For example, if you have a Ruby on Rails app that mixes packages from Rails 3 and Rails 5, well sure, then you're in for some serious trouble.
But a lot of packages are much more narrow in scope. Rust has a lot of small crates for basic data structures that have their APIs change betweeen versions. So you could end up with your logging package internally depending on some_data_structure=1.2, while your RabbitMQ client library internally depends on some_data_structure=2.3. As long as they do not expose some_data_structure in their API, you will never even notice (except maybe through increased binary size).
Yes.
> If two things require different versions of a package, those seem like separate projects to me
Basically we’re talking about libs that have version conflicts, but also very slow release schedules, such that the version conflict isn’t going to be reconciled any time soon; or where one of the deps may even be effectively abandonware, but “works just fine” stand-alone so nobody’s going to update it. It just depends on extremely old versions of common shared deps. These are, effectively, irreconcilable version conflicts.
The only thing to do, in most languages, is to fork one or the other dep to fix the conflict; or to split the project into microservices such that the conflicting components can live on opposite sides of a process memory boundary. Even these are sometimes impossible for office-political reasons.
> I'm not familiar with any other package management system which supports simultaneous major versions in a single project.
Node’s npm. Versions aren’t globally resolved; instead, each package effectively gets its own version resolution. Your root-level package gets a node_modules/ directory with your direct dependencies checked out into it; but your deps' deps are checked out into further node_modules/ directories that exist within their parent dep's checkout directory. It's somewhat as if each parent dep had vendored its deps, and had a vendor/ subdirectory; except that, in this case, the vendoring process is recursive.
When a given package require()s a library by name, then, it’s actually importing its own npm-vendorified copy of the library from its own node_modules/ directory, rather than looking in your package's root node_modules/ directory. (In fact, there is no root level; you have "package isotropy", where a dep NPM checked out has the exact same NPM-populated subdirs as a project you git clone'd yourself.) Since these libs aren’t entered into a global namespace by require(), rather just have the module returned as a runtime object for the parent scope to bind to a local variable, there are no runtime conflicts between the separately-vendored versions, either.
In a language without such runtime support, though, you would need your package-ecosystem tooling to reach into your downloaded lib and do some name-mangling of the exported symbols, and then either create a router symbol that resolves calls based on the lexical scope, or further mangle your deps' parents to call the dep by its mangled name rather than its original name. I think that’s what’s being proposed here, since Go has no such runtime support. (Effectively, in Go terms, you can simply think of it as all the references to git-http URL depspecs in import statements, being amended to qualify the repo name with the full git SHA, such that two different commits of the same project act as if they were two differently-named projects from Go's perspective.)
http://npm.github.io/how-npm-works-docs/npm3/how-npm3-works....
Yes, that's right. See for example NPM's docs on how this works: http://npm.github.io/how-npm-works-docs/npm3/how-npm3-works....
NPM notably doesn't have any problems with different versions (afaik), since imported modules are simply variables in the importing ones, and not in a global namespace. I think it's also possible to implement that in Python due to similar semantics, though it's not currently available.
Because resolving it, in general, doesn't scale. If the dependency graph is
A -> B -> D
A -> C -> D
B and C have to coordinate their upgrade to Dv2, so as to not break A. But B and C might not know of each other and A might not even know about their dependencies on D. Worse, C might not be able or willing to upgrade to Dv2 in the foreseeable future, for lack of bandwidth, so B is kept from upgrading to Dv2 or has to break A.
If you allow both Dv1 and Dv2 to coexist, B can upgrade to Dv2, A stays unbroken and at some point, C can upgrade to Dv2 too. No need to coordinate the timing or undue rush.
Now multiply that graph with a hundred dependencies and a thousand projects A, and it becomes clear that avoiding the need of coordinated upgrades is a good thing.
I presume that the language spec itself hasn't change DRAMATICALLY over the past few years. But so much is unfamiliar in the project structure / build process area (e.g. "vendor directories", modules, dependency managament, etc).
All the tours and tutorials focus on the language syntax and standard library. Looking for a refresher or intro to the "how do you actually get work done in the best manner?" plumbing aspects.
Web glue frequently recommended around here:
Github pages like https://github.com/avelino/awesome-go have become a good starting point for finding interesting projects.
Nothing that affects usual daily work has changed and that's what I insanely love in Go. Recently I had to pick up and work on project I did back in 2013, and it was so damn cool to see that it just works, compiles and there is almost zero problems with dependencies (the only one I had was due to removal of `code.google.com` and moving projects to github, so it was a matter of renaming import path and introducing vendoring).
https://golang.org/project/#go1
As to the recommendations to learn dep, personally I didn't (currently we use an inhouse tool at my workplace, and I expect we'll transition to the new go modules when they will be released as part of Go 1.11). The "modules" are actually not there yet, will officially land as a prototype feature in 1.11, feature-gated and with the old GOPATH way still working.
Yes - Russ should have been much clearer much earlier and not let dep get as far along as it did. But dep mentions the issues, particularly for small teams with go's new approach - I'd love to get a quick version of that.
Just suck it up.
You went off and did your own thing, and ignored everybody else, sticking pretty much to what you wanted to win the race in the first place (gopkg).
Right, now thats out the way can we just get on with it and use modules?
Decisions have been made, they’re here to stay. It doesn’t make everyone happy... but hey, we’re all massively relieved the mess has been sorted out, so thank you.
Lets stop these deep soul searching what he said she said wastes of time and get get started using it~
If one were to start a serious project with Go in 2018, what's the recommended way? Are people using Go modules? How to deal with 3rd party deps which dont have go.mod files?
When I previously explored Go, having all code under GOPATH was recommended, but seems like things have changed with `vgo`.
Even worse, I could not find a single link back to Twitter, the canonical source.
It's just as easy to read on the twitter website, it provides additional context, and allows for interaction.