The rather late support of "out-of-tree" building might be caused by a conflict between groups pressing to refrain from the $GOPATH approach (rising in size due to rising Go popularity itself) and Go Dev Team / early-adopters especially as the $GOPATH approach is a central part of Go.
While I can see the intention of Go modules, I believe that many people which belong to the first group migrated from other languages like JavaScript and expect to migrate the workflow as well. I highly encourage everyone to try the $GOPATH approach before rooting for any side.
I've tried it. Can we now get real modules?
Go is a Google project, and Google has a very unique approach to package management: commit everything to the monorepo. The GOPATH is, in essence, a monorepo. If you want to change the API of a library, well, you can just change all its callers across your GOPATH, too. And so for a long time the Go team was unconvinced that package management was a problem.
For example, the Go FAQ [0] still has this to say:
> How should I manage package versions using "go get"?
> "Go get" does not have any explicit concept of package versions. Versioning is a source of significant complexity, especially in large code bases, and we are unaware of any approach that works well at scale in a large enough variety of situations to be appropriate to force on all Go users...
> Packages intended for public use should try to maintain backwards compatibility as they evolve. The Go 1 compatibility guidelines are a good reference here: don't remove exported names, encourage tagged composite literals, and so on. If different functionality is required, add a new name instead of changing an old one. If a complete break is required, create a new package with a new import path.
It is true that if you write perfectly backwards compatible code, then you don't have a versioning problem, but if you think that's a viable solution you're ignoring certain realities of software engineering.
It wasn't until early last year that Russ Cox [1] publicly declared that versioning was a problem and set out to introduce a package manager into the Go toolchain. As it turns out, GOPATH is entirely incompatible with the approach to package versioning that the Go team settled on. You simply can't have two versions of the same package in your GOPATH, unless you're willing to rename one and rewrite all the import paths. Given that public opinion had turned again GOPATH [2], it was finally time to do away with it.
So it took about a year and a half from the time the Go team admitted GOPATH was a problem to shipping a release that made it unnecessary. That's really not too bad. The frustrating part of this saga were the first seven years during which the Go team refused to admit there was a problem at all.
[0]: https://golang.org/doc/faq#get_version [1]: https://research.swtch.com/go2017 [2]: https://github.com/golang/go/issues/17271
That's fine and works perfectly well in the appropriate environment (e.g. large structured work environment with processes etc etc), but for my personal work I prefer to just checkout wherever the hell I want and go from there. Really looking forward to module support so I can use golang for some personal small-scale projects easily without having to go through a lot of the ceremony of setting go up on say a raspberry pi - just checkout and go (no pun intended) will be a breath of fresh air.
My personal theory is that what made Russ Cox cave in was his discussions with Sam Boyer. Cox thought Boyer was going down the wrong path, and thought he had a better solution. Unfortunately, the Go community didn't seem to have read the discussions the two were having, because pretty much everyone thought Dep (Boyer's tool) was blessed by the Go team and was going to be the official package management tool. I can forgive the drama of the end result is a real, non-Google package management system, though.
(While I didn't appreciate the drama, I'm somewhat relieved Dep is not going to be the official solution. Dep is okay when it works, but inherits pretty much all the warts of Glide, which Boyer also worked on. Glide has been an absolute nightmare to work with. Dep is in fact worse than Glide in some respects -- due to weaknesses in its solver, it's completely incompatible with certain significant community packages such as the Kubernetes client. Of course, Dep is not yet 1.0, but I would not say things were looking that promising.)
That's certainly my understanding of the situation. Matt Farina has a great commented history of dep and vgo [0] if you haven't already seen it. The comments are particularly enlightening.
Still, it's not clear to me what made the Go team get into the package management game at all. As you say, for years they were happy to leave that as a community problem. But something spurred them to declare that Dep was an "official experiment."
> Dep is okay when it works, but inherits pretty much all the warts of Glide, which Boyer also worked on.
Funny, I've had exactly the opposite experience. Glide caused us plenty of trouble at CockroachDB, but Dep has worked flawlessly, if slowly. I've also found Sam to be exceptionally friendly and responsive to feedback [1] [2].
[0]: https://codeengineered.com/blog/2018/golang-godep-to-vgo/#co...
> The issues pale in comparison to the horribleness of Glide, but it's interesting just how these tools end up being so damn flaky.
That's the crux of it, isn't it? The dozens of Go package managers that have come and gone over the years have provided us with substantial evidence that building a stable package manager requires several years of development. I think that's why I'm frustrated that the Go team hit the reset button again. Dep has accumulated plenty of bug fixes over the years to handle more and more of these edge cases, but vgo had to start from scratch.
On the bright side, vgo essentially can't fail.
I have numerous thoughts about developments like this one. Mostly, that I've seen a similar situation happen in numerous communities already, Go totally not being the first nor the last one, that the steering commitee have the last say, and may have different taste. I learnt to accept that their choice usually does have merit and usually actually ends for the better. I learnt that it requires a lot of humbleness and sometimes gritting one's teeth, learning to let go of hurt feelings, and accepting that someone else may have reasons you still need to grow to understand. Personally, my own view is that for Sam, this was probably the first time something like this happened, and he wasn't prepared for the hit. And I agree those never stop tasting bitter, given the work one has put in a project of this kind. A good will, hard contribution, being de facto rejected in the end. A child being "lost". But that's not the whole truth, because the child is in this case reborn, though in somewhat different shape. The experiment has served its purpose and brought a lot of value, a significant contribution. On the other hand, I do sometimes wonder, can such situations and misunderstandings be avoided somehow? Or is the world just not perfect enough? And by the way, I also think that Russ was actually taken by surprise by the extent of the reaction. I suppose that's why it took him so long to react, which let the situation and complains get somewhat louder than necessary.
But that's too just my personal opinion. One of many in this somewhat unfortunate situation. I just wanted to also let my steam off in the end, starting to grow more and more tired of the recurring claims that "everybody is surprised". On the contrary, I'm personally one of the people looking forward to vgo, and strongly unconvinced by what dep has become.
The Dep situation is very similar to that of Eric S. Raymond's attempt at replacing the Linux kernel's config tool. Instead of presenting a design proposal and discussing it in public, he pretty much finished the project on his own, perhaps thinking that a working version would lead to adoption by users and thus forcing the kernel team to accept something users liked. Or perhaps he assumed he had clout in the Linux community, which of course he didn't. Either way, this kind of brute-forcing just leads to wasted work and resentment.
You are right that it is solvable. And in many other languages the package management etc have grown independently of the core language itself, from the community. So the attacks on the golang team seem a bit strange. Maybe they thought that these solutions would come from outside the core language team.
In fact, the one main reason I'm looking forward to this release is because of the elimination of GOPATH. It'll make my day-to-day operations at work FAR easier.
It's true that it encourages working with all the code in GOPATH. This is a good thing. Your GOPATH is a view of the whole Go ecosystem. You fix a bug where it makes sense and it is picked up by all users. Sadly, vendoring already messed this up.
I think it's insanity when every program has a different idea of what code a given import path refers to (like is often the case with project-based package managers and vendoring as well). It's no fun to juggle the version differences in your head while working on multiple projects.
Go modules have some good ideas here. Semantic import versioning hopefully reduces the number of different versions you have to consider.
Doing the version selection not per-project but globally for all of GOPATH should still result in a working (but not necessarily reproducible or high-fidelity) build. It definitely reduces the amount of code you need to deal with.