Experimental Dependency Vendoring in Go 1.5 | Hacker News Reader

Experimental Dependency Vendoring in Go 1.5

groups.google.com

174 pointsNateDad47 comments

I really like this proposal. It's basically the best of all current dependency tools - you get reproducibility without needing external configuration files, without hugely long vendor import statements, and still supporting raw go-get.

This is pretty much what we've been doing at DO, but we were dealing with GOPATH manipulation to achieve it. By having `GOPATH="third_party/:docode/"`, `go get` pulls packages into `third_party` automatically, and `go install` finds our binaries in `docode/src/services/foosvc/cmd/foosvcd`.

Migrating to use `vendor/` instead of `third_party/` will be trivial, and not having to modify GOPATH will be nice.

Having a GO_VENDOR=1 env var on top of the `go get -vendor` feature flag would be nice, so that the feature works with minimal disruption.

How do you handle a custom GOPATH like that when the current repository needs to be in there too? We've tried manually symlinking the repo dir into the custom '$GOPATH/src/github.com/user/repo' dir, but it feels really hacky.

You can separate multiple paths with a colon.

More specifically, with os.PathListSeparator. (colon on Unix, semicolon on Windows and \x00 on Plan 9)

We just keep all our go code in one repo.

Am I correct in thinking that this proposal only lays the foundation for other tools to actually manage the vendor folder?

In other words, you would still want something like godep or gb, because manually copying dependencies into a folder (or using git submodules, ugh), and then tracking their versions through their lifecycle, isn't a workable solution in most situations.

This is clearly an improvement over "go get", but I don't understand why Go is so insistent that dependency management is something for the community to solve, and not something that should be provided by Go itself.

Go is so opinionated about so many things, and yet so little interested in having a strong opinion here. The Unix philosophy is great, to be sure, but the Unix philosophy is about increasing modularity through simplicity, and this problem is about modularity.

> I don't understand why Go is so insistent that dependency management is something for the community to solve, and not something that should be provided by Go itself.

They don't want to try solving a problem they don't really have. As RSC says:

> the community understands and encounters these problems more than we on the Go team at Google do.

I find it worrying that Go is seemingly entirely stewarded by a company whose technical workflow is so different, to the extent that it curtails technical decisions. It's designing for the lowest common denominator in an equation that emphasizes Google.

Even if your company doesn't need or want Bundler/Cargo-style dependency management doesn't mean that you are intellectually unfit to design out a good solution together with the community. Dogfooding is great and all, but it's actually possible to design technical solutions that you won't personally use.

> I find it worrying that Go is seemingly entirely stewarded by a company whose technical workflow is so different, to the extent that it curtails technical decisions. It's designing for the lowest common denominator in an equation that emphasizes Google.

There are thousands of programming languages out there, yet Go has quite a bit of traction. They haven't forced anyone to use it, so there must be some reason for that. Maybe Go is successful because it was made with Google's technical workflow in mind.

> Maybe Go is successful because it was made with Google's technical workflow in mind

Sure, but Dart has a fully featured package manager, are you saying Dart was designed at Google without Google's technical workflow in mind? It's silly.

> There are thousands of programming languages out there,

More like hundreds of viable programming languages, not thousands. Anybody can code a lisp in his bedroom,doesn't mean it's production ready.

> They haven't forced anyone to use it

It's a childish and arrogant statement. Just like the "You don't need that in Go"TM. Opening source something was a political decision, but if you're saying Go doesn't need a community then by all means, state it clearly.

I think the point was that the large community that has formed around Go is made up of willing participants, not conscripts.

I get really disappointed when reading such responses. Even when large companies admin they don't know everything and want to see and listen to the rest of the world, its not good enough.

> I find it worrying that Go is seemingly entirely stewarded by a company whose technical workflow is so different, to the extent that it curtails technical decisions.

Having worked at a few such, this is utterly unsurprising. Large development organizations, ala Google, Amazon, etc. are utterly unlike anything on "the outside". This happens slowly over time and is a combination of many factors: centralization of infrastructure, sheer scale (the developer tools teams in many of these shops are often bigger than many other entire software companies), the ability to make cross-cutting decisions for the organization in ways that aren't appropriate for a decentralied external community, etc.

Given all that, I give huge props to the Go team for this kind of self-restraint. It's an enourmously painful decision for the community, but absolutely the right one if you aren't actually feeling the pain. I've seen too many communities suffer because of bad decisions and/or churn as the packaging system breaks and is forced to evolve.

> They don't want to try solving a problem they don't really have.

yet Dart has an official package manager ,which is called pub.

So the Go team don't want to solve this problem because they don't have this problem , but the Dart team wants to solve this problem even though they don't have this problem? Do you know what is the name for that? it's not technicality is is called POLITICS.

EDIT: and both teams work at Google.

At google, the entire codebase is basically one big repo. Everything uses the newest version of everything. There is essentially no dependency management because there are no dependencies.

cough Everything uses the _same_ version of everything.

It's not a political thing. The Go team and the Dart team have different design philosophies. If this were not the case, then they wouldn't be making two different languages.

> Am I correct in thinking that this proposal only lays the foundation for other tools to actually manage the vendor folder?

Yes.

> I don't understand why Go is so insistent that dependency management is something for the community to solve

Well, they solved it, and now we are beginning to adopt their solution. Seems to be working.

Given that you still need an external tool not provided by Go, I'm not sure it's "solved" until the community rallies around a single tool, like Ruby did with RubyGems and later Bundler.

He's saying the community solved it, and the Go Team is using the best ideas from the community to start implementing a solution built into the Go tools.

This has been their policy from the start. The Go team knows that Google tends to handle dependency management differently from your standard project, so they left it to the community to help build and decide which tools and ideas work best for dependency management. Several have arisen, and the Go team has started implementing some of those ideas.

It's solved in the sense that the community has converged on a particular approach, and now we are supporting that approach in the go tool.

Which is fantastic. Thank you!

Yes and no. This is about agreeing on a common file tree layout. It's not the end of the story: there is a separate effort (github.com/kardianos/vendor-spec) about agreeing on a common format for recording where code came from.

Once there is agreement on those, however, I am not sure how important it will be to agree on a single program, although there may end up being natural agreement anyway.

The Go community need not rally around a single text editor, because instead we agree on what Go source files are and that they are named *.go. It may be that something similar can happen here.

Thanks, I didn't know about the vendor spec.

RubyGems was a separated project in the beginning. Ruby ships with RubyGems built-in only since version 1.9. And Bundler is still an independent project. Go is following the same way, letting the community experiment and gain experience, and then adopting what works well into the core project.

Is the criticism here really that we have declined to express a strong opinion about something we don't know a lot about?

A quick comment to tell you that I really really appreciate how you solve problems step-by-step, without rushing. The improvement you proposed for supporting vendoring in Go 1.5 is great.

> I don't understand why Go is so insistent that dependency management is something for the community to solve

The expectation that every new language should come with a dependency management ecosystem is a bit questionable. A lot of major languages' dependency management solutions were created by the community:

Java: Maven, Gradle

Python: PIP

Ruby: Gems

Node.js: NPM

Some of these have been adopted by the official language maintainers, but as far as I know, they originated in the community.

The problem is that Go implements a half-arsed way of dependency management (go get) and enforces things like GOPATH that clash with community created solutions.

This is nice. Finally it seems like we are going somewhere. The call for proposals had not yielded anything groundbreaking so far. Tools like gb are nice but as Russ says it's not gogettable. This seems like a good compromise.

Am I understanding correctly that this proposal again side steps / ignores pinning dependencies for libraries? To me, this is the biggest current problem in the ecosystem and, as a library author, this doesn't solve any of my issues. I hope I misread somehow.

With this scheme libraries can vendor their dependencies, if necessary.

Can you reference the part of the proposal that states this? I'm reading the following points:

> $GOPATH/src/foo will be imported as foo by non-main package $GOPATH/src/mypkg/p.

> $GOPATH/src/mypkg/external/foo will be imported as foo by a non-main package p anywhere, when p is being built as a dependency of command $GOPATH/src/mypkg/c.

It seems to me this is saying that libraries will use external packages only when built as a dependency of a main package. Again, this means library authors are out of luck.

There are two proposals.

You seem to refer to the Google doc proposal that was linked to in the first message of the mailing list thread several months ago.

The link on hackernews points to an answer by Google that was written just a few hours ago that instead "propose" the following:

> If there is a source directory d/vendor, then, when compiling a source file within the subtree rooted at d, import "p" is interpreted as import "d/vendor/p" if that exists.

Ah, thanks.

You're pushing the Go ecosystem in this direction:

    https://spot.livejournal.com/312320.html

How we solved it in Nix: http://lethalman.blogspot.it/2015/02/developing-in-golang-wi...

We don't need go get or any other tool, and we have about 174 packaged libraries in our repository.

This is a great and necessary improvement. Being able to vendor dependencies without having to rewrite import paths is like having the best of both worlds.

The proposed mechanism looks similar to what Node.js does with the node_modules directories:

https://nodejs.org/api/modules.html#modules_loading_from_nod...

That's really nice.

This is a royal PITA with .net and nuget and I've get to find a half decent solution that allows you to actually have the source rather than a bunch of symbol-less dll dependencies or a ridiculous build stack.

Since MVC6/VS2015 beta you can actually get the source instead of dlls. However your build times will increase dramatically so it's a bad idea unless very needed.

Interesting - thanks for pointing it out. I was completely unaware of that. Going to go and play with it on my win10 test box now :)

A vendors B and C, but C vendors D and E. According to this proposal do you lay out the files like this: (A (vendor (B C D E)))?

Edit: I guess HN tree markup is not working :)

Yes. That would work. However, I also believe this will work: (A (vendor (B (C (vendor (D E)))))).

https://groups.google.com/d/msg/golang-dev/74zjMON9glU/LjrVU...

https://groups.google.com/d/msg/golang-dev/74zjMON9glU/0e7M7...

https://groups.google.com/d/msg/golang-dev/74zjMON9glU/qGzsS...

and if you have both: (A (vendor (B (C (vendor (D E))) D E)))

C would pick D E in its C/vendor before D E in A/vendor before D E in $GOPATH/src

ps. I have edited these comments about 5 times to get the parentheses right. :)

Reply on news.ycombinator.com