Introduction to Go Modules
roberto.selbach.ca
roberto.selbach.ca
Resolve your deps once, at build time, and you're done.
Compare that to dealing with the externalized deps that is deploying a python app on a machine that might have the correct version of python and all dependencies, or not. virtualenv? good luck. You end up bringing in docker and other tools just to deal with it. But what do you do if the machine is locked down by corporate IT and they don't allow docker or the installation of other python deps?
I have ... rarely had this experience.
Every go package manager has found new and frankly fascinating ways to lead me into a corner where I think I have committed and pushed working code, but which causes highly-visible CI barfing.
I should add as an aside: I've done two tours of duty as a Cloud Foundry Buildpacks maintainer. Everyone's package system sucks, but some suck a lot less.
* Maven is basically sane if used with levelled starters. Otherwise it's a preview of fighting the Many-Tentacled Ones of the Deeply Nested Dependency Graph using nothing but a dull butter knife held upside-down.
* Gradle is the worst allergic reaction to XML in history.
* NPM used to have amazing bugs and missing features but has laboriously, slowly improved.
* Python is about whether you hit the sweet spot of the particular packaging system you use.
* PHP is a mess, unless you use Composer, in which case you're still stuck with the weird reality that PHP's package system is really a mix of in-process modules which need to be compiled and slabs of plain old PHP.
* .NET Core had what seems like a deliberate policy to come up with the most confusing naming and versioning scheme humanly possible and was wildly successful at doing so. Oh, and no canonical reference for versions. None. Apparently this improved but it was hell on earth.
* Golang. A new package manager every ten minutes. A different vendoring model, different assembly model, different bugs, every damn week.
Only Bundler is basically sane. We hit lots of corner cases but for the 20/80 cases it essentially worked.
What I find so strange is how the systems that came long after Maven -- particularly NPM and .NET -- that they could be so much worse than Maven. Somehow these people weren't aware of the importance of reproducible builds... despite that being a "thing" in Maven almost 20 years ago. Then, as noted above, you have go devs building off of git master branches.
All of this suggests a real problem in software engineering. There's a distinct lack of knowledge sharing that does not lead to gradual, linear progress. It's quite disconcerting. Things are not getting better, every day, after all.
But the whack-a-dep thing is a real, genuine painpoint. Spring Boot Starters make this fairly close to invisible, but there are still billions of lines of Java code where the POM is like the map of an incredibly fragile ceasefire between 813 variants of some 118 different dependencies at 18 levels of nesting.
Like Game of Thrones, except bloodier, more complicated and much more likely to cause everyone to give up.
That said, yes this is a shameful hole in the golang world (caused by the project leads’ inside-Google myopia) but I’m glad they are finally addressing it at this level of integration. And despite a few oddities it looks pretty solid.
My view is that if it can't be built from scratch in a cleanroom environment -- ie, in CI or a buildpacks stager -- it can't be built. This goes for container images too.
The amount of "works on my machine" and "where's Gerald? this module only builds on his laptop" I saw as a consultant was depressing.
Speaking of containers, though: they solved the contending-interpreters problem quite handily. Cloud Foundry has used containerisation since before Docker existed and it really shines for buildpack usage.
We started with Heroku's buildpacks, which had to deal with the nightmare of a pre-container world and showed all the pain of it. There was a great deal of stuff we just didn't have to care about and we later rewrote the bulk of the buildpacks to jettison all the stuff we didn't need.
It's all come full circle: Heroku now use containerisation and both Heroku and Cloud Foundry will soon be adopting jointly-developed buildpacks-lifecycle code.
Of course, "Gerald" is an anagram for "Gradle".
if you really wanted to, you could so something exactly like go is doing in python and create a virtual environment with all of your packages and version of python and ship it. there are also ways to build binaries from python.
if your machine is locked down by corporate IT, you use a virutal environment
Unless you're using CGO, in which case you still have to make sure your target environment has the shared libraries your bindings will make calls to.
I picked 5 "important" Go projects without checking first, and here's the solutions they use:
Kubernetes: The deprecated tool Godep + a pile of Make and Bash (which is the Go-est thing I've ever heard in my life)
Docker (Moby): vndr
Hugo: dep
Etcd: A bit of a mish-mash of dep and vndr, but mostly dep.
Cobra: Doesn't vendor dependencies, isn't a reproducible build (which is somewhat okay, as Cobra is primarily a library, not a tool in and of itself, but it does also have a CLI that probably breaks a lot).
In short, things are not currently fine. Dep is an okay tool, but fails for some use cases, and the community has not really rallied around it. Lots of important projects are sticking with what they've got. Glide, gvt, vndr, and Godep all remain important.
We're deluding ourselves if we discard an outsider's view that Go's dependency management situation is a dumpster fire, because it is. However, to GP, it isn't quite as bad as you suggest, most projects using Go have found some way or another of getting reproducible builds, and don't just run everything from tip of everyone else's master.
I am cautiously optimistic that modules will finally solve this mess, but we'll see to what extent they win in the marketplace of dependency management solutions.
I haven't written Go in a couple years, but I lol'd. This exactly what we used to do, except replace Godep with glide.
Godep, dep, glide, make files, you name it. it's a total dumpster fire.
Having the ability to do so does not mean it should be the easiest and simplest way to get from A to B.
Both pip and cargo can pull from repositories, but it's not the default and it's basically not covered in any tutorial. It's easy to do, but a developer is unlikely to find the information before they're actively looking for it.
> If a developer doesn't understand proper git and package versioning they will have a hard time in any language producing quality software.
I don't understand why you're using Go if your teaching methodology is to throw live chainsaws at people and berate them for not understanding proper power tool safety, why are you not just cutting out the middleman and using C++?
To characterize Go's ecosystem (but not Python's) as "throwing live chainsaws at people and berating them..." is pretty blatant dishonesty, doubly so once Go modules makes its official debut.
clearly you just don't like Go, which is fine. Of all the languages I've picked up Go was far and away the simplest and most productive, which is why I continue to use it. When I wrote python we were constantly using pip to pull from repos, so moving to Go I liked their repository centric view. Just because python recommends pip, doesn't mean junior devs don't totally botch dependency management with it. Dep management and git are core skills developers need to learn well to produce good software. There is simply no substitute for a bit of elbow grease on the matter. I'm not berating anyone, software is hard and being good at it means taking the time to learn some core tools.
https://stackoverflow.com/questions/19198166/whats-the-diffe...
So Go can do what it wants as long as it is clear and you can find its definitions easily.
I agree with the GP that go package/module are similar to Java package/module
Thankfully the whole split-package nonsense will eventually go away, since the JPMS doesn’t allow a package to reside in multiple modules.
That doesn't seem weird to me. Lots of languages can split their namespaces across packages.
Right now it seems near impossible with go to do that other than manually cloning the repositories into the correct path. We can't host our things publicly and have to use SSH to clone the repositories at my company.
It is especially frustrating in our CI/CD process if we need to manually clone our packages for setting everything up.
To make matters worse, dep tried to stuff too much of a DSL into the package specification on the command line. example.com/path/pkg@hashish made it impossible to specify git@example.com/path/pkg as the location because the parser wasn't robust enough, and the package location parser wasn't/isn't smart enough to honor ssh://git@example.com/path/path as a way to be explicit about how you wanted this done.
dep did work for our use case if you edited the toml file directly, once I made a 2 character change to a regular expression in v0.3.0. We use dep and stopped upgrading with that version; I'm hoping go modules make non-public repos easier, but I'm not holding my breath.
git config --global url.ssh://git@github.com/.insteadOf https://github.com/Its references are canonical and it’s up to you to set up the relevant process for it to retrieve the source for those references.
I get the impression that that’s what the proxy stuff is about - you’ll just set up a proxy which deals with retrieving the code.
> Its references are canonical and it’s up to you to set up the relevant process for it to retrieve the source for those references.
Git has a proper notion of references; it properly separates transport from references of remotes. Go uses just `https://` links, and expects the website to have a single `<meta>` tag containing a vcs clone URL (i.e., transport) to use. VCSes have had separation and multiplicity of transport for a long time, but Go will deal with only one URL (i.e., transport), with no way for the user/distributor to specify preferences otherwise.
For example, everything from git.kernel.org to github.com allows the user to chose which URL to clone with. The remote is not supposed to know or dictate a single transport.
I think `go get`'s limited method of transport discovery is just a hack that got released into production and stagnated in its form. It was a perfectly fine hack for public repos (on the internet or on Google's intranet), but it just never got any features (let alone documentation) specifically for repos that need an authenticated way to clone them. The fact that git has a nifty way to rewrite URLs is just a luck.
I didn't claim Go supports only git.
> so HTTP and SSH aren't the only options.
My entire point is that VCSes support multiple transports since before Go came about. I never claimed Go should support HTTP and SSH only. In fact, I never claimed it should support either. I claimed it shouldn't force the VCS host to chose for the user which clone URL (and thus transport) to use.
Sure build on top of it, but don't expect your users to implicitly know the defined, yet obscure, features of git you build on.
> It’s hardly magic.
I didn't claim this is a magic feature in git. I meant the way of `go get`-ing a private library by letting `go get` try and clone from https URLs while silently changing them to git:// URLs in a completely separate layer under it is magic.
That's what documentation is for. Every application should be documenting (e.g. in README.md) how it can be built.
Only when the application developer looks online for a solution to this do they find some StackOverflow post detailing this workaround. Or other workarounds, like the insecure option of telling git to save your https credentials.
Sure, a knowledgeable application developer can put whatever quirks their build system needs in their documentation; but a developer ignorant of these workarounds shouldn't have to go beyond `go get`'s documentation in the first place.
git config --global url.ssh://git@bitbucket.org/.insteadOf https://bitbucket.org/
That being said, I share your hope that this can remain smooth in the new Go modules implementation. $GOPATH is always a hurdle for new-comers to wrap their heads around in my experience.
Feels like it would have been better to use a different delineating character instead of a slash to make it abundantly clear at a glance that it's effectively a major version tag and not a subpackage.
Other than that, seems relatively straightforward. Hope this is the last "new and completely different" attempt to tackle dependency management in Go.
Also it's unclear how initial development (as per server) is to be handled. If the first few versions are v0.1.0, v0.2.0, etc. - how are these handled on the import paths.
What if a package URL actually contains "/v2" at the end, but this isn't actually referring to the version and should be used when fetching the package. Seems very odd. Agree that it should have been more like "@2".
What if you want to include two different minor versions, for example, as you can include two different major ones. Is that possible/allowed?
v0 and v1 are treated as the same package, you're supposed to use v0 as long as you're adapting to vgo / potentially doing breaking changes and use v1 when you know you'll keep a stability guarantee.
As I mentioned in the sibling comment, it's all described in Russ's blog post series, long, but well worth the read: https://research.swtch.com/vgo
This was answered in the post. It's handled in the mod file and not via import path. Import path only changes when there is breaking changes, ie: a major bump and a new /v# for the import path.
Upgrade minor or patch to most recent:
go get -u
go get -u=patch
go get package@versionCheck out Russ's blog post series: https://research.swtch.com/vgo
[0]: https://futhark.readthedocs.io/en/latest/package-management....
1. How does one use a branch of a dependency in go.mod? (scenario: local dev where api & impl repos are separated)
2. How does one use branch from a fork in go.mod? (scenario: fixed bug in someone else's repo and now want to use my fork)
go get example.com/repo@branch-1
go mod edit -replace example.com/repo@version=example.com/fork@version
---
4. Major version zero (0.y.z) is for initial development. Anything may change at any time. The public API should not be considered stable.
No, it doesn't mean there is no API, it means there is no compatibility promise; it means there is an API per version, and the relationship between them i s not known.
> And with zero API any change is compatible and cannot break anything
That would be true, but this isn't "zero API", it's "zero promise of API compatibility"
That makes your statement "With zero API compatibility any change is compatible"...
That's very obviously wrong.
Please don't jump through these bizarre logic hoops to try and justify what go does here.
Or is my only "supported" option to push consumers to the new major version - with the possible bit-rot of the old code?
[ed: that is manual backport fixes/improvements to a v1.x version]
See https://research.swtch.com/vgo-import, section "Automatic API Updates".
Hooray!
Holy hell, it's eight!
$ gcc -xc++ -E -v -
#include <...> search starts here:
/usr/include/c++/7
/usr/include/x86_64-linux-gnu/c++/7
/usr/include/c++/7/backward
/usr/lib/gcc/x86_64-linux-gnu/7/include
/usr/local/include
/usr/lib/gcc/x86_64-linux-gnu/7/include-fixed
/usr/include/x86_64-linux-gnu
/usr/includeDoes not mean of course that include paths are better.
I really don’t see these git based hacks can improve situation much.
"We want to make it possible, at some future point, to introduce a shared proxy for use by the Go community, similar in spirit to those used by Rust, Node, and other languages. At the same time, the design must work well without assuming such a proxy or registry."
Of course, even this can be considered decentralised since different package repos can be used easily. This is especially useful for private packages.
Everyone was up in arms about npm allowing this a few months back. Rightfully so, I think, but I feel like it's going to be easy to hit this issue when pulling straight from github.