Using Go Modules
blog.golang.org
blog.golang.org
I do think it's kind of odd that it decides to put packages in GOPATH, but disallows you from coexisting go mod projects in GOPATH. I organize all my code for all my languages using the go repo as dir. I've had to maintain a separate tree for go mod which is not ideal.
After using Go dep and mod, I felt that dep was the super straightforward and obvious way to do it. It's how I would have done it. Go mod is much more Goish, it's opinionated and based on a philosophy that fits with the language. It Gives me hope that if they do add generics they will make them uniquely Go as well.
The main problems I've had with go modules are fast-and-loose upstreams that happily rewrite history. go mod detects this and stops the world, and you have to manually edit go.sum.
I also feel like "go get github.com/foo/bar" doesn't always get me the latest version. You see that there was a commit 1 hour ago, but then "go get" adds a version like 20181130-93874837 to your dependencies. I assume they know what they're doing? But I'm probably wrong.
This exists in other package management systems. At least, it did when I used composer and came across the same issue some 3/4 years back. I think it's one of those issues that are quite rare and in well maintained packages are even less likely.
This is really irritating. I have the same problem with TLS: every time a malicious actor tries to manipulate my connection and steal my cookies I get an error page and can't do online banking anymore!
Sarcasm aside: this is not a problem with Go modules. This is a problem with the libraries you're using and you should avoid those projects until they get their act together.
Between the versioning behavior and trying to figure out how to document transitive dependencies correctly go mod has made my life as a maintainer harder.
I used to be annoyed that I had to put every go project into a dir 7-8 layers below that e.g ~/work/go/src/github.com/joshklein/project/cmd/hello_world.go, but now I can have ~/work/go_hello/src/main.go. Right?
I understand this isn’t the main point, but frankly it’s the thing I care the most about.
That said, mod allows you to put your project anywhere but in the GOPATH, so the answer to your question is yes. The path is virtualized in the mod file.
I've since started to also organize all of my code the gopath way and I just set $GOPATH to $HOME. When forking I just add a new remote and work out of the upstream repo, which seems saner now as it reduces object duplication.
I don't know if I will ever move away from organizing my node this way if I can help it.
> That said, mod allows you to put your project anywhere but in the GOPATH
I don't know if I've run into this yet, though the only project[0] I've used gomod with has a makefile so it might do something to handle it.
yes
But yes, with modules you don't need to do this anymore.
go mod init github.com/joshklien/my-go-project
https://github.com/golang/go/issues/15507#issuecomment-24158...
For a commercial entity shipping software which may include upstream components, it's important to have options for maintaining (and versioning) in-house forks of upstream components. This becomes a problem when you fork v1.2.3 from upstream and want to internally release your fixes, but now your internal 1.2.4 and upstream's 1.2.4 both have the same number but diverge in content.
I like the Debian solution to this, which permits freeform textual suffixes, so that in the hypothetical scenario above, you can release v1.2.3bigco1, v1.2.3bigco2, with the final number indicating the version of the patchset being applied onto the upstream release; then it's also clear what to do when you rebase your fork because you've maintained the integrity of the upstream version.
The nice thing about not doing what debian does is that there is consistency. Freeform usually means diverging opinions, which is not the Go way.
Patched versions are "sub" versions of the main version, and those sub versions also keep evolving for the same upstream version.
https://github.com/golang/go/wiki/Modules#when-should-i-use-...
But if you have an internal version that's used in lots of modules, it's probably better to use an import path under your own organization, to avoid confusion.
Debian packages can Replace each other too, and that's certainly an option for managing this scenario, but I feel it's often more disruptive— it's a much more invasive change to have to go in and mutate the package name all over the place, and then it's harder to unwind it later when upstream merges and releases my change and I don't need my fork any more. Maybe this is better in go land?
If your binaries are spread across multiple modules, each one would need its own replace directives, so you'd have a bit of duplication. (But maybe this independence is good, since you can upgrade them one at a time?)
If you're publishing a library and don't control the binaries (this is something your customers build) then it looks like "replace" isn't going to work (it's under their control, so they have to do it). If you need to control the exact versions used by your library, you'll want to look into vendoring.
Doing a fork/replace on everything in the chain will have no affect; `replace` is only obeyed for the top-level `go.mod`, `replace` in dependencies is ignored.
Go also allows freeform textual suffixes, it just needs a `+` in there: v1.2.3+bigco1, v1.2.3+bigco2.
Where semver is ivory tower idealism, Debian is the battle-tested, pragmatic reality.
> A pre-release version MAY be denoted by appending a hyphen and a series of dot separated identifiers immediately following the patch version. Identifiers MUST comprise only ASCII alphanumerics and hyphen [0-9A-Za-z-]. Identifiers MUST NOT be empty. Numeric identifiers MUST NOT include leading zeroes. Pre-release versions have a lower precedence than the associated normal version. A pre-release version indicates that the version is unstable and might not satisfy the intended compatibility requirements as denoted by its associated normal version. Examples: 1.0.0-alpha, 1.0.0-alpha.1, 1.0.0-0.3.7, 1.0.0-x.7.z.92.
Item 10 explains how + can be used to add a build tag.
> Build metadata MAY be denoted by appending a plus sign and a series of dot separated identifiers immediately following the patch or pre-release version. Identifiers MUST comprise only ASCII alphanumerics and hyphen [0-9A-Za-z-]. Identifiers MUST NOT be empty. Build metadata SHOULD be ignored when determining version precedence. Thus two versions that differ only in the build metadata, have the same precedence. Examples: 1.0.0-alpha+001, 1.0.0+20130313144700, 1.0.0-beta+exp.sha.5114f85.
Instead of using v1.2.3bigco1 or v1.2.3+bigco1, we should use v1.2.3-bigco1. If we need to add a build tag we can use v1.2.3-bigco1+001.
Looks up where? GOPATH? The tree below the current file? The current directory? What's the "current module", anyway? Where is that stored?
"go: downloading rsc.io/sampler v1.3.0"
Downloading from where? Github? From what repository? The page for "go get" now talks about "module-aware" mode, but doesn't make it clear how that works.
This is the usual headache with dependency systems. There's some specific directory layout required, and you have to know what it is. This article needs to be more explicit about that. Otherwise you end up depending on lore and monkey copying.
No you don't. If it supposedly works, why would you care where these modules end up in when they are downloaded? There's currently no indication that you need to be aware at all of whatever directory layout the module system needs, as it takes care of it automatically.
Finally, the article does actually mention where the modules are downloaded:
> ... the downloaded modules are cached locally (in $GOPATH/pkg/mod)
Given that the import paths are URL-ish, I personally find it fairly obvious where things are coming from - far more so than any other language/dependency manager I've worked with. That's something unchanged with go modules vs GOPATH.
The article uses "current module" like you might use "current git repository"; it's the module containing the directory that you are currently `cd`ed to. Just as git identifies the repo by looking for `.git` in successive parent directories, go identifies the module by looking for `go.mod` in successive parent directories.
> > "go: downloading rsc.io/sampler v1.3.0"
> Downloading from where? Github? From what repository?
From the same place that non-module-aware `go get` downloads it. The only difference on the "remote side" of that from old `go get` is that it grabs the `v1.3.0` git tag, instead of git HEAD; if you don't specify a version it grabs the most recent `vSEMVER` git tag instead of git HEAD.
> > "the go command automatically looks up the module containing that package"
> Looks up where? GOPATH? The tree below the current file? The current directory?
Looks it up on the network, à la `go get`. Exceptions:
- The `replace` directive in your `go.mod` can manually override where it looks up a specific package.
- Telling Go `-mod=vendor` will have it look up all depended-upon modules in `vendor/`, rather than via the normal mechanisms; you can create the `vendor/` directory with `go mod vendor`.
It doesn't use the network every time; a module cache lives in `${GOPATH:-${HOME}/go}/pkg/mod/`; but you don't need to know that any more than you need to know that a build cache lives in `${GOCACHE:-${HOME}/.cache/go-build}`.
> There's some specific directory layout required, and you have to know what it is.
Not really, the only requirement is that you have a `go.mod` file that identifies a package name for the directory that it's in. So that if I have github.com/lukeshu/foo, it just needs a `go.mod` saying
module github.com/lukeshu/foo
> This article needs to be more explicit about that.
Does it, though? It's a blog-post, not the "normal" documentation. The full docs are much more explicit and nitty-gritty than a high-level blog post.
I may be missing something here (it's been quite a while since I did serious Go stuff so `go get` may have changed too)
> Does it, though? It's a blog-post, not the "normal" documentation. The full docs are much more explicit and nitty-gritty than a high-level blog post.
It's a bit weird to have a blog post about the new module system with nothing about how modules can be published, a pretty common task.
Why would you try to `git clone` it directly instead of trying to `go get` it?
$ GO111MODULE=off go get -v rsc.io/sampler
Fetching https://rsc.io/sampler?go-get=1
Parsing meta tags from https://rsc.io/sampler?go-get=1 (status code 200)
get "rsc.io/sampler": found meta tag get.metaImport{Prefix:"rsc.io/sampler", VCS:"git", RepoRoot:"https://github.com/rsc/sampler"} at https://rsc.io/sampler?go-get=1
rsc.io/sampler (download)
created GOPATH=/home/lukeshu/tmp-go; see 'go help gopath'
Fetching https://golang.org/x/text/language?go-get=1
Parsing meta tags from https://golang.org/x/text/language?go-get=1 (status code 200)
get "golang.org/x/text/language": found meta tag get.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/language?go-get=1
get "golang.org/x/text/language": verifying non-authoritative meta tag
Fetching https://golang.org/x/text?go-get=1
Parsing meta tags from https://golang.org/x/text?go-get=1 (status code 200)
golang.org/x/text (download)
The TL;DR is that it fetches `https://${pkgpath}?go-get=1`, and looks for `<meta name="go-import" content="...">` to tell it where to `git clone` from (with some hard-coded hacks for common websites that don't support that, like github.com).
> It's a bit weird to have a blog post about the new module system with nothing about how modules can be published, a pretty common task.
Because that hasn't changed. You still publish packages the same way you always have. The only difference in how you publish is that now "vSEMVER" git tags mean something.
Thanks.
The example using a less-commonly-known package distribution method is probably causing the confusion in Animats' original comment.
replace pkg => ../../pkg
> go mod init creates a new module, initializing the go.mod file that describes it.
> go build, go test, and other package-building commands add new dependencies to go.mod as needed.
> go list -m all prints the current module’s dependencies.
> go get changes the required version of a dependency (or adds a new dependency).
> go mod tidy removes unused dependencies.
And you can use some sugar to declare exact versions you want.
[0] https://github.com/Microsoft/vscode-go/wiki/Go-modules-suppo...
If you're using vim-go, HEAD has now gopls support. I believe vscode also added recently support for gopls.
Other than, I agree that it broke all other tools, but most of them are migrated to use the go/packages package so it'll be better going forward.
I tried to use the profiling flags to get some data to file reports, but ran out of yak-shaving time when I realised was trying to flush out a bug in the tool command helpers in golang/go where it would write empty profiles if the process was shut down the exact way vim was shutting it down. That may have been fixed since, I haven't had a chance to look back into it.
This issue is tracking compatibility of common tools https://github.com/golang/go/issues/24661
I was coincidentally converting my Go project to use Go modules yesterday. I depended on a Google API which was originally retrieved via 'Go get', corresponded to docs and worked fine as this pulled from HEAD. `go mod` did not work out of the box, as it required the latest semantic version (v.0.2.0) of my this Google API import. This version, however, is behind documentation and broke my code.
I understand I can require a specific commit in the go.mod file, but the strings for specific commit seem cryptic. Where can I look up the version hash that matches the doc site?
s/$SEMANTIC_VERSION_BRANCH/$BRANCH
In this case, $BRANCH == 'master'. 'go build' will resolve 'master' to a specific commit hash version.
I haven't felt the need for them.
At what point do they become necessary? Which pain-points should I be on the lookout for?
It also gives you the freedom to have your projects wherever you want.
Realize this is a tooling issue and not necessarily a language one, but do feel like they should be mentioning this. It makes working with modules pretty painful. (we are doing it but man do I miss simple refactors / usages etc..)
This thing apparently doesn't solve my problem. `go mod tidy` simply removes a dependency from a module, but that dependency is still cached in a big arcane directory at $GOPATH/src/mod. Why?
I think we all should be using something like Nix for dependency management that solves all problems, but that is so hard to setup!
If you have a disk space problem, Nix would seem to be the opposite of the solution to that. One of the ways Nix does its magic is to chew through disk space a lot more freely than most distros do.
(Ironically, pnpm doesn't have an auto cleanup feature.)
And my point is not that disk space is never an issue. That's why I said it was a minority issue, not "not an issue". My point is more than Go is not a language about addressing every fiddly minority's issues. It's definitely about the 20% that does 80% of the work. So waiting for a language lead by a philosophy like that to address a disk space issue is probably not a good plan.
To be a bit more constructive, I'd observe I've had great experiences with cross-compiling. On the chance you're disk-space limited because you're developing right on a target resource-constrained device, you may be able to move your development to a more powerful system, even of a different architecture, and cross-compile fairly easily. I often have a workflow where I just "go build blahblahblah && rsync blahblahblah target:blahblahblah && ssh target blahblahblah" and I just press up & enter on a shell when I want to push & test the code. As long as you've got half-decent bandwidth to the target device, it's fine. There may be a couple of other buttons you may want to push to speed that up, because IIRC when cross-compiling it'll end up building the entire app, and you'll want to pre-compile things for the new arch.
And sadly, to use. Nix is a great idea but it could do a lot to help itself out if it ever wants to become more than niche:
1. Use a more approachable package definition language; Starlark (https://github.com/google/starlark-go) or Lua would probably be great choices.
2. Make it reasonable to figure out what the "type" of a package dependency is so we can figure out how to use it and/or find its source code
3. Document package definitions. We document source code in typed languages; Nix expression language is untyped and generally less readable--why not document it?
4. Nix tools have a `--help` flag that only ever errors with "cannot find manpage". This is just user-hostile.
5. Using almost-JSON for the derivation syntax, but then providing a "pretty printer" that keeps everything on one line but with a few more space characters.
6. Horrible build output--everything for every build step (megabytes of useless gcc warnings) gets dumped to the screen. Contrast that with Bazel and company which only print errors.
Plus a long tail of other things I'm forgetting.
I stopped at the point that I had to keep track of an enourmous build.nix file in all my project directories that did a lot of magic if I wanted to use Nix for Go or JS package management.
I also failed completely to search and install executable packages from npm and other minor things I tried.
I did that after going through the complete Nix tutorial and managing to understand pretty much everything and love it.
- There's a duplicated section starting with: "Note that our module now depends on both rsc.io/quote and rsc.io/quote/v3:"
- In the example of upgrading a major version, Hello was renamed to HelloV3, but the caller isn't renamed. It should be "quoteV3.HelloV3".
other than that it works like a charm, here's a tutorial i wrote: https://getstream.io/blog/go-1-11-rocket-tutorial/
replace (
github.com/repo/module/path vX.X.X => /path/to/github.com/repo/module/path
)I am baffled to why Go has not figured this out and are presenting recent developments as some kind of breakthroughs - they are not.
To me, and I'm not trying to be disrespectful here, because of the shortcomings it has, Go is in the experimental/keep an eye on bucket. It does some things really, really well but it's far from the panacea most developers that use it think it is.
But go mod is great, and it is very different than cargo, pip, npm, etc. I don't think they are claiming it's a spiritual revolution, but it solves the problems people who needed traditional package versioning where having in an elegant and Goish way.
Go is also far from experimental. It has areas where it could be better, but it's a panacea compared to writing Java in an IDE with Maven for a living.
imho, go wouldn’t be a thing if it didn’t have [initial] backing from google.
Go is also just really great. You don't need an IDE or a big crusty language to write good software. Go is small, compiles quickly, runs efficiently, is easy to teach, and has great tooling that lets you get up in running in just a few minutes. The ecosystem is centered around open source, the Go project itself including the name, logos, and specification are open source, and the Go conference in Denver is fantastic.
Go is not kool-aid.
the Google brand helped immensely and containers/docker are their own flavor of kool-aid. The art of actually thinking about how you’ll package and deploy your software is lost. Put it in a docker container and fuck it. Fuck security, fuck ever understanding what you’re running, ship it fast/ship it now. My test for docker and its own flavor of koolaid is to ask its supportes what a container is. 95% don’t know. Yep
to me, only the idea of using a git submodule pointed to a repo that has all my dependencies is...
repeat after me: a source control system is not a dependency management system.
The monorepo approach and vendoring does work, and there are a lot of advantages to having everything work on each revision, you just need to be conscious of it.
there is a reason we modularize and version things in software.
in the context in which you pull in external libraries you don’t control, using a monorepo is madness.
> Go me, and I'm not trying to be disrespectful here, because of the shortcomings it has, Go is in the experimental/keep an eye on bucket. It does some things really, really well but it's far from the panacea most developers that use it think it is.
Go is definitely a production-ready language, and indeed it powers much of the most important software that has been written since it went 1.0 (particularly in the Cloud and DevOps spaces). It's not perfect, but it's really, really good and improving steadily.
Not really the first time. They made attempts before. Dumped it along with person who created earlier solution and then developed current solution.
To their credit they identified early that an official solution is essential.
this made me smile.
you know what’s written in go? terraform. i have yet to meet someone that has actually leveraged terraform in a production setting and does not think it should be banned. what else? k8s? the favorite poster boy of our generation. solving problems you don’t have and replacing figuring out how to deploy your stuff with the anguish of keeping the cluster up and uptodate.
Most important software is quite a claim as well.
go? pull shit from git hub and pretend that it’s acceptable and works