Problems with Go Get
0x74696d.com
0x74696d.com
Projects need a manifest so that proper dependencies are declared. People who say "just use make",or "just use git submodule" or "X feature of your VCS" are just pushing for more fragmentation. I download a package.Now I have to check what VCS feature it uses, wether it uses a shell script, make , or build tool Z to fetch dependencies, do people advocating for these solutions really want to use a language where libraries aren't compatible with each others? and then some people say : just vendor and commit dependencies. Ok but let's say I find a bug in one dep. Patching all the different versions of the same dep is just not something manageable.
I think it's time go devs acknowledge the fact that there is an serious issue here, instead of resorting to the ususal "you don't need that in go".
Godeps is a first step, but we need a official way to manage packages.So everybody is on the same line.
I would really like something like composer for Go, with flat dependencies. Which means that, unlikes NPM that downloads the entire world each time one installs a library, dependencies are flat,which keeps API stables and reduce the the amount of useless libraries and forks.
I'm out of the loop, so honest question: does golang-dev still say that? I thought they were not in denial of the problem, but indecisive about the solution. Like with generics? Or do they consider this a non-issue?
I know Cheney recently came out with his own solution, so it can't be that bad, right?
Yes, and so have others. What they really need to do is say "this is our blessed solution" and bundle it. Otherwise, there's just a bunch of competing solutions floating around and nothing is really going to take hold.
And yes I'm well aware that Java doesn't have a dependency manager but Maven emerged as the de facto way to handle dependencies. I think that has a lot to do with developer attitudes. I see Go developers as much more "don't tell me what to do, I'll roll my own solution" than Java developers, and that's why I fear a third-party dependency manager won't win out in the Go world.
The core of my work is on the operations side rather than on development (although I do a good bit of that as well), so my perspective is that this initial overhead is a one-time cost for using a library, and that I can live with that.
> I think it's time go devs acknowledge the fact that there is an serious issue here, instead of resorting to the ususal "you don't need that in go".
Encountering this and a lot of "well this is the way Google does is so we should too" was a big part of what inspired this post.
This was a frequent occurence on the JVM platform as libraries moved from Ant to Maven to Gradle over the years.
While building the program, Go should search in my $GOPATH for the specific libraries, and if they're not found, search the libraries included with my package.
Some of the issues they've talked about included, "What happens when you can't `go get` a library because internet dies/repository moved/etc.?", which IMO is a good reason to include the libraries _with_ your Go package.
You cache the libraries in a central repository like Nexus or Artifactory and back it up.
This is standard practice in enterprise development where everything is restricted from downloading software except for this particular server.
[1]: https://www.dartlang.org/tools/pub/get-started.html
It solves every single one of the problems listed here. It has a very simple workflow:
1. You make a pubspec.yaml file to list your package's immediate dependencies and the version ranges you allow for them.
2. You run "pub get". It finds all of your transitive dependencies, picks versions that satisfy the constraints, and downloads them.
It also creates a pubspec.lock file that specifies which versions it picked for everything. You check that into source control and now everyone on your team will use the exact same versions of all of your package's dependencies.
3. When you want to get a new version of a dependency, you run "pub upgrade <package>".
4. To add or remove a dependency, just edit your pubspec.yaml file then run "pub get" again.
We have a central repository where packages or hosted, and you can also pull in packages from Git or your local file system.
Packages do not at all interfere with each other. Two packages on your machine can use different versions of the same dependency without any problem.
Bundler is an absolutely brilliant package manager. My prediction is that every language either has a Bundler-style package manager or will eventually move to one.
I do wish Bundler would simply be bundled into Ruby at this point; it's become so ingrained that there's no longer any reason to not make it a first-class citizen. The one big problem with Bundler is "bundle exec", which is required because project-local gems can conflict with system-wide gems. If all of Ruby honoured Bundler, we could let the Ruby runtime itself handle the gem isolation.
My main complaint about Bundler and RubyGems (and NPM for that matter) is that we're still unpacking packages with no good reason. RubyGems would be easier to deal with if you could just treat .gem files like Java does with JAR files. (The sore point is gems requiring compilation of binaries, but that's solvable.)
Actually, I have another complaint: Even after numerous security gaffes, gems still generally aren't signed.
export RUBYGEMS_GEMDEPS=-
To handle shared constraints.
Let's say myapp uses foo and bar, which both use shared_thing. To solve this, we need to pick a single version of shared_thing that both foo and bar work with. If foo and bar have narrow-to-the-point-of-precise constraints like shared_thing 1.2.3+bug-fix, then it's very likely that no version of shared_thing makes both foo and bar happy. The end result is no solution.
To accommodate that, packages are strongly encouraged to semantically version themselves. Then, when you depend on a package, you use as wide a range as you can. If foo wants shared_thing ">=1.2.3 <2.0.0" and bar wants ">=1.3.0 <2.0.0" then the solver can pick, say, 1.5.3, and both are happy.
Ideally I think each library would have its own internal copy of any dependencies, but there are some issues with doing that at least in go.
Perhaps what you want is npm where what you describe is built in, first class, and looked down upon by much of the sane world.
npm is amazing for javascript, but terrible for compiled languages and is especially terrible for security updating; the giant mess of transitive dependencies of the same dependency of varying versions can get messy fast.
JacaScript can cope perfectly well with modules A and B relying on different versions of C. It'll often cope even if A calls B with an object it got with C, or some other module broadly compatible with C. Static typing proponents aren't comfortable with the idea, so it's not "sane" to them. Sane or not, many argue it's not just not slowing down the Node community, but actively contributing to its explosive growth.
In C#, you get MethodMissingException if A.DLL and B.DLL refer to different copies of C.DLL and A calls B with an object it got from C, even if the copies of C.DLL are identical.
1: Perhaps I should contrast with strong typing, not static typing. Meh.
That works if and only if the library never exposes the use of that dependency, but it can break in horrible ways otherwise.
For example, let's say library foo uses some hash_table package to define some hash table data structure. Foo returns hash tables from its public API. Now imagine your app uses foo, gets one of those hash tables, and passes it to bar.
Bar also uses the hash_table package, but--since dependencies are encapsulated--has its own copy of a different version on hash_table. How is that hash_table code going to handle being given a hash table that was created from a different version of itself?
In a dynamically-typed language like JS, it may work, but this seems super sketchy to me, which is why I think an app should only have a single version of any given dependency.
Provided that everyone does anything even resembling semver, you could just use Gadget 1.2.>=3, Tool 2.3.>=4, ssl-wrapper 3.4.>=5 instead and still be fairly sure the api won't suddenly change to something incompatible.
But other languages seems to get away without it.. clojure with lenigen seems to do it nicely.
So what are the problems with it?
http://go-talks.appspot.com/github.com/davecheney/presentati...
That may be true if by "nobody" he means Go users, but outside of the Go community things are different. I haven't heard many complaints about them in the Ruby or Dart ecosystems, and the value they provide (repeatable builds!) is very real.
it sounds like you some-what 'ported' bundler to dart... what's your impression of how difficult it would be extend a tool like this to work for many languages?
it seems like i'm always asking them all to do the same thing... go get this at that version from there, include anything it needs, and stick that somewhere this thing can find it.
More or less. Most of pub's code is devoted to stuff that's specific to Dart, but it's underlying "philosophy" is Bundler's model.
> what's your impression of how difficult it would be extend a tool like this to work for many languages?
This comes up a lot, but my hunch is that we're unlikely to see a successful language-agnostic package manager. The package manager has to be written in some language, and users of language X generally don't want to be told that they have to install language Y before they can download packages for X.
Also, most of the code for a package manager is fiddly details related to the specific language in question. Thinks like how files are organized, running tests, etc. The amount of code you could share is actually pretty small.
https://github.com/skelterjohn/wgo
It's a workspace layer on top of the go tool, with added support for repository pinning. You 'wgo get' the repo you want, and 'wgo {save,restore}' will pin/fetch it in the future. Also, 'wgo FOO' will do 'go FOO' with GOPATH set for your workspace, so all the normal goodness is still there.
It also makes it so you don't have to manage the GOPATH env var anymore, which I always found to be a pain.
Most package managements solutions that are popular in other ecosystem (pip for python, npm for node, cpan for perl etc.) combine 2 functions: downloading code and versioning.
go get only does the first part: downloading the code. If you ask me they solved the problem better than pip/npm/cpan by getting rid of intermediary and going straight to the source.
go get has never claimed to be versioning solution so while I understand why people make shallow comparisons to pip/npm/cpan and assume that it does and complain that it does it badly (including this article), the reality is that go get is not a versioning tool at all.
There are good reasons why versioning is not supported by default in go: there isn't one way of doing it that is clearly superior to other ways. That's why we have so many solution (including the one advocated by this article) and none of the solutions became a de-facto community standard.
In summary: stop complaining that go get doesn't do versioning. There are plenty of working versioning solutions to choose from, pick the one you like the best.
> go get has never claimed to be versioning solution so
> while I understand why people make shallow comparisons
> to pip/npm/cpan and assume that it does and complain that
> it does it badly (including this article), the reality is
> that go get is not a versioning tool at all.
You've summed up the entire reasoning behind my post more succinctly than I could. Thanks.
I'd like to hear these "good reasons" why it doesn't though, what are they exactly?
Wouldn't a common standard on these things be brilliant and totally in line with go's overall philosophy?
Also, the author says:
> go get fetches dependencies, and their dependencies,
> and so on. But it doesn’t help you to figure out what
> you’ve got. You can’t do the equivalent of a pip list.
Go has so few commands and options that it's easy to be familiar with all of them.
There are plenty of tools that works quite well: bundler, cargo (even with some young rough edges) and at some degrees npm.
So there are tools and better ways to do it.
It is just that Sir. Pike doesn't like them and Google doesn't need them.
> In summary: stop complaining that go get doesn't do versioning. There are plenty of working versioning solutions to choose from, pick the one you like the best.
The reason why there are plenty of them is just one: none of them works well.
Exact opposite of what you said. You don't have such mess of pkg managers in others languages.
It has had far less issues than bundler, npm, etc.
Right now for example I have a giant repo with tons of dependencies and:
godep: unable to detect version control system for code.google.com/ path
Absence of a clear winner doesn't mean doing nothing is the best strategy.
> There are plenty of working versioning solutions to choose from, pick the one you like the best.
Unfortunately, that doesn't compose.
The whole point of a package manager is to deal with acquiring dependencies and their transitive dependencies. If the ecosystem doesn't agree on a single package manager, you can't handle transitive dependencies.
What do you do when you want to use four packages, each of which uses a different package manager for its dependencies?
Being able to dispatch a co-routine with `go ...` is beautiful. Being able to fetch a dependency with `go get` is just as succinct and expressive.
I'm not up on Go language development but imho the world would be a better place if they could fix the behaviour to preserve the syntax!
There's obviously no way they can fix the behavior in a backwards compatible way because it is so fundamentally broken, and I think that doing something because of a pretty name is absolutely ridiculous when it comes to creating a technical product. You need names like 'heartbleed' if you want to talk to management, but go get is a fundamentally developer focused tool and thus its utility should come over its naming.
I almost see its naming as a bad rap against google developers because it's an un-googleable phrase and who cares if it sounds pretty if it's un-googleable and also utterly terrible.
TL;DR: Versioning is important.
- All releases go on master. Dev goes on separate branches.
- Do not make backwards incompatible changes.
If you need to make big changes, make another project.
I kind of wish that if go is going to use github targets for dependencies, that they at least support semver pinning and require tagging in github to support this. There are other issues, but I think that would help a lot.
It's always going to be an issue...
----
Of all the developer package managers I've used npm is probably the best... and even that has some issues.
1. Platform binaries are problematic
2. Nested hierarchies result in unexpected duplications
3. Nested hierarchies result in long paths (windows issue).
1. I'm hoping to see a consistent implementation for tagging modules with binary dependencies, or build dependencies so that there is a build platform in place for at least common targets. Who knows if this will ever happen, and likely will be tied to paid accounts, which isn't exactly a bad thing here.
Like go get, but you specify exact versions, including transitive dependencies. One GOPATH per project. And most importantly, you don't have to use go get to install the tool in the first place.
Ugh, more Linux-Only. Don't follow this please.
(This isn't necessarily aimed only at blog post, but guys and gals there are other OSs supported by Go than Linux/OSX. Most of the time just having "go get" work is enough. Trust me, I've done it a lot as an outlier. I use FreeBSD and Windows.)
What's wrong with using something like party[1], or nut[2]? Or just vendoring in a way that go build still works. Seriously, what's wrong with more source files in your project's source repository?
This way I can go get your command (package main).
I dunno, maybe I'm just grumpy.
[1] https://github.com/jingweno/nut [2] https://github.com/mjibson/party
True on Docker, though.
Maybe once that new fancy OneGet is on Windows (and maybe even becomes as good as brew looks on OSX, which I've never used) it will no longer bother me.
Meh. I get your sentiment here, but I build and deploy on Linux and I post about the stuff I work on. I don't post about the Windows stuff that I haven't worked on in years. And I assume a level of intelligence in my readers that they can translate whatever might be generally applicable into their own platform in the same sense that I don't grumble about Raymond Chen's `Old New Thing` not being directly applicable to my own work but still enjoy it.
Java //src/com/yext/..
Go //gocode/src/yext/...
/gocode/bin/
/gocode/pkg/
/gocode/src/*/
!/gocode/src/yext/
The whole thing works very well at 50 developers and ~100 dependencies.
[1]: http://java.dzone.com/articles/why-gpm-right-go-package
[1] http://blog.rocketpoweredjetpants.com/2015/04/monorepo-one-s...
[0]: https://github.com/git/git/blob/master/contrib/subtree/git-s...
Maybe I have misunderstood the documentation, but what I truly, truly don't get about Go's $GOPATH is that it wants (1) my project directory to have some kind of canonical path, and (2) to pollute my project directory with dependencies. I have done a bunch of Go development and I still don't get it.
So for example, I have myproject, which I naturally want to organize in this way:
$HOME
Projects
foo
.git
src
main.go
stuff.go
assets
photo.png
something.xml
scripts
setup_database.sh
config
config.sample.json
Dockerfile
Makefile
$HOME
go
bin
[...]
src
github.com
BurntSushi
toml
[...]
zenazn
goji
[...]
myaccount
myproject
.git
main.go
stuff.go
assets
photo.png
something.xml
scripts
setup_database.sh
config
config.sample.json
Dockerfile
Makefile
I have tried fixing this by symlinking my project into $GOPATH/src/whatever, but Go doesn't let me to run "go" commands on things outside the $GOPATH, which makes this rather painful to do in a shell.
Adding to the general confusion, Go doesn't manage canonical package paths, so github.com/zenazn/goji/main.go is in the package "goji" and github.com/zenazn/goji/graceful/[asterisk].go are in the package "graceful", but those package names only mean something to the importer. When "goji" wants to use the package "graceful", it imports "github.com/zenazn/goji/graceful", which, being a full URL, must be resolved from a root folder in GOPATH/src.
At least Java (for all its... etc.) had the good sense to make dotted package names mirror the folder structure of a project, not URL, thus being internally consistent. Go wants everything to live in the same sea of things. Vendoring is just half the problem.
It means I have to run full commands such as "go install <my package>" or "go get <some package>". To install all dependencies, I have to do "go get <my package>".
If I want to work on a project that depends on other projects that haven't been pushed to master yet, I have to clone those projects and symlink them in manually. It's pretty painful.
Working with certain tools that assume a certain directory structure (protoc, for example, which computes paths relative to your Go root) is also painful.
"go build ." does work.
"No worse than virtualenv" isn't exactly a rousing endorsement. It's one of the poorer package-management systems out there. Go should be as simple and easy as Bundler. There's no excuse these days, I think.
I did not mean to endorse anything.
https://groups.google.com/forum/#!topic/golang-dev/nMWoEAG55...
>>> ('{:c}'* 3).format(0x74, 0x69, 0x6d)
'tim'https://github.com/tools/godep
Semver support was what got me using godep and I haven't looked back since (at least for larger projects).