Hacker News Reader

Comment by arianvanp | Hacker News Reader

The go vendoring stuff and GOPATH sounds like a complete nightmare to me. Also the fact that imports are full git URLS sounds totally crazy. Is `dep` fixing all that stuff? Last time I checked out go a few years ago, I didn't continue because I found the story of actually installing dependencies and keeping track of versions very confusing. You couldn't even `go get` a specific git tag or something.

I'm happy that there seems to be some movement. But now we have godep, govendor, glide, dep. and every project uses one or the other. Are they all compatible? Or is it a minefield?

View on HN

Go vendoring gets such a bad rap undeservedly. In practice it's the simplest and most straightforward dependency system (since 1.4/5 added vendoring) I've used.

Step 1: Check out code that you want to use. That's actually the only step and you can do it by hand, or with one of a multitude of tools. All the tools differ slightly, and any metadata or manifests or things like that they do aren't compatible, but the code they put in your repository is, enabling anyone to clone a repository, and instantly build the exact version of all your dependencies.

Imports also aren't full git URLs, they can be as a convenience that tells you where to get a package. You can't `go get` a specific tag, but that's fine, because `go get` isn't a package or dependency manager, it's just a convenience to grab code to your system, not to freeze a version as a dependency into your project.

Frankly, the amount of people that put their nose in the air at go is just fine with me, just makes it more of a 'secret productivity tool' I guess.

Edit: not to say it wouldn't have been nice if `dep` had been the one blessed path at version 1. Still, I have extremely little to complain about with the language and tooling since 1.4/1.5.

The fact that the full url is used to import packages make it impossible to fork a repository.

Once you forked it, you have to change all the imports across the repo, and then it's kinda either very hard to make Pull Requests, or pull the new commits from the original repo.

This is one thing I really dislike about go dependencies.

If you have a solution for this, let me know, I would be very interested to know more about it.

But overall, I love go. I use it all the time anyway.

You don't have to change the package name just because it's forked.

I threw together an example of doing this with `dep`. I forked Fatih's color package, and am using it as `github.com/fatih/color` no problem: https://github.com/sofuture/colortest (forked dep here: https://github.com/sofuture/color)

As I said in my other comment (https://news.ycombinator.com/item?id=15632049)

It works for you because it's a small project and you are not actually importing other of your own packages inside it.

But if you look at "echo", it import some other "echo" packages inside of it.

That doesn't matter is what I'm saying :) The package name of my fork isn't changed, and doesn't need to be, regardless of what else uses it. The only thing I had to do was specify 'use my fork's repo' in the Gopkg.toml file.

Edit: I'll fork echo in my example and show you.

Okay, updated my example with `echo`. Check it out:

Using the original package name: https://github.com/sofuture/colortest/blob/master/main.go#L7

I can refer to my fork: https://github.com/sofuture/colortest/blob/master/Gopkg.toml...

Oh I see.

Thank you for the example :)

You don't have to use the full url to import a package. The name of a git repository doesn't have to match the remote name though, so I don't see why it would be a problem to fork a repository.

Let say I would like to fork "echo".

Echo is using some imports to their own packages inside the project.

IE: https://github.com/labstack/echo/blob/a098bcd3b0c445dde3d380...

So if I fork "echo", I have to replace the urls in their imports so it imports my fork packages.

I agree venturing is actually good with just a simple folder, though IMO (with the benefit of hindsight), gopath was a mistake, and a per project vendor folder for dependencies would make much more sense for collaborative projects outside large companies.

Go get does have support for tags, but it was intended for they have code in go get to find tags like go1 and go2 which are as yet unused (I wish they'd used it for dependency version tags like v5.2 etc instead, it may never be used):

https://golang.org/src/cmd/go/internal/get/get.go#L524

Really some simple additions to go get would have gone a long way - recognise semantic version tags like v1.2 on go get and put them in vendor with some command like 'go vendor my/dep -v 1.2'.

Not really sure they need diamond dependencies, updating them automatically up to version x, manifest+lock files and all the other intricacies which in theory a package manager should solve - humans can resolve them as they come up and in real life use they're not a huge deal (as the incredibly simple go get we currently have shows).

GOPATH isn't a real problem; it's just a search directory like PATH or PYTHONPATH or CLASSPATH or NODEPATH, and it has a default value that makes it painless. Vendoring can be truly painful, but `dep` aims to fix that. Imports are not git URLs or indeed any kind of URLs; they're just directory paths into your GOPATH which the `go get` program can use if the directory path resembles the URL of a git (or hg/bzr/svn) repo. Don't think of `go get` as a package manager; think of it as a convenience utility to play around with a package before locking down the version via vendoring. Go's package management has a lot to improve on, but many of the common complaints are misinformed.

> I'm happy that there seems to be some movement. But now we have godep, govendor, glide, dep. and every project uses one or the other. Are they all compatible? Or is it a minefield?

They're not all compatible, but it's not really a problem because (last I checked) the prevailing philosophy was "libraries should not vendor--only binaries" which is to say "libraries shouldn't specify versions for their dependencies; the downstream binary projects should figure out what versions of each library should be used to build the binary". This means dependency tool compatibility isn't a problem because the dependency tooling punts on the dependency-resolution problem altogether, which seems like an even bigger problem.

Remember that this is only painful if you're trying for deterministic builds, and in the absolute worst case, you forego Go's build tooling and use something like Nix. This is absolutely not a reason to change language.

The existence of the GOPATH environment variable is not a problem. The fact that the toolchain is violently opinionated about where you locate your working copies, and goes out of its way to fight you if you try to fake it with symlinks, is.

The GOPATH issue will be solved when I can clone a project to anywhere in $HOME and build it without issue.

This is how I felt when I started using go, but I was unable to rationalize why beyond 'it's different'.

What's the real difference between `cd ~/dev` and `cd $MYGO` (~/dev/go/src/github.com/myusername)?

The difference is that I don't want all my Go programs in a different place. I have my place where I put projects. I have ~/projects, ~/oldprojects, ~/defunctprojects. I have ~/contribprojects for things that aren't mine. I decide where my files go.

Well imagine you use language foo, bar and go. Go insists all code lives in ~/go/src and foo insists it all lives in ~/foo/source. Neither likes symlinks, which one wins?

Also people like to keep other artefacts alongside source code like scripts, resources, templates, tools etc and often have an existing elaborate project structure in order to accommodate that. Go forces them to throw that out and use something under gopath. It's annoying and an unnecessary stumbling block for people starting out in Go.

Go doesn't insist that your code lives in ~/go/src. It can live anywhere beneath a directory of your choice. Note that there's no requirement that GOPATH contains only Go source code. You can put your scripts and resources, as well as other source code, right next to the Go sources.

So you just put your code into ~/src/project/a.go and ~/src/project/b.foo (assuming a GOPATH=$HOME).

Note src vs source in the original comment. Go forces you to change everything else to conform to gopath. I use go a lot and am used to it now but IMO that is a Bad Thing.

Put Go stuff in $GOPATH/src and Foo stuff in ~/foo/source. Problem solved. Go doesn't dictate where non-Go code lives...

This is the least of all problems.

So now your go stuff and foo stuff lives in different dirs based on which programming language it happens to use, not which project it is for. To you perhaps this is a non-problem, to others it appears to be and it is a recurring cause of grief for newcomers to go.

    export GOPATH="~/go:~/foo"

GOPATH will use ~/go as default for go get and importing and fall back to any further folders specified.

The difference is the Go tooling is dictating your whole code layout that may not match your personal preference. I work in a multitude of languages and I organize my code in ~/code/work for work/contracting code bases, ~/code/personal for my code bases and ~/code/third-party for source code I want to look at and maybe contribute casually to. I have never figured out a good way to fit GOPATH into this without completely rearranging how I organize my code, which I have zero desire to do for one language, especially one that's not my main.

You can specify multiple GOPATHs which will be searched in order for packages;

Your GOPATH could be set to "~/libs/go:~/code/work for work/contracting/go:~code/personal:~/code/third-party"

which would put any go-get libaries into ~/libs/go and allow using packages from the other folders. Keep in mind that Go will still use these Gopaths like any other Gopath repository; create a src/pkg/bin folder set for use with the compiler

That's good to know, I had never come across that! Thanks.

> The go vendoring stuff and GOPATH sounds like a complete nightmare to me. Also the fact that imports are full git URLS sounds totally crazy.

It is.

> Is `dep` fixing all that stuff?

Not from what I can tell; it lets you pin to git tags and branches, but the only versioning you'll get beyond that is if the maintainer of your dependency is gracious enough to use tags. It certainly isn't changing the way import paths work. Vendoring is still part of the workflow for projects I've seen using dep. I've heard there's work being done to get rid of the GOPATH nonsense, but from what I can tell people have been saying that for a while, so I'm not optimistic that it'll land any time soon.

'dep' does have plans to address this, but yes, it is not going to happen soon.

'dep' seems great for Noddy projects where all the code you are working on is in a single package, and fails (like most of the vendoring tools) when you need to work on multiple packages. It is has unfortunately been focused on vendoring at the package level rather than at the project level, and has many assumptions buried deeply into the code. The recommendation for larger projects is the 'vg' tool, which wraps 'dep' with some GOPATH tricks to help manage your dependencies at the project level.

What's wrong with vendoring? You put the package you want inside the vendors/ directory. I think it's much better than a package manager that tries to download some version that satisfies some version specification in a file that declares dependencies.

If the new version has moved from Git to new suppa-duppa source control, you will need to update all import statements after replacing it on the vendors directory.

With proper package names via a package manager, usually the name stays the same.

Managing vendoring yourself is impractical once you end up with more than a handful of dependencies, or once one of your dependencies pulls in more than a handful of transient dependencies. A larger project will have hundreds of dependencies and transient dependencies, and if you just pull them into your vendor directory and forget about them then you have a problem waiting to happen.

I find the opposite to be true. If you casually add dependencies without understanding the dependency tree, you have a lot of problems waiting to happen.

I prefer to be conservative about adding dependencies, and to always manually include them into the project.

You can be conservative about adding dependencies and still end up with hundreds. Database driver, database utilities, gRPC framework, monitoring & metrics, CLI & configuration tools, /x/ packages such as crypto or net, logging, linters, libraries to talk to other moving parts like Rabbit or Vault, and then all the transitive dependencies like yaml, toml, websockets, and the various 'standard' helpers the authors of your dependencies use... all for a microservice and administrative tool... and one part of the larger project.

That's the definition of not being conservative about adding dependencies.

Oh, that could certainly be a conservative list. It depends on the requirements of the software. Drop any item on that list and you are either losing capabilities or have more software to implement yourself. And it is certainly tempting to reimplement things like Cobra, Viper, sqlx, prometheus end point, and everyone knows we should all implement our own crypto, but overall it ends up costing you time, and probably usability since your time is limited.

That's definitely not conservative.

Maybe you don't know what a conservative attitude about libraries is because you've never seen one.

Your original comment:

> You can be conservative about adding dependencies and still end up with hundreds

No. Just no. If you have hundreds of dependencies, you are by definition not conservative about adding dependencies.

> Database driver

About the only legit thing on the list.

> database utilities

Not needed.

> gRPC framework

Not needed.

> monitoring & metrics

Not needed.

> CLI & configuration tools

NOT NEEDED!

.. etc.

> Drop any item on that list and you are either losing capabilities or have more software to implement yourself

What are you losing by not using grpc? You can do a lot of things by just using the builtin rpc package or the encoding/gob with the http package.

Being conservative about adding dependencies means resisting the urge to add them and figuring out a way to do without them.

Every dependency you add is a potential point of failure and headaches.

Just like every layer of abstraction you create.

The cost for dependencies is huge, so every dependency has to justify itself by giving such a huge benefits that it's worth the cost.

> Cobra

Do you need a special library to handle command line arguments? How complex is your CLI interface? How many commands and sub commands do you have? 10? 20? It's trivial to handle that much manually using the builtin libraries.

> Viper

How hard is it to just read a json file using builtin libraries? How much do you lose by not using "Viper" or whatever?

How many times will you have to read config files? One time when the app launches? Does not justify adding a dependency.

> it is certainly tempting to reimplement things like Cobra, Viper, sqlx, prometheus end point, and everyone knows we should all implement our own crypto

You don't even need to re-implement anything. Just do the simplest thing that works using the standard library.

You know the standard library comes with a crypto package, right?

The mindset you are presenting is the exact opposite of conservative. You are just grabbing every library that you think can save you 20 lines of code.

Yes, you can certainly be conservative by ignoring the requirements of the project and throwing away features. It will be a complete failure, because it doesn't do what it is supposed to, but it won't have any dependencies.

I will stop instrumenting my projects because its not needed, no matter what operations says about our SLAs, and my gRPC endpoint will stop talking gRPC because its not needed, and it will be more reliable because clients can no longer talk to it by the expected protocol, and I'll ignore the usability requirements of the CLI and configuration management because specification documents and usability requirements be damned, and I'll implement by own helpers for the database code because NIH syndrome is fantastic, and I'm sure my employer will be happy for me to reinvent the wheel. Except I probably won't have an employer any more, because I seem to have stopped delivering the software they tasked me to.

No, it will not be a complete failure when you are careful about what you include.

It will be a complete failure when things get so hairy with so many layers and failure points that you can't tell what's going on anymore.

i believe dep and glide allow you to pin to specific versions. i actually really like that imports are URLs, makes it really obvious where packages come from, and it's a better system imo than a centralized package manager like pypi / rubygem where names need to be unique.

> it's a better system imo than a centralized package manager like pypi / rubygem where names need to be unique

I'll have to disagree there; while having namespaced packages gives some advantages, having a centralized repository gives you things like proper versioning, which is worth all the terrible package names in the world to me.

A central repository doesn't automatically give you versioning, and URLs as package IDs doesn't take it away. These are orthogonal issues. It just happens that Go has chosen an approach for now that results in a lot of pain.

I don't think they're completely orthogonal; I think it's much easier to enforce versioning with a central repository than URLs as package IDs. It's not a coincidence that the most common centralized package managers (e.g. Ruby, Python, NodeJS, Rust, Linux distro package managers, and brew) all enforce versioning and things like Go and Vim packaging don't.

Go and vim don't have "distributed package managers"; they just punt on package management altogether. The next step up from that is a centralized package management system, and if you care that much you'll probably build in some notion of versions (even if they're nondeterministic or otherwise broken, as in the case of pip, npm, and many Linux package managers). Building a distributed package manager is quite a lot more work.

Reply on news.ycombinator.com