A Roadmap to Merging 'dep' into the Go Toolchain | Hacker News Reader

A Roadmap to Merging 'dep' into the Go Toolchain

github.com

138 pointsnumo1644 comments

This can't happens fast enough.

I have been having a fantastic time with Go... Up until the point I have to add a new package to my projects and go through CI and deployment.

Glide works mostly OK, but it's not the final solution, that's for sure. Even big packages on github fail to produce releases, so you have to depend on master branches... One day your builds just decide to break. It's a nightmare.

I haven't used Glide (only govendor) so I'm not sure exactly how it works.

Are you checking your dependencies into your repository (we do this, it works very well).

If not, are you pinning specific commit hashes? Is that even possible in Glide?

AFAIK those are the only two ways to get even close to reproducible builds.

Commit hashes seem to be the way to go, for sure, at least to be safe. I really don't want to put our vendor content in the repos.

It's just a tad absurd when you're used to mature package managers in other languages.

Honestly, most package managers just bring their own problems. I'd rather commit source code than troubleshoot npm, pip, nix, maven, etc errors all day long. The only package manager I've found that works reliably has been Cargo (Rust).

There has been a lot of talking between the committee that created `dep` and folks who've implemented other systems, including Cargo. I have high hopes.

indeed, we've talked with them quite a bit :)

I don't think I've ever had a problem with Bundler.

Me neither. In fact, I rarely have trouble with package managers. They're amazing.

I envy you. I use Nix, pip, and npm on a daily basis and it's an uphill battle. I haven't used Bundler.

Well, nix looks awfully complicated. I've had trouble deploying applications using pip. Npm hasn't given me any trouble.

Bundler is lovely.

Once I did 'sudo npm -g update npm', just 3.x to 3.x+1 and the hell broke loose. I had to wipe entire nodejs and reinstall from scratch.

In other cases, 'npm update <package>' doesn't work and I have to do uninstall/install. Very common with typescript, for example.

i would be a happy man if every package manager for any language has magically become cargo overnight.

Until cargo starts to support binary crates across projects I would be a very unhappy man.

I have better things to do with my time than watching the same crates being compiled multiple times.

Well go builds so fast you wouldn't be waiting.

It is still slower than Turbo Pascal (MS-DOS). :)

In any case, I appreciate that Go at least supports binary packages, but Go isn't a language for me, given the type system decisions.

I believe the current intended use is a human-editable "manifest file" where you specify desires/intent, preferably semver, ideally only as necessary, and a machine-generated "lock file" which specifies the actual hash of every transitive dependency, so builds are fully specified and reproducible.

But they all have to vendor as well if they want reliable builds. Otherwise you end up with everything breaking when a developer decides to do a forced push or take his repo off github.

Well, you can vendor, or you can have an internal mirror/cache of repositories.

Force pushes - by far more common than straight-up repository removal - are handled without problem; we let you stick with your old version. (At least, that's how it should be - there might be a couple more test cases to write. I know I designed for this problem early on).

Repo removal, renaming, or whatever, are still problems, for sure.

Today, dep populates vendor/ with dependencies, and works equally well whether you decide to commit them or not.

Not if there is a central repository which doesn't allow removal of packages.

> Are you checking your dependencies into your repository (we do this, it works very well).

it work if you are working on a project, it does not if you are library author who rely on other dependencies.

in a post-dep world, this won't matter anymore. dep strips vendor directories out of any dependencies it pulls in.

When dep removed the vendor folders from the other libraries, How does dep handle multiple libraries that depend on the same library but different versions of it?

What I don't get is why more folks don't just use per-project GOPATHs.

I have all my stuff in $PROJECT/src/$PROJECT, and all the dependencies are git submodules in $PROJECT/src/, and it all Just Works. Perfectly. git submodule is a bit of an annoyance, but it works. Anyone fetching my project just checks everything out, and it all Just Works™.

It's a lot better than using a dependency-management tool and shared GOPATH, at least in my experience.

> What I don't get is why more folks don't just use per-project GOPATHs.

because that's not how go works. You need extra work to make it work. That's not up to debate anymore anyway, dep will be the official package manager and will be merged with the rest of the official ecosystem.

What do you mean 'that's not how go works'? Go provides GOPATH: using a different GOPATH for each of my projects works, and it works really well.

> You need extra work to make it work.

What extra work? I set GOPATH, and everything just works. What more is there?

> That's not up to debate anymore anyway

Neither were type aliases, until they were, except they aren't.

I have no idea what dep adds that using the standard GOPATH doesn't do: I've read the background and the docs, and I'm familiar with using the vendor directory in the past; frankly, now that I've switched to using GOPATH all of those things look like a mistake. It all reads like tha Javafication of Go.

Glide lets you pin dependencies to specific commits (the "version" key takes a semver version constraint, a branch name or a commit hash), and transitive dependencies will be pinned via the lockfile, so it works even if the implicitly included dependencies don't use versioning.

The main problem with Glide is that it's buggy. It also lacks some important features. You still can't do "glide update" on a single dependency, for example. It's all or nothing.

yeah, this is one of the main pain points there. the reason it's so difficult to solve in glide is intrinsically tied to the engine glide uses for version selection - otherwise we'd have dealt with it long ago.

I agree. I have a friend who likes the concept of Go but refuses to try it out unless it has a good dependency management solution.

This podcast (iOS) is really worth listening. Straight from the horse's mouth:

"Dependency Management, Semver, and Community Consensus" https://itunes.apple.com/my/podcast/go-time/id1120964487?mt=...

It explains how go dep fundamentally works alongside the other tools.

Web version of same podcast https://changelog.com/gotime/36

If you have any skills or interest in this area, the dep team could use your help. Even just trying it out, running into trouble, and filing bugs would be helpful. Join #vendor on the Golang Slack.

Great initiative. If it goes well we hopefully will have an integrated dependency management solution with go command by next year this time.

The `dep` repository: https://github.com/golang/dep

This podcast (iOS) is really worth listening. Straight from the horse's mouth:

"Dependency Management, Semver, and Community Consensus with Sam Boyer" https://itunes.apple.com/my/podcast/go-time/id1120964487?mt=...

Started with dep recently on a real go project at work. It worked initially fine until I started adding dependencies and I had to hunt down the right commit hash by looking at the dependencies. I am probably going to stick with it and maybe contributing upstream as well.

Anyone know how dep's approach differs from the two-step process in `gb`?

(gb vendor fetch github.com/some/pkg, gb build, git commit -am "all of your vendor'd source")

sadly, gb is one of the extant systems i haven't had a ton of time to explore. but...

i guess the analogue to what you're describing would be

1. <add an import path> 2. `dep ensure` 3. `git commit -am "all of your vendor'd source`

a bit more detail, starting at a high level: gb is a replacement toolchain. dep is focused strictly on dependency management. there is no notion of a `dep build`.

with dep, there's no explicit command to actually fetch a dep; the import graph is still queen, as is customary in go. so there's no direct analogue to `gb vendor fetch github.com/some/pkg`. if you want to add a dep, import it in your source code, and run `dep ensure`. (there's some flux in exactly how that works right now, but what i'm describing is the state we're moving towards)

`dep ensure` is really the workhorse command, and pretty much the only thing you'll ever need to run. (in fact, the only three subcommands we currently plan on having are `init`, `ensure`, and `status`).

`gb` has a package-management-only version in `gvt` by @FiloSottile .

In summary, via gb:

    git add vendor/ && git commit

In dep:

    echo vendor >> .gitignore

Ie. Dep is designed to easily restore specific versions of packages online, like other package managers. Gb expects you just commit them.

Practically speaking I guess the manifest that dep uses makes it easier to see at a high level exactly what versions of what packages are installed and from where.

Our team runs _the_ largest Go middleware. Checking in external dependencies into your repo scales - you don't want 200+ users hitting Github every 10 minutes, when you can hit the local Git repo on a 10Gbps n/w. Also, using GOPATH with a simple Makefile is pretty much all you need -- unless you don't grok Makefiles. But whatever floats your boat and as usual the latest hotness sells.

I use gopkg.in. Personally I would like to see the community avoid adding more to the 'go' tool. It is already doing too much. I think the solution provided by gopkg.in is better than other more conventional methods (e.g. npm or maven like). I love Go. But it is ironic that 'go fmt' was designed in from the beginning. And now we are still dealing with semver and package dependencies. I went with gopkg.in because it made the dependencies a non issue from the tooling point of view. It is made possible by having 'import' direct from the repositories, which archive different versions and tags. Perfect for something like gopkg.in. I am probably in the minority.

You say you don't want the go tool to be more complex -- but gopkg.in just moves dependency management elsewhere. You have to have it somewhere.

gopkg.in only works properly if everyone uses it. The second you have a non-gopkg.in package, you have no way to manage that dependency.

Does gopkg.in work for non-public libraries?

another problem with gopkg.in it uses only the major version number and you can't import a specific minor version. what if version 2.5 works great but 2.6 breaks something for you? you can't use pkg.in until this issue is resolved in 2.7 .. and most package authors don't want to bump the major version on every change ..

Reply on news.ycombinator.com