Manul – the madness vendoring utility for Go with Git submodules | Hacker News Reader

Manul – the madness vendoring utility for Go with Git submodules

github.com

97 pointskovetskiy58 comments

"Pros" are listed, but "cons" are not. While evaluating git submodules as a possible solution for vendoring at my work, I found the following disadvantages with this model:

- you can use only git dependencies (no bzr, hg, ...); though most stuff is on github nowadays (especially with closure of Google Code), still not necessarily everything. With pure submodules you're locking yourself in the git monoculture.

- you must trust your dependencies will not ever disappear (see "the leftpad problem", or the mentioned closure of Google Code).

That said, all approaches have pros and cons. Though I admit I usually don't like advertisements which only show advantages and don't list downsides.

Also, it is not possible to express constraints (eg. follow 2.x tags). A manifest file is handy for that.

Personally, I like glide better since it takes a lot from Ruby bundle. I commit only glide.yaml and glide.lock in the repository. It is not flawless either as each operation triggers an update of all dependencies.

Another con is that submodules will be missing in the tarballs that Github autogenerates for tags and releases. I initially used submodules for my own Go binaries, but moved to a custom vendoring solution since then.

Wouldn't git subtrees be a better option than git submodules?

(see: http://blogs.atlassian.com/2013/05/alternatives-to-git-submo... and https://codingkilledthecat.wordpress.com/2012/04/28/why-your... )

These are little pleasures and forgive the tiny vanity moment: I wrote the first article you linked. I also wrote more recently a follow up on just subtree: https://developer.atlassian.com/blog/2015/05/the-power-of-gi...

I remember reading that article a year ago. Thank you for it, it's served as a reference for me since :)

git subtree have one bis disadvantage, git log will look like a hell, because all subtrees' commits also will be shown.

Unless you use --squash, but you have to remember to pass it.

I found that this makes rebasing a feature branch really difficult. The alternative is not much better.

I'm using subtrees to vendor C dependencies and it works well. Submodules are problematic as they may disappear or the git repo might be down. If your dependency is read only, subtree is a better option IMO.

Golang doesn't need more package management tools - it needs a standard build-in package manager! My vote goes to: https://glide.sh/

Glide isn't ideal either. It gets confused about vendored dependencies by 3rd party libraries and tries to download them separately. Also, it's invented it's own format for the deps and lock files instead of trying to standardise on the ones already out there.

I thought Golang already had its own package manager: "go get X". Or is that not enough?

It's not enough. `go get` understands only how to pull the latest version of the master branch of things.

Quite interesting, I would vote on that one as well.

Is go package management still an unsolved problem then?

Yes. It's a terrible mess. But as of GopherCon a couple of weeks ago, the way forward is becoming clear. Most important, the Go Team is now in direct communication with the authors of the most widely used vendoring tools, and it seemed to be widely acknowledged that this is a severe problem, that a standard is needed, and that even things like "go get" might have to change. I'm very hopeful.

I'd much rather them forget reverse compatibility and fix the core problem once and for all. I personally like gb and the ability for each project to be its own GOPATH.

Being willing to change `go get` is forgetting reverse compatibility.

Do you know if the talks will be made available?

I haven't found anything on the conference site.

Unfortunately, the discussion took place on the "Hack Day", and wasn't recorded (I think they're going to try to change that for next year). Chris Broadfoot took pretty comprehensive notes, though: https://docs.google.com/document/d/1xMJ0c-YxvcgNglzjbALzncs5...

Wow! Thats a great write up there. Excited to see some action and discussion on this problem.

Thanks

Putting your dependencies in a folder called "vendor/" is standard, but how you get them there isn't.

There's a collection of tools for managing that, like this one , and others like "godep" and "glide". Which one you use for that doesn't seem to be settled.

Yes, it's still an unsolved problem. All attempts at solving it feel like ugly hacks.

Manul is one of the very limited set of tools which provide idiomatically go-gettable projects without need of vendoring tool be installed on the user machine.

Shameless plug: I also wrote one, with the added bonus that it does not rely on Git submodules: https://github.com/holocm/golangvend

It is "solved", the problem is go devs using 15 incompatible solutions.

> What's the reason for yet another utility?

> Some copy the source code of dependencies into the vendor directory:

I don't agree with any of the stated problems there. Unless you are building a library you should commit your dependencies and not rely on third party repos being available.

I can't stress enough how useful govendor [1] is if you do Go professionally.

[1] https://github.com/kardianos/govendor

Copying the source code of the dependencies into your repo isn't the only alternative to relying on third party repositories, though.

You could fork the dependencies that you work with into locally managed repos and use submodules to include them in your own project. That way you can easily update your dependencies without breaking everything and also maintain their commit history.

That's how we started back in the day. It works but setting up everything and doing the rebase/commit dance to update can be quite cumbersome when you have more than a couple dependencies. Nothing wrong with it if you are willing to do the work though.

Does the go toolchain allow you to freeze your dependencies to specific versions like pip, etc? I think that could be preferable to committing all your dependencies.

The go toolchain doesn’t but many of these vendoring tools do something similar. But you still don’t want to depend on the origin repo being up and have missing/altered repos break your builds.

This doesn't solve the higher level problem.

If you depend on A and B and both depend on C, you now have the npm2 style heirarchy of nested dependencies.

As with all 'use submodules' solutions, the devil is in the detail with the nested heirarchy of dependencies and version resolution.

I mean, fwiw, its quite nice for simple 1-level-deep dependencies, but this isnt a simple problem. What if A -> C@1.1.5, and B -> C@1.1.9. Do you install both? Only one?

You might still be able to solve it somehow, if the submodules are required to have semantic version tags on their versions, you parsed the entire hierarchy and then resolved conflicts (somehow), based on the semver semantics... I guess?

That's outside the scope of this tool. Go has already adopted the npm style submodules that you mentioned (I.e. Answer = you install both versions in their respective vendor/ dirs).

This tool just tries to help you manage that. Not that I'm a big fan, either way, but still.

If you're not a fan of submodules, but are in need of a this kind of tool, I'd recommend looking at `glide` [1] . I've had much better experiences, as a user, with `glide` than other tools, and it supports package aliasing which is really helpful. (Package aliasing allows for satisfying an import path with a private fork of a package, without having to rewrite all the import statements inside my fork.)

[1]: https://github.com/Masterminds/glide

I'm in the same boat - glide is the one that has Just Worked for me and my projects, so seeing some consensus around it would be nice.

Vendetta does exactly the same thing: https://github.com/dpw/vendetta (but seems to have been developed after Manul).

I've used git subrepo [1] for vendoring and been quite happy with it. Doesn't seem to have the problems of git submodule and it's easier to use than git subtree.

[1] https://github.com/ingydotnet/git-subrepo

I'm still hoping for all the great ideas from nix [1] and guix [2] to catch on.

Package manager shouldn't be that thing, that tries to integrate into existing fragile environments. It should be the thing that manages those environments, that manages all the compilers and libraries in a reliable and reproducible way, with rollbacks and everything.

[1] https://nixos.org/nix/ [2] http://www.gnu.org/software/guix/

While the gifs are perfect to demo how a tool works, it hard to perceive when a user needs to find solution for a concrete case.

As someone who has thought Go was a poor choice for almost any project for a long time and argued that many times, I used to come up against the argument that Go package management was superior to other languages. Six months ago I said:

> Packaging and deployment aren't problems in Go OR Python if you're only solving trivial problems.

> With a larger project, you might have an easier time packaging and deploying Go code, but that's largely because there are no libraries to package. Admittedly packaging can be a pain in Python, but that's usually because of poor choices in dependencies. Packaging Python with a few mature dependencies isn't hard in my experience. It's the projects where some idiot has pulled in every 0.x versioned library in pip that are hard to package. When Go has as wide a variety of libraries as Python people will run into the same problems in Go.

> You could argue that at least for now Go doesn't allow you to shoot yourself in the foot that way, but I'd rather have the option.

> I'm not defending Python in particular here. I'd say the same things I've said here about any mature language with extensive libraries.

> If you're espousing Go because of easy packaging and deployment, I strongly suggest that you consider whether that's actually a feature Go will have long term, and whether you're currently paying for that feature by having no libraries available to you. The only lesson I would take away from Go's easy packaging is to only use mature dependencies that pull their weight.

And in a different post:

> give it a decade and Go packaging will be just as miserable as packaging in any other language.

I've been coming up against that argument a lot less lately. It looks like my prediction is coming true a bit ahead of schedule.

I thought vendoring means, you are safe even if the original repo was removed from GitHub. This tool still looks nice for a Wrapper to handle submodules for you in a Go way. But still you are only safe, if you fork and keep your fork up to date and "vendor" your fork using Manul.

This is true but it's only a problem if you don't have a copy of your repository checked out anywhere. You can always make a copy of the repository later as long as you have at least one checkout.

`go get` has to change for proper vendoring in Go. Right now its such a mess. Go needs a bundler like functional. e.g go bundle which will read a Gemfile like file to install your external packages.

I hope this comes before the end of the year. This has been put on the back burner for too long and I can understand GC was priority but now GC is very good.

For projects where all of the dependencies are git, I use git-freeze[0] to vendor into submodules. Works great, and no external package manager dependencies for consumers.

[0] https://github.com/nicerobot/git-freeze

What so what happens with the libs that aren't version controlled by Git... say Mercurial?

There are interop tools to/from Git to pretty much all the popular version control tools, including Mercurial. They might not work out of the box with Git submodules, but you should be able to set up a mirror of the original repo in Git.

Not ideal, but not a deal breaker either.

Erm, as a developer if I use a package manager that requires me to mirror the original repo then I don't think I'm going to use it at all. I love the idea of git submodules for handling dependencies, but the assumption here is that all packages use git. This is a minor, but definitely a fallacy when it comes to using some of the packages from Google, mainly legacy.

Edit: IMO it's a deal breaker

It's a good idea to be doing mirroring anyway if your business depends on it. Otherwise the original repo disappearing (or corrupted otherwise) or network going down is going to hit your productivity. And things are faster with mirroring.

And yes, it's annoying, but a workaround exists.

This is the tool I was hoping would exist. Looking forward to trying it out.

Other than being able to clean up unused dependencies, I don't really see any advantage of this over just using git submodule directly.

because git submodules are the f*cking devil

I partially agree with this thesis, but I dunno any solutions that better than git submodules. Everything is just the worst, manul just sucks less.

I'm no fan myself, but please don't make HN worse because you don't like a software feature.

Did you mean sob modules?

So much for reproducible builds. What if the internet connection flakes on your build server. Or if the source to the library is deleted from wherever you're pulling it from.

Why the reluctance to add ALL SOURCE REQUIRED TO BUILD YOUR APP to your source repository. I don't get it.

Also, no matter what vendor tool you use, please use one such that I can "go get" your command (main package).

Although we're really at the point where I don't care what tool is used, so long as the community can pick one.

> Why the reluctance to add ALL SOURCE REQUIRED TO BUILD YOUR APP to your source repository. I don't get it.

That would make the Git history incredibly messy to work with and makes the repository much larger in size, making work slower. This kind of "monorepos" are known to be a pain in the ass with Git.

However, Git submodules is pretty much adding all the required source code (or at least pointers to) in your repo, and pinning them to a specific version.

The network connectivity issue is solved by having local mirrors.

Reply on news.ycombinator.com