Comment by scrollaway | Hacker News Reader

Comment by scrollaway | Hacker News Reader

Parent Full thread

[This question is purely informational and doesn't directly relate to golang, but I want to ask someone who has been involved in domain-specific package management.]

Why is the protocol side of package management not a solved problem already?

At least Go reuses the git protocol to its advantage. But the general problem of package management applies to a lot of things.

"There is a package with a name and a description. That package has versions, which can be greater/lesser than others. Those versions may have deltas from one-another. They contain one or more files. Those files must have their integrity checked. etc, etc..." -> Why does every single language reimplement that?

Nevermind languages actually. Why does everyone reimplement that? Firefox/Chrome extensions, Android/ios app stores, a bazillion different game addons and app-specific stores...

We have a mostly-identical problem being solved a million different ways. pypi, npmjs, glide, they should be running the same server software. It'd be a first step towards standardization and avoid a bunch of mistakes repeated every time a new solution comes out.

Is anyone even trying to do something about this? (and before anyone says "do it yourself", a package management protocol is something I used to work on, not anymore. It keeps getting worse - it's not just a technical problem, also a political one.)

This is a good question. I have a few thoughts.

First, different languages have subtleties that can make a different in package management. For example, the way nested dependencies can work in JavaScript, Go, and PHP are all different. Or, the way you can scan a codebase to determine things is difference.

It would be great if different languages had virtually the same interface. That way metadata and interaction could be about the same. Like diving a car from different manufacturers. In many ways the different package managers are similar enough for programming languages were sorta there (but not all that well).

Second, there is the cultural problem. Many languages are designed on islands from others. Reinventing things, trying different things, and wanting to be their own special flower. Sometimes this is good because a new language is not like the others and that can be a good thing. Other times you reinvent a fairly common wheel.

I'm not aware of anyone solving this in a common way.

At least for the dependency solving part, a generic exchange format has emerged: CUDF (http://www.mancoosi.org/cudf/).

Several efficient solvers have been implemented that supports this input format. You can read about it more in http://opam.ocaml.org/doc/Specifying_Solver_Preferences.html...

[1]: https://www.youtube.com/watch?v=E-gtFnbHcv0&list=UUP9g4dLR7x...

[2]: http://ocaml.org/meetings/ocaml/2014/ocaml2014_17.pdf

This is at least used in OPAM (the OCaml package manager) and is also available for apt. If you are implementing a SAT solver for your package manager nowadays, you are doing it wrong. :)

Sam Boyer, one of the other people in the document wrote a nice article that dives into the complexity of dependency management.

https://medium.com/@sdboyer/so-you-want-to-write-a-package-m...

Discussion https://news.ycombinator.com/item?id=11088125

Sam is currently working on a library for Go https://github.com/sdboyer/gps

In fact, here's Sam and wycats (of Cargo) talking about how a generic library (obviously Go is not the best choice for that) would be nice: https://twitter.com/sdboyer/status/755036174016806912

> At least Go reuses the git protocol to its advantage.

I think this is a mistake. Packaging should come from a central server (that can be overriden with a mirror). Allowing downloads from adhoc git repositories is a real pain. First, it makes mirroring the package repository impossible. Second, it makes it difficult for companies to disallow certain packages in a centralized fashion. Third, it makes distro packaging a royal pain. And finally Git simply wasn't built for the this kind of work, compressed tarballs happen to work great for this purpose. Git's mutable history also presents other challenges for pinning to specific versions of packages. Simply put, sometimes a central authority has advantages.

That all said, vendoring alleviates most of these problems for apps and some of them for libraries. Reusing tools does make it a little easier to get started with too. I'd hate to see these features removed in whatever go get becomes in the long term.

> That package has versions

This is a statement that doesn't always apply.

For example, some packages don't have versions, they only have a single version: the latest version. It's a moving target, but it's there. Older versions are just an unimportant and unsupported implementation side-effect.

Of course, many packages do have versions, but this one factor makes a big difference on how you look at vendoring.

First, your described software still has versions. Each daily snapshot or git commit or whatever is a new version. There's no reason you can't put an auto-incrementing version number on each release and call that your version number. You can even stick a "0." on the front of it and be SemVer complant.

Second, just because some people do this doesn't mean it's a good idea. If you're writing a full application and just want to distribute it, do whatever you want with your versioning (although using a dependency management tool to distribute your webapp or CLI tool is probably not a great idea in the first place), but if you are writing something to be included in another project please use a sane versioning system. Anything else is selfish and disrespectful of your users.

In Go, each version corresponds to a given import/Pkg path.

If you want to change version (major), you need to create a new package.

The advantage is that you do not have to download a manifest such as package.json or whatever. It also provides a constraint on library authors to provide a backward compatible API for their published package.

Eventually, I guess we could use go/types to enforce/check the API backward compatibility requirement.

The main issues for now are:

* make sure that packages importing a same vendored one are able to use the same latest minor version in use.

* allow a project based approach to go get for people who need reproducible builds (or alternatively allow for multiple $GOPATH and make being able to switch between them easy)

Without versions you don't have package management, you have a download script.

Agreed, and you also cant guarentee reproducable builds. Builds that are dependent on when you compile them, even if no changes have been made due to ephemeral dependances are a nightmare.

There are many people who need versions and don't run the tip of master. Even for projects that always try to keep the tip of master as a stable build.

I think of OpenStack here. Multiple times per year they produce releases because many people need them. Yet, the CI/CD system keeps it in such shape you can always run the tip of master and some people do that.

Both options need to be available.

To not have release versions means you're not willing to support a certain class of users. It says something about your intent.

There's Molinillo used in CocoaPods and Bundler: https://github.com/CocoaPods/Molinillo

When I wrote "Seeing How The Other Side Lives, A Package Manager Overview For Go Developers"[1] I noted CocoaPods. Was not aware of Molinillo though. Thanks for sharing.

[1] http://engineeredweb.com/blog/2015/pkg-mgr-overview/

Reply on news.ycombinator.com