Hacker News Reader

Comment by bobjordan | Hacker News Reader

Recently found a command line library written in go that was so useful to me in a python webapp I'm building, I decided to just wrap the go app and call it from a python subprocess. It is a bit slow to do it this way, but premature optimization, and all that. Anyhow, the bottom line is due to this I was forced to get a handle on Go this week so I could at least build the go library and make a few changes as needed. Wow, was I absolutely shocked to find out current package management in this go app is just pulling libraries from master on github. It really made me feel like go is far behind python in maturity. Pulling all these dependency libs from master definitely does not feel production ready. Maybe this will help resolve things.

View on HN

I was likewise shocked at how good go compared to python in this regard.

Resolve your deps once, at build time, and you're done.

Compare that to dealing with the externalized deps that is deploying a python app on a machine that might have the correct version of python and all dependencies, or not. virtualenv? good luck. You end up bringing in docker and other tools just to deal with it. But what do you do if the machine is locked down by corporate IT and they don't allow docker or the installation of other python deps?

Pipenv.

We use pipenv, but it's been buggy and quarelsome so far. I like what it promises to do, but so far my experiences with Go modules have been better (to my surprise, given how much newer Go modules are than pipenv).

> at build time

Unless you're using CGO, in which case you still have to make sure your target environment has the shared libraries your bindings will make calls to.

Isn't this true for Python as well?

Unless you statically link your dependencies, but that comes with its own can of worms.

good? go package management is awful. it makes tools like searching through a git repo a chore because all of the source code for your module is in the repo too. the OP was pointing out how far behind go is behind something like python.

if you really wanted to, you could so something exactly like go is doing in python and create a virtual environment with all of your packages and version of python and ship it. there are also ways to build binaries from python.

if your machine is locked down by corporate IT, you use a virutal environment

> Resolve your deps once, at build time, and you're done.

I have ... rarely had this experience.

Every go package manager has found new and frankly fascinating ways to lead me into a corner where I think I have committed and pushed working code, but which causes highly-visible CI barfing.

I should add as an aside: I've done two tours of duty as a Cloud Foundry Buildpacks maintainer. Everyone's package system sucks, but some suck a lot less.

* Maven is basically sane if used with levelled starters. Otherwise it's a preview of fighting the Many-Tentacled Ones of the Deeply Nested Dependency Graph using nothing but a dull butter knife held upside-down.

* Gradle is the worst allergic reaction to XML in history.

* NPM used to have amazing bugs and missing features but has laboriously, slowly improved.

* Python is about whether you hit the sweet spot of the particular packaging system you use.

* PHP is a mess, unless you use Composer, in which case you're still stuck with the weird reality that PHP's package system is really a mix of in-process modules which need to be compiled and slabs of plain old PHP.

* .NET Core had what seems like a deliberate policy to come up with the most confusing naming and versioning scheme humanly possible and was wildly successful at doing so. Oh, and no canonical reference for versions. None. Apparently this improved but it was hell on earth.

* Golang. A new package manager every ten minutes. A different vendoring model, different assembly model, different bugs, every damn week.

Only Bundler is basically sane. We hit lots of corner cases but for the 20/80 cases it essentially worked.

I think the point above was that once the binary is built you don’t have to worry about maintaining the exact same libraries on the server where they might be competing/mixing with various incompatible versions of shared libraries for Python or Ruby or whatever.

That said, yes this is a shameful hole in the golang world (caused by the project leads’ inside-Google myopia) but I’m glad they are finally addressing it at this level of integration. And despite a few oddities it looks pretty solid.

> once the binary is built

My view is that if it can't be built from scratch in a cleanroom environment -- ie, in CI or a buildpacks stager -- it can't be built. This goes for container images too.

The amount of "works on my machine" and "where's Gerald? this module only builds on his laptop" I saw as a consultant was depressing.

Speaking of containers, though: they solved the contending-interpreters problem quite handily. Cloud Foundry has used containerisation since before Docker existed and it really shines for buildpack usage.

We started with Heroku's buildpacks, which had to deal with the nightmare of a pre-container world and showed all the pain of it. There was a great deal of stuff we just didn't have to care about and we later rewrote the bulk of the buildpacks to jettison all the stuff we didn't need.

It's all come full circle: Heroku now use containerisation and both Heroku and Cloud Foundry will soon be adopting jointly-developed buildpacks-lifecycle code.

> "where's Gerald? this module only builds on his laptop"

Of course, "Gerald" is an anagram for "Gradle".

I'm going to pretend, just this once, that I was clever enough to do it on purpose.

I would agree except I find Maven to be easily the most sane and flexible and extensible build tool out there. I've used Maven to consistently build and deploy vast systems made up of hundreds of libraries. And this was ten years ago. Maven's only "flaw" is that it doesn't support incremental builds and, frankly, I've come to believe this isn't a flaw at all and incremental builds are an inherently unpredictable anti-pattern. (That might just be the Maven trauma speaking. There are tools out there like buck that claim to get incremental builds down perfectly but I have doubts and we've seen some strange bugs in buck builds.)

What I find so strange is how the systems that came long after Maven -- particularly NPM and .NET -- that they could be so much worse than Maven. Somehow these people weren't aware of the importance of reproducible builds... despite that being a "thing" in Maven almost 20 years ago. Then, as noted above, you have go devs building off of git master branches.

All of this suggests a real problem in software engineering. There's a distinct lack of knowledge sharing that does not lead to gradual, linear progress. It's quite disconcerting. Things are not getting better, every day, after all.

These days I appreciate Maven much more than I did at first (ugh, XML, lolenterprisejava amirite??).

But the whack-a-dep thing is a real, genuine painpoint. Spring Boot Starters make this fairly close to invisible, but there are still billions of lines of Java code where the POM is like the map of an incredibly fragile ceasefire between 813 variants of some 118 different dependencies at 18 levels of nesting.

Like Game of Thrones, except bloodier, more complicated and much more likely to cause everyone to give up.

I don't think your impression of the current state of dependency management in Go is accurate. Pretty much everybody uses this:

https://golang.github.io/dep/docs/Gopkg.lock.html

Some do, but many don't.

Every major stable library uses a form of package management. Anyone pulling from master is a new go developer

The fact that new go developers do that tells a lot about the state of documentation, tutorials, tooling etc. A language and its ecosystem should encourage best practices from the start. Not as some kind of late afterthought.

Yeah thats the problem they have been looking to solve for a couple years now, but for what its worth the https://github.com/golang/go/wiki has information on dep tooling. My experience is that most people don't read it. Any tutorial worth its salt will mention this as well. Go has made it possible to pull right from master on deps fairly easily so many new developers do that so they don't have to learn dependency management. Proper dependency management is a skill that must be learned in any language, and often separates an entry developer from a mid level one.

There are many languages that make it explicitly difficult to pull directly from master. Nodejs packages are near impossible to install directly from a repository, you always have to go through npmjs.com. Haskell too (before stack became widespread), but even now most people install packages from hackage/stackage, where packages are strongly versioned.

This is where opinions will differ, I quite like go's ability to pull straight from a repository, and see outcomes like this as an unfortunate side effect of that feature. If a developer doesn't understand proper git and package versioning they will have a hard time in any language producing quality software. I'm not sure I want to give up the speed and ease of `go get` just to protect entry developers.

> This is where opinions will differ, I quite like go's ability to pull straight from a repository

Having the ability to do so does not mean it should be the easiest and simplest way to get from A to B.

Both pip and cargo can pull from repositories, but it's not the default and it's basically not covered in any tutorial. It's easy to do, but a developer is unlikely to find the information before they're actively looking for it.

> If a developer doesn't understand proper git and package versioning they will have a hard time in any language producing quality software.

I don't understand why you're using Go if your teaching methodology is to throw live chainsaws at people and berate them for not understanding proper power tool safety, why are you not just cutting out the middleman and using C++?

:sigh:

clearly you just don't like Go, which is fine. Of all the languages I've picked up Go was far and away the simplest and most productive, which is why I continue to use it. When I wrote python we were constantly using pip to pull from repos, so moving to Go I liked their repository centric view. Just because python recommends pip, doesn't mean junior devs don't totally botch dependency management with it. Dep management and git are core skills developers need to learn well to produce good software. There is simply no substitute for a bit of elbow grease on the matter. I'm not berating anyone, software is hard and being good at it means taking the time to learn some core tools.

This is a poor characterization of the issue. First of all, "do the packages come from git?" is not a real concern (similarly, "but it came from Pypi!" is not itself a feature). The real concerns are all "what reproducibility/security guarantees does my package manager provide?". For `go get` and vanilla `pip install`, the answer is "mostly none, but this is more or less okay for newbies/prototyping--it's okay though because the packaging docs point them to the better-but-tedious things". Cargo and Go modules do a pretty good job about providing reproducibility and usability at the same time, Python's answer seems to be `pipenv` which makes some pretty great promises, but has been quite buggy/quarelsome for us so far (and it's not even "official" as far as I can tell).

To characterize Go's ecosystem (but not Python's) as "throwing live chainsaws at people and berating them..." is pretty blatant dishonesty, doubly so once Go modules makes its official debut.

I disagree with your characterization. `go get` is the right tool for new developers (hell, I've been using Go for 7 years and I still use `go get` for the initial phase of my projects); they don't need reproducible builds, they need to install third party libraries and compile code. The current official reproducible build tooling in Go is vendoring, and that's quite unnecessarily painful for the things new developers need to do. Fortunately, Go 1.11 will be out soon with support for modules, so new developers can learn how to install code with the added bonus of not needing to learn a new set of tools down the road when they need to care about reproducibility.

I would say the state of Go dependency management is somewhere between what GP implies: "everyone is using go get to run things directly from master and that's terrible" and what you imply: "everyone has standardized around dep and that's fine".

I picked 5 "important" Go projects without checking first, and here's the solutions they use:

Kubernetes: The deprecated tool Godep + a pile of Make and Bash (which is the Go-est thing I've ever heard in my life)

Docker (Moby): vndr

Hugo: dep

Etcd: A bit of a mish-mash of dep and vndr, but mostly dep.

Cobra: Doesn't vendor dependencies, isn't a reproducible build (which is somewhat okay, as Cobra is primarily a library, not a tool in and of itself, but it does also have a CLI that probably breaks a lot).

In short, things are not currently fine. Dep is an okay tool, but fails for some use cases, and the community has not really rallied around it. Lots of important projects are sticking with what they've got. Glide, gvt, vndr, and Godep all remain important.

We're deluding ourselves if we discard an outsider's view that Go's dependency management situation is a dumpster fire, because it is. However, to GP, it isn't quite as bad as you suggest, most projects using Go have found some way or another of getting reproducible builds, and don't just run everything from tip of everyone else's master.

I am cautiously optimistic that modules will finally solve this mess, but we'll see to what extent they win in the marketplace of dependency management solutions.

this is an accurate assessment. i work in the dystopia of microservices and every service has it's own unique build.

Godep, dep, glide, make files, you name it. it's a total dumpster fire.

> Kubernetes: The deprecated tool Godep + a pile of Make and Bash (which is the Go-est thing I've ever heard in my life)

I haven't written Go in a couple years, but I lol'd. This exactly what we used to do, except replace Godep with glide.

Note that the Kubernetes people tried to migrate to Dep, but weren't successful. As I recall, it was due to several blocking issues (e.g. see https://github.com/kubernetes/client-go/issues/7).

Reply on news.ycombinator.com