Hacker News Reader

Comment by sqs | Hacker News Reader

Here's an interesting portion of a Q&A panel from the same dotGo 2014 conference (http://dotgo.sourcegraph.com/post/99652344343/go-team-q-a-de...)

Q: There are several dependency management tools in the wild: godep, gpm, etc. Are there any plans to provide this functionality in the core?

Brad Fitzpatrick: We don’t want to dictate a policy, so we hope the community fights it out and a victor emerges. Then maybe we’ll bless that one. Then if everyone likes it and it has been stable for a couple of years, maybe we’ll add it to the core.

Brad Fitzpatrick: Part of the reason why we don’t care as much about dependency management inside Google is that we don’t use the go tool inside Google.

Andrew Gerrand: The lack of versioning built into the Go tool incentivizes library authors to provide good, stable APIs.

View on HN

Andrew Gerrand: The lack of versioning built into the Go tool incentivizes library authors to provide good, stable APIs.

Are you kidding me??? So if some invariant behavior on an otherwise stable API changes, I don't know until runtime that a bunch of the code that I shipped, haven't changed, and then recompiled a month later has changed it's behavior?

Yeah, this seems crazy to me. Also, honestly, while stable APIs can be nice, I would rather have people have the option to change the API in later versions, if it makes things better. But I want to be able to consciously decide to upgrade when I'm ready.

You do have that option. As I explained at the time (but this was not included in the live blog), the convention is to change the import path when the package API changes.

If you're shipping serious code you should be vendoring your dependencies.

Why?

Because you want to have control over all software you're shipping to production.

If "vendoring" precludes one from using a dependency installed on the system, then this practice is static linking all over again. Static linking has its place, but dynamic linking with sane versioning solves many problems.

I'm pretty sure all Go binaries are statically linked.

But they said there are plans for dynamic linking

Sure, but that's orthogonal to what's being discussed here. You don't check your compiled binary into source control, but proponents of vendoring do check their (transitive) dependencies in.

People like Go because you have a single executable to drop on a box, no worrying about what version of what libraries are installed. Though, the Go team realizes there is a need for dynamic linked libraries, so they at least have a proposal for it[0].

[0] https://docs.google.com/document/d/1nr-TQHw_er6GOQRsF6T43GGh...

I've been following that proposal, and I'm a bit surprised that it isn't mentioned on the roadmap. Looks like dynamic linked libraries will come sometime in 2.x land?

One of the other talks made reference to it[0].

[0] http://dotgo.sourcegraph.com/post/99652344343/go-team-q-a-de...

In the era of containerized application platforms and immutable infrastructure, static linking is the path forward.

What problems does dynamic linking with sane versioning solve?

Sure, but manual vendoring is just one (tedious, error-prone) means to that end. Any sane package manager with something akin to lockfiles gives you the same guarantee without all of the headaches of vendoring (and unvendoring when no longer needed!) all of your transitive dependencies.

It's not tedious or error prone. Just use the tool.s

So, uh, why is the go-supported option encouraging me to download it from github at compile time without so much as crude versioning support?

Lots of people run internal Maven repos, they have every dependency vendored, but they also want the ability to choose between versions of that dependency.

This is quite interesting, because after the Python fiasco in 2012/13 I thought a new language like Go would have learned from that and done that right from the beginning and I always thought Go did it. "We don't care" is even worse than "we just don't know how, yet".

I didn't get the sense they were celebrating their lack of care, or that they chose not to care. (They didn't create Google's build system.) They were simply stating the reality.

Different people have different requirements. It makes no sense to declare some tool to be THE STANDARD(TM) if it's of no use for a large proportion of their users.

The way $GOPATH works and dependencies are managed is well-documented. Everybody is free to develop their own tooling. Right now, many people flock towards godep, but that might change in the future. Also, since this is only relevant during development and for compilation, getting it right is not as important as if you had to deploy every single one of your dependencies. With Go, you get one binary, and that's what you deploy.

And there are people like me who think that the programming language tooling shouldn't have to be tracking dependencies. If, for no other reason, than it is common to use multiple languages for a single project, and then a language-agnostic method should be used.

I've leaned towards just having everything it our DVCS (git in our case). External libraries are handled using git subtree.

It is also using a defined way to handle dependencies. I have no problem with that either.

So there is a protocol that is standard, but not the tool which implements it? That is not what I understood after I've read the article. But that would definitely mean it's actually not bad to have different tools.

There must be some standard, though. If you have neither a defined protocol, nor a defined tool, then you have a broken community. That you need to support different package managers for different Linux distros is one of the reasons people don't support Linux for their tools, and I personally would even argue that it is one of the reasons languages like Ruby, Python and Go need to solve that themselves.

It's only an issue for where source code is put, not binaries or shared libs. All Go programs are statically linked, so end-users of a program don't care at all about dependancies. The only people that will care about it are those that want to build the binary themselves.

Also, if you are unaware, Go has great cross-platform building. I can generate the executables for all platforms/architectures Go supports from my single dev machine.

What Python fiasco?

I'd guess he's talking about virtualenv + pip moving into the Python core with 3.3.

...I don't remember reaction to that really qualifying as a fiasco, but I could just not have been paying attention at the time.

Any clue as to what they do use internally?

Yeah, check the comments at http://www.reddit.com/r/golang/comments/2j65lb/go_qa_at_dotg... (specifically the blogspot link and Andrew Gerrand's (enneff's) comment).

Google has an internal build system that builds everything, and it's language independent. Sort of like a distributed make.

Some publicly available information: http://google-engtools.blogspot.ie/2011/08/build-in-cloud-ho... (https://news.ycombinator.com/item?id=2899467)

They vendor everything.

"Vendor everything" was my conclusion to solving dependencies as generally as possible, while trying to knit PostgreSQL modules (locally produced schema chunks) and extensions with a base of versioned Rails apps.

Yes, we do vendor everything, in that we have a snapshot of all of our dependencies in our source control system. But we do it across the entire codebase, not per project. That is, we typically only ever have a single version of a library for the entire company (with a few exceptions). If a project needs to update to a later version, they basically update everyone using that library. For widely used packages, this can sometimes be a time consuming process, but we've found it to be preferable to the alternative of having version conflicts all over the place. This is generally true not just for Go, but all languages. So the idea of a project needing to pin to a very specific version of a dependency and never update doesn't really fly.

That sounds like a wildly unproductive way to work. Discourages ever changing anything.

It sounds similar to the Linux kernel. If all the providers and consumers of an API are in the same repository, then you can change an API at any time as long as you update all consumers of the API at the same time.

It does discourage upgrading until you really need it. On the other hand, if one person decides to do the work then everyone benefits.

That sounds quite implausible. Sure, one person can go and upgrade the library.

But I find it hard to believe that one person can go and fix up all the projects he's hardly ever heard of that depends on that library.

We have good tools for this. A Googler can create a patch that upgrades a library and run tests for all projects to see what breaks. The project owners review the changes. Of course, some upgrades are easier than others.

In some ways this is similar to what Linux distros do, but sharing common source control, build system, and test runner makes it easier.

It also results in having a huge version control repo. From what I've heard, Google can't move to git because of this... they're stuck with Perforce because it's the only VCS that can support a repo of their size.

I'm going to guess you're a Rubyist. I had to look up what "vendor everything" referred to. There's a reasonably well referenced blog post by a Rubyist -- but only referenced by other Rubyists. I see that the Bundler tool looks for a 'vendor' subdirectory, so I'm going to guess that's how it became a part of the Ruby lexicon.

Anyway, just wanted to point out that it's possible you're using Ruby lingo and not general-purpose tech terms. I could be wrong, maybe I just missed the boat.

I think "bundle everything" might make sense to everybody, including Rubyists.

I remember reading about "vendor" branches in the cvs documentation (probably in the early 2000s), which if I recall correctly were trying to solve a similar problem to sub-modules in git. So I don't think the Rubyists invented the term.

The Subversion book covers the concept of vendor branches extensively:

http://svnbook.red-bean.com/en/1.7/svn.advanced.vendorbr.htm...

That's where I first read the term, but it predates svn. As far as I know, it's been the common term for committing external dependencies to your own VCS for as long as there's been such a thing as a VCS.

We are Nodeists (not nudists - to the best of my knowledge) - we like to use the term "shrinkwrap."

Shrinkwrap isn't quite the same. npm install --save seems to be the closest equivalent, based on discussions I've had with people.

Vendoring means having the physical source code on hand. Not version numbers, not SHAs -- actual checked-out code.

> Andrew Gerrand: The lack of versioning built into the Go tool incentivizes library authors to provide good, stable APIs.

It certainly encourages stable APIs, but I don't see how making it very painful to iterate on your API is a strategy for good ones. My experience is that doing anything well requires the ability to gather and respond to feedback. If you can't iterate, then you're doomed to be stuck at 0.0.1 levels of quality.

So, do you consider the Go1 compatibility guarantee to be a mistake? Most suggested stdlib or lang improvements are DOA for the time being as a result, which is annoying, but of course so too would be frequent breakages. I'm curious about your take, since you have some relevant experience.

> So, do you consider the Go1 compatibility guarantee to be a mistake?

No, I think a language basically has to declare that level of compatibility for a major version. The important part is to get as much real-world feedback and iteration in as possible before you bang the 1.0 gong.

Reply on news.ycombinator.com