Application Development in Go | Hacker News Reader

Application Development in Go

jasonroelofs.com

87 pointsjameskilton25 comments

Although the author touches on some real issues with writing applications in Go, I do not recommend this article. The author does not follow the recommended practice for writing Go applications and the author has misconceptions about the Go tools.

a developer can not expect anyone to individually go get 20-plus different packages

Because the 'go get' command fetches dependencies recursively, there's no need to 'go get' packages individually.

the developer is forced to remember which packages the application uses, and which are other projects

The 'go list' command will tell you the dependencies for an application: go list -f '{{.Deps}}' path/to/application

For Slartibartfast I decided to give this project its own Workspace, with its own $GOPATH, and put the entire Workspace into git.

This is not the recommended practice.

each package is expected to be in its own source repository

A repository can contain more than one package. There's no requirement that a package exist in a repository.

go get probably works great for the needs inside of Google

Google does not use 'go get' internally.

anything placed in the src/[package] directory gets compiled into the resulting binary

Go files matching the build tags are compiled into the resulting binary. Everything else is ignored, including Go files that don't match the build tags.

That's enough nitpicking. The take away that I get from reading this article is that "How to Write Go Code" (http://golang.org/doc/code.html) needs improvement. The document was overhauled earlier this year to make it easier to understand. Apparently, it's not easy enough.

    > Although the author touches on some real issues with 
    > writing applications in Go, I do not recommend this 
    > article. The author does not follow the recommended 
    > practice for writing Go applications and the author has 
    > misconceptions about the Go tools.

Strongly agree. The author's workflow is full of antipatterns.

Perhaps the people who click with Go will just be different in their natures than people who click with other systems and frameworks. Programmers are not a uniform population, and tools which map well with subsets have real value.

The document was overhauled earlier this year to make it easier to understand. Apparently, it's not easy enough.

The substantive criticism I saw here was the one about versioning packages, which I think has led to some of the awkward choices. If you're collaborating and using packages from github etc (as go get encourages you to do), it isn't possible to document which version you're using so that go get will fetch one version till you decide to upgrade everyone. You have to freeze the dependencies somehow by hand (which is mostly what has led the author beyond the pale).

If this is seen as not a problem Go wants to solve, perhaps it should be documented up front, as it's something which comes up again and again when people start using Go coming from languages with an established package ecosystem. Almost every other language I can think of versions libraries explicitly, so Go is quite unusual in this regard. For example under 'Remote packages' in that page you link, there is no mention of versions at all.

Articles like this -- even if they have errors or arguable non-best-practices -- should be recommended because they drive conversation. Thus far one of the most unfortunate things I've noticed about the Go community (and this is seen even in discussions about Go on here) is a silence that results from a fear of doing things wrong. That improves no one.

The 'go list' command will tell you the dependencies for an application.

Does it? I thought it simply showed every package in the application, not differentiating what is yours and what is not. If I am correct, in no way does it solve the problem that they were trying to address.

This is not the recommended practice.

They specifically note that it isn't the recommended practice, but that it was a compromise to make a more easily reproducible structure. It is absolutely odd -- from a traditional developer perspective -- how the structure of a go application works when there are many dependencies, and I'm not sure what is the recommended practice for actually source controlling that (though we know how individual packages should be managed).

Go files matching the build tags are compiled into the resulting binary

Can you describe what you means? Such as

go build github.com/mypackage?

I've built several solutions with Go now and remain sure that much of what I'm doing is wrong, despite trying to find the relevant information.

> > The 'go list' command will tell you the dependencies for an application.

> Does it? I thought it simply showed every package in the application, not differentiating what is yours and what is not.

How is that not telling you the dependencies for an application? And differentiating yours from others ought to be trivial if you use the recommended directory structure with the path under /src corresponding to repository locations.

Many real applications consist of tens to hundreds of packages, tens to hundreds which might be in development in the application, and tens to hundreds which might be dependencies.

I fail to see how following the recommended directory structure (which is assumed) helps that, beyond that you can then correlate with a separate list of "things that are mine".

In a Visual Studio project, just to bring up a comparative example, I have my solution and projects. And then, elsewhere, I have nuget dependencies. There is absolutely no question which is which (nor is there any confusion on how to source control the whole, including that the dependencies include nothing more than pointers to their location/version). In my go project there are hundreds of seeming equals.

> I fail to see how following the recommended directory structure (which is assumed) helps that, beyond that you can then correlate with a separate list of "things that are mine".

If you are following the recommended directory structure, you don't need to "correlate" to "a list" of things that are yours, as all of your things will be under a path that corresponds to your remote repository, while external dependencies will be under paths that correspond to other remote repositories.

> In a Visual Studio project, just to bring up a comparative example, I have my solution and projects. And then, elsewhere, I have nuget dependencies.

In Go, you have the solutions and packages under (e.g.)

  /src/github.com/corresation/

And then, elswhere, you have dependencies that aren't yours.

The issues with package versioning and dependency management are well known and frequently discussed in the Go community. This article does not cover new ground in that discussion.

To list the dependencies for the application github.com/user/myapp, run the command:

    go list -f '{{.Deps}}' github.com/user/myapp

The command

   go build github.com/mypackage

builds github.com/mypackage and its dependencies. If github.com/mypackage is a command (a package with name 'main'), then the resulting binary is written to the current working directory. If github.com/mypackage is not a command, then you probably want

    go install github.com/mypackage

This command builds and installs the package.

> The issues with package versioning and dependency management are well known and frequently discussed in the Go community.

Is there a good place to see the current status of that discussion, or is it just spread across the mailing list?

Nathan Youngman's document is a good summary. https://docs.google.com/document/d/1_IJTRD6dDQvyCfyim4KJexq8...

There are many threads on the mail list. There was probably one this week :).

> Articles like this -- even if they have errors or arguable non-best-practices -- should be recommended because they drive conversation.

I disagree. You could use the same argument to admit any old troll. As someone who's often critical of Go, I found the quality of TFA to be too low to be worthwhile. It gets a lot of things wrong and in my opinion breezes past many of the real problems.

More importantly, with new languages people are very quick to sling whatever FUD they can regurgitate against languages they've decided they don't want eating up the mindshare they'd rather some other language had.

For Slartibartfast I decided to give this project its own Workspace, with its own $GOPATH, and put the entire Workspace into git. This is not the recommended practice. ... That's enough nitpicking. The take away that I get from reading this article is that "How to Write Go Code" (http://golang.org/doc/code.html) needs improvement. The document was overhauled earlier this year to make it easier to understand. Apparently, it's not easy enough.

When I started with Go, I was doing something similar: each project got it's own directory and I had to update the $GOPATH environment variable for each. It was probably a month or so before I had the "Ah-hah" moment that I didn't need to do that. Looking over the (nicely revised) "How to Write Go Code", this clarification is still missing from the The GOPATH environment variable section.

The Go Tool page (http://golang.org/cmd/go/) is required reading if you're going to get anywhere with Go.

It would also be really nice to have a resource that demonstrated some Go design patterns (at least a factory...).

First, a developer can not expect anyone to individually go get 20-plus different packages (Slartibartfast has more than 10 and growing), not to mention whatever 3rd party dependencies the application uses, to work on a single application.

go get fetches dependencies, so this is not an issue with go get, anything imported will be fetched.

There is no place to put such resources in the normal Go Workspace, as anything placed in the src/[package] directory gets compiled into the resulting binary and is not accessible or editable from outside.

It's quite possible to put resources into a go pkg folder without causing problems, and those are go gettable along with the code. I don't see the issue here.

Running go get for a package clones the repository at its current HEAD/master/trunk, with no way of specifying a version or commit you might want.

Go get doesn't (yet) get versions of packages, just fetches the master. This is something that's been discussed to death on the mailing list, there are plenty of threads:

https://groups.google.com/forum/#!forum/golang-nuts

Some solutions to that are:

Go get everything once, and then stop using go get (the solution in the article, though he brings workspaces into it too, which is unnecessary) - this is problematic as he points out.

Put all your dependencies into subpackages, frozen at a particular version, I like this solution for now.

Put all your dependencies into your own repo, again frozen at a particular version, and only go get or checkout from there

Use something like gonuts.io to add versioning information (relies on the pkg authors to do this though)

Add versioning to go get to check out versions to a specific path.

It would actually be pretty easy to add version dependencies to Go by adapting the go get source. go get already versions for version of the language by looking for a certain tag, but not versions of the package. You could just check out to a path like src/github.com/jasonroelofs/slartibartfast/1.0/ and use that path for imports. Someone should just go ahead and do it, to see how it works out - if it's popular, people will use it, and eventually it might make its way into the default distribution. The folks at Google just don't seem to see it as a problem so presumably their setup/usage doesn't hit this problem.

My fantasy vision, predating go, was publisher/project/version with /src and /bin below all that (separate bins and srcs). I am open to re-balancing that tree ;-), but somewhat surprised that version wasn't in from the beginning. Perhaps there were more worms in that can that I expected. (version should be constrained to x.y.z numeric form)

My fantasy vision, predating go, was publisher/project/version with /src and /bin

Yes in a way that feels more natural to me too - all your stuff is in one place, including src and binaries, not mixed up with other binaries/source and all under one version. The current scheme does follow unix conventions though and it's not too important.

The only persuasive criticism I've seen of versioning is that it doesn't deal with dependency hell automatically, however I'm not sure it can or should, and having simple versioning would be a nice step up from simply checking out master at the time you happen to go get, which means go get is not a reliable tool for fetching dependencies if you don't freeze them somehow (hence the contortions of this developer). It's not a huge deal though, and I'm sure it'll be addressed somehow at some point and the meantime there are various possible workarounds.

We use per-project work spaces too and it has simplified a lot of things for us. The biggest reason for this is that our projects aren't just Go based, their are other languages in there, many different file types. Building the project doesn't mean running "go build", that's just one component of a much larger code base. A complete build for a web application means:

* compile the Go app

* compiling the Sass files down to CSS

* combining and minifying Javascript files

* minifying HTML files

* optimizing/compressing image assets

* etc.

These things need to happen together, and than are organized into the deployable file structure. Ofcourse this is all automated with scripts, but a single global Go workspace doesn't cut it. It assumes that the Go app is king, while in reality its just another component.

An easier approach to the submodule problem is to use "git subtree" to copy in any dependencies of the project into your tree. Then (1) your project always builds against exactly the version of the dependency you tested against, and (2) if someone else deletes their repo it doesn't affect your project (unlike git submodules).

I was unaware of `git subtree`, I'll take a look at that option, thanks.

What if my dependency depends on something else and that something else breaks things when it's updated? Is there a decent way to lock all dependencies recursively?

I have similar problems building Go apps for internal use. My solution so far is to use per-project $GOPATH and `git subtree` to manage the whole $GOPATH as a single repository.

The primary benefit of doing so is build repeatability: any dev can `git clone` the repo and build the app successfully after setting the $GOPATH to the repo directory. Changes in external dependencies have no affect on us until we manually incorporate them into our repo via `git subtree`. There is no need to `go get` and we avoid its associated problems (e.g. versioning, availability of remote repos, etc).

It seems the go team chose a method that would produce zero name collisions under parallel development. That's the highest goal, but of course the burden of it is that paths below /src must be kinda long to support it. A "rare collision" solution could have been more concise without being as robust.

A common theme in Go is "Do important things the correct way," even if that equates to slightly more work for the developer.

People will inevitably argue about (a) what's really important, and (b) what's really the correct way. But they didn't create Go. It's clear where Go's creator stand on the issues.

I had a funny experience. Before I ever worked with Go a developer I'd worked with (and argued small details and big philosophies), told me "why don't you use Go? It seems your kind of language." From my later glances it seems like he was right. So I might be in the fortunate subset for which the philosophy matches, more or less.

> It seems the go team chose a method that would produce zero name collisions under parallel development.

Well, that and also one which produces a mapping between local paths and repository locations so that you don't need much (or any, in most cases) package metadata or a central directory service, creating a decentralized/distributed package management system.

> That's the highest goal, but of course the burden of it is that paths below /src must be kinda long to support it.

I suppose, but I'm not sure long paths is really all that big of a burden.

Reply on news.ycombinator.com