GitHub's “hub” tool rewritten in Golang | Hacker News Reader

GitHub's “hub” tool rewritten in Golang

github.com

98 pointsakerl_58 comments

Reasoning behind porting it to Go: https://github.com/github/hub/issues/475

"My Ruby implementation of context.rb is getting unwieldy and in hub v1.11 I feel I've reached the limit of how much I can speed up hub exection. I can't make it much faster than 50 ms due to Ruby interpreter slowness. Go port can execute in under 10 ms, and it's not even specially optimized yet. Go port can also be distributed as a pre-compiled binary, skipping the need to install Ruby, which can be especially tricky and error-prone for Windows users."

Somebody rewrote hub in Go a while ago and called it gh. The team behind hub only decided recently to merge in gh and make it the new, GitHub-approved version of hub.

I don't really get what hub does, it's just a wrapper around some git commands + a CLI to github right? What is this 'context.rb' doing that somehow can be sped up by moving to Go?

I have the feeling that the sort of thing Hub is what Ruby's supposed to be good at, and Go not particularly. So the move sounds a bit strange to me.

Anyway, super convenient for them that a Go port already existed, obviously the move makes sense when the source is already there and it has the features and performance characteristics they desire, not to mention dropping the dependency on Ruby.

Quite the opposite, to me this sort of thing is go's killer app. Simple command line tools written in go are considerably easier to both build and use than ones written in ruby, python, or node.js in my experience. I'd much rather have the dependency hell happen at build time than run time. Keeping up a virtualenv/rvm/etc. setup to use some command line tool, or worse having their entire dep tree come bundled with them ala vagrant in a custom ruby install prefix, is a complete pain in the ass.

I've recently been writing command lines tool (for internal use mostly) and wrote them in bash because running other command line tools and checking their exit codes is simpler in bash than in Python for example. Problems start to come up when I have to deal with APIs that return or expect JSON.

From what you're writing it seems that it might be worth checking Go out for that. Would you happen to have pointers for using Go for that purpose?

It's certainly worth looking at the standard flag package (http://golang.org/pkg/flag/) if you just need flag parsing.

I've also heard good things about https://github.com/spf13/cobra and https://github.com/codegangsta/cli.

Interestingly, the tool that turned me on to using go as a commandline tool builder was a little thing called jq [1] that's a huge huge help in dealing with json from bash scripts. I have no specific pointers, though, so I'm glad some other people jumped in.

[1] http://stedolan.github.io/jq/

Read this as well: http://blog.golang.org/json-and-go

I have a blog post talking about the intention of building gh which ends up being hub 2.2.0: http://owenou.com/2013/12/22/fast-github-command-line-client...

>I have the feeling that the sort of thing Hub is what Ruby's supposed to be good at, and Go not particularly.

Actually go is very good at command line applications. The fast at start up time and easy concurrency make CLI applications in go very nice.

Yeah, it gives you the ability on the command line to interact with GitHub functionality, like browsing issues, opening a pull request, and so on.

I'm not familiar with their Ruby implementation either, but hey if it works the same way, is faster than it was before, and some developers enjoyed working on it then I guess it's a net positive :)

Not uncommon. Cloud 66, a platform for hosting Rails apps, rewrote their toolchain in Go after it was originally written in Ruby.

https://github.com/cloud66/c66toolbelt https://github.com/cloud66/cx

Mike Perham, of Sidekiq fame, built his latest product in Go. Ditto for Mitchell Hashimoto: Vagrant is one of the most popular Ruby products out there, but most of the other tools his company builds are written in Go.

Cloud Foundry is also being progressively rewritten from Ruby to Go. As with GitHub and Cloud 66, the command line client is a prominent example:

http://blog.cloudfoundry.org/2013/11/09/announcing-cloud-fou...

I'd be interested to see a transition matrix for rewrites (or comparable writing-the-next-thing efforts) between various languages. Here we have Ruby projects moving to Go. I've seen Java projects move to Scala. Twitter moved a bunch of stuff from Ruby to Scala, and i think they moved at least one thing from Scala to Java. Puppet Labs are moving from Ruby to Clojure. But i've never seen anyone move from Java to Go, and i have no idea if there's a popular transition from Python or Node.

It would be really, really interesting if there was a lot of flux to Go from Ruby, but not from Python and Node.

At Snowplow we are steadily rewriting 3 Ruby apps, one each into Go, JRuby (not really a rewrite) and Scala, choice driven by a few domain-specific factors.

Go, JRuby and Scala all meet our new criteria for ops-friendliness, which are: 1) the app must be user-installable by a single wget from BinTray 2) the app must have 0 or 1 [the JVM] external dependencies 3) the app must not require any special environment to run in. From a packaging perspective, we're using `godep go build`, `warble` and `sbt assembly` respectively.

I've never worked with anything as install&run-hostile as Ruby.

Random tangent: if you're getting the binaries with wget, how are you handling versions? Do you have a way to say "install version 0.0.72 on this machine"? Do you have a way to ask "what version is installed on this machine?".

I've worked with (mostly, built!) deployment systems that used rsync, scp, wget, Maven, and apt-get, and i've come to be strongly in favour of systems which bake versioning in right at the bottom, in such a way that you can issue commands and make queries in terms of versions right there on the box, without having to refer to some external ledger. Both for convenience, and for the consistency of code and metadata.

We're basically following the HashiCorp way, where each binary is in a zipfile specifying its version, the binaries don't have version-numbers in them but all binaries respond to a --version usage flag. [1] We then unzip the binaries into versioned folders so we can have multiple versions on the same box. This works pretty neatly with our Ansible playbooks too. [2]

[1] http://www.consul.io/downloads.html [2] https://github.com/snowplow/ansible-playbooks

> It would be really, really interesting if there was a lot of flux to Go from Ruby, but not from Python and Node.

What exactly would this tell you?

Keybase is being rewritten from node to go

Afaik it was previously in ruby right? I need to check out the source code ,that's definitely an interesting project.

any windows build out there?

Yes, it was previously written in Ruby. https://github.com/github/hub/releases/tag/v2.2.0-rc1 has a standalone binary release for Windows.

Is there anything like this for gitlab?

GitLab B.V. CEO here, thanks for asking, a couple of command line tools for GitLab are listed on https://about.gitlab.com/applications/

Isn't it more lines of code now?

This is mainly due to the current issues in Go with how package imports are handled. Since the Go port of hub relies on third party libraries, it uses Godep[1] to vendor its dependencies.

With /Godeps: 44838 LOC

Without /Godeps: 14733 LOC

[1]: https://github.com/tools/godep

I don't know the answer to your question, but even if we assume it is. Why do you ask? This is just one of many tradeoffs between the two languages. Do you think the number of lines of code is particularly important in this case for some reason?

Number of lines of code can be used as a (very) rough metric for complexity; expressing the same application in more lines of code may mean that the new implementation is a bit more complex, harder to work with, more to review, etc.

Of course, this is quite a rough metric; some languages are much denser per line (like APL/J/K and others in that family), so it's definitely not an absolute rule. But it can be interesting to compare a direct port of a reasonably complicated application from one language to another, to see how the abstractions in each language hold up; just like it can be interesting to compare the speed when porting from one language to another, even though there are plenty of caveats there too.

Lines of code is a pretty poor measure of anything except lines of code. More lines could be more clear or less clear. Unless we're talking an order of magnitude, which seems unlikely.

> Number of lines of code can be used as a (very) rough metric for complexity

When the code is a port and represents 1:1 functionally, then line of code is totally irrelevant, except for it's performance to execute (which clearly GoLang has demonstrated in this case to outperform ruby). A better "rough metric" for complexity is the number of functions or as Ruby Flog puts it - ABC metric: Assignments, Branches, Calls.[0]

[0] - http://ruby.sadi.st/Flog.html

Why was this necessary? Since hub is a distributed tool, users will now have to download Go on my local machine. Pretty big dependency, if you ask me.

It was written in Ruby before, requiring the Ruby runtime to be installed. With Go you have actual binaries. They also aren't linked to anything Go specific, so you can easily distribute them without installing anything Go specific on those machines.

So in the end the removal of the Ruby dependency should clean up your local machine ;)

To build it? Yes. To run it? No. Go apps don't require a runtime. The only thing necessary is distributing the platform-specific binary.

> Go apps don't require a runtime.

Go applications do require a runtime, but the executable is statically linked, so the binary executable ships with the runtime.

The rest of what you said is correct. The only reason you need the Go development tools installed is to develop the application or to compile it from source.

It may be worth noting that there's only a dependency on Go if you want to build from source. The tool is usable on its own after compilation to a standalone binary. See the Readme's "Standalone" section.

Or like any compiled language they can just grab a binary: https://github.com/github/hub/releases

And like any binary releases, there are platforms that are missed/forgotten/not cared enough about.

Yes, binary releases are a convenience for users of common platforms, and not a catch all for every version of every relevant platform.

This is why the source is available. Implementation in Ruby, or any other interpreted language, does not guarantee compatibility. Not all systems have current interpreters or dependencies.

If I'm on a semi-obscure platform like Solaris or GNU/kFreeBSD I'd rate my odds much more highly with Ruby than with something that builds binaries.

Why? If you are using a platform like that, would you not be adept at building packages from source?

I agree that Ruby is likely already installed on those, but having watched co-workers fight version dependency problems with Ruby Gems on OSX, I tend to believe nothing is a universal solution.

Adept at building C programs that use standard autoconf? Yes. Adept at building go programs? Not so much, and I probably don't even have the tools installed (unlike with ruby). We'll probably reach a point where go is as much a part of the standard unix install as ruby is, but we're not there yet.

The go toolchain bootstraps a small kernel of itself from normal gcc and I have yet to encounter any outrageous configuration issues. It should be at least as simple as building C programs that use standard autoconf, if not simpler.

I've done some work cross-compiling golang on platforms that are not yet officially supported. I think you'll find the support is really remarkably good.

> I probably don't even have the tools installed (unlike with ruby)

Provided correct versions are in place.

In my aforementioned observations, Gem installation was complicated when the parent app relied on gem versions below the code currently in GitHub.

This is of course a problem not unique to Ruby (see: Default Python version on CentOS), based on the number of versioning utilities for Go. I have also had plenty of problems with autoconf's that don't work.

> and I probably don't even have the tools installed (unlike with ruby)

So your arguments is basically "I'm a Ruby user" ?

I'm neither a Go nor a Ruby user, but I have Go installed on more machines than Ruby... and Ruby is a requisite for running unlike Go!

It takes 2 minutes to google for Go support on all of these platforms. Did you even bother? Go is supported on Solaris and kfreebsd platforms.

brew install go, apt-get install golang, etc

I have ruby installed on those servers right now. Not so much go. I'm not saying go would be impossible, just that it would be harder.

Ok? I bet you can figure this out and I bet it's just as easy as Ruby. http://golang.org/doc/install

Given that getting Ruby installed on those servers involves doing literally nothing, no, it isn't as easy.

You're basically arguing a technicality at this point. How important is it that it takes N time to do X and N+epsilon time to do Y? And how much should this affect one's decision for choosing which programming language to use?

Like C?

Yes. I bet it would take less effort to get a random ruby program running on those machines than a random C program.

You could've installed Go on all your machines with a provisioning script in the amount of time you've wasted being deliberately obtuse in this thread.

Most users can just download a binary from [1], they don't have to compile hub itself.

[1]: https://github.com/github/hub/releases

No, you can just download a binary. It's necessary because the Ruby version was unbearably slow.

"users" will grab the binary and be done.

Was Ruby a dependency of the prior version?

Yes.

Bigger than Ruby?

No, smaller, because you don't need the Go compiler to run it, just to build it. Therefore, you won't need the compiler at all, just the binary.

Reply on news.ycombinator.com