Ask HN: Good Go codebases to read? | Hacker News Reader

Ask HN: Good Go codebases to read?

167 pointsra752 comments

I'm a Python programmer and I just started learning Go. What are some good Go codebases to read? I feel it helps me learn better if I look at some existing high quality open source code.

There was a similar post about Python a couple of years ago (https://news.ycombinator.com/item?id=9896369) and it helped me tremendously. I'm looking for something similar with Go.

Thanks!

View on HN

Humble suggestion / shameless plug: try using Sourcegraph to read through some of the excellent codebases mentioned in other posts. This is basically our prime use case (I'm one of the creators): reading through and understanding code structure with jump-to-def / find-refs / symbol search in your browser. I'd suggest starting with the main function and tracing through the static structure of the codebase from there:

https://sourcegraph.com/github.com/golang/go/-/blob/src/net/... https://sourcegraph.com/github.com/mholt/caddy@master/-/blob... https://sourcegraph.com/github.com/btcsuite/btcd@master/-/bl... https://sourcegraph.com/github.com/camlistore/camlistore@mas... https://sourcegraph.com/github.com/boltdb/bolt@master/-/blob... https://sourcegraph.com/github.com/go-kit/kit/-/blob/example...

I was trying to understand the go source code the other day, my process was:

`git clone https://github.com/golang/go` then `cd go`, and `tree .`, `git grep ...`, `vim ...`, etc.

Anyway, couldn't find how to get the tests to pass, but found this:

https://golang.org/doc/codewalk/markov/

And a little explanation: https://golang.org/doc/codewalk/codewalk/

Also: https://golang.org/doc/install/source

In terms of codebase, kubernetes and upspin seem pretty good too:

https://github.com/kubernetes/kubernetes https://github.com/upspin/upspin

I imagine upspin is one of the best examples because Rob Pike is one of the main contributors

Funny thing is that I couldn't get up spin to run :).

But yes having a global namespace is a genius idea.

Thanks for sharing! Just wanted to let you know it doesn't work on firefox android.

Do you have any plan on supporting C?

HashiCorp put out good stuff. CoreOS can be hit or miss; Torus was nice, etcd is a wreck. I like https://github.com/go-kit/kit and https://github.com/oklog/oklog. Definitely avoid Kubernetes and Docker -- they're Go by committee, look and feel like transliterated Java.

Seconding the Hashicorp projects. I learned a lot about HTTP from reading the Vault code.

go-kit and oklog are also fantastic. Peter Bourgon has written a good article about Go best practices and his open source projects really aim for clean, readable code.

Also join the go slack if you haven't yet.

I know very little Go, but I have seen the Kubernetes code and my impression was "I bet these people were Java programmers in a past-life". You could tell by the multiple layers of abstraction, a lot of which felt unnecessary.

Having said that, Kubernetes is a solid piece of engineering, so they are clearly doing something right and I don't want to denigrate their hard work.

Kubernetes also came from the same engineers that worked on Borg (Google's internal Kubernetes), which is written in C++. I imagine a lot of the constructs and design/layers were inspired by their experience in Borg.

Some of the style adopted for Kubernetes mirrored both previous Java experience (inside Google and out), but most of it was simply observation of other large Go codebases.

I don't think much of the layering / structure has anything to do with Borg except in a few areas:

1. Scheduler predicates

2. The massive ball of yarn that the kubelet started off as and was unable to achieve escape gravity on (until CRI landed)

Everything else was focused on pure efficiency and is only slightly biased by general experience building large software projects (pass dependencies down, avoid complicated abstractions, rigorous style enforcement). A lot of that leaked as Kube grew into a larger and larger community and more people contributed.

The other abstractions people complain about mirror traditionally hard problems (API version stability with conversion across semantic changes, general API surface area management). Other than that we decided not to build on relational stores but a partitioned document model (etcd, inspired by chubby, but not fundamentally different from any other object store), I'd say the code is mostly just a reflection of being good enough to maintain now and refactor later. Pretty code is easy when it's a few people writing it. Harder when it's a team of tens / hundreds

> Pretty code is easy when it's a few people writing it. Harder when it's a team of tens / hundreds

Yeah, this is a good point that I was overlooking. It's easy for me to be critical when I feel the pain digging through multiple levels of abstraction to find out how something works, but I didn't the feel the greater pain of trying to manage tens or hundreds of developers working on the same codebase.

i second the k8s and docker comments. took me a long time to realize this isn't how i should be doing it.

I'm flattered. If you like the Torus style, https://github.com/coreos/go-tcmu is a package I'm quite proud of (SCSI interactions wrapped in a nice pure-Go interface) and then a plug for https://github.com/cayleygraph/cayley which is a little rougher but a nice community and large codebase I maintain.

Bolt (also here mentioned) is one of my favorite beautiful codebases. https://github.com/mdlayher is a prolific author with lots of good protocol implementations. https://github.com/chihaya/chihaya is a fun project if you're into BitTorrent.

Standard lib source is good. I gained a lot of insight from Dave Cheney's blog: https://dave.cheney.net.

This would have been my suggestion. Read the standard library. It's mostly easy to understand and well documented. In general if you have a question of how a problem should be solved, this is a good place to refer.

+1 on the standard lib. It's done so nicely in most places, that it's easy to convince myself to take a minute and read implementations of routines I'm reading godocs for, even though the docs are 100% clear... cuz it's actually a pleasant read.

In contrast, I've worked in places where I needed a drink or 3 before I can build up the gumption to open my text editor :-)

check out the videos that Francesc Campoy is putting out on his youtube channel "JustForFunc", he does code reviews and the one of him implementing the context package is great for learning the go way of doing things.

https://www.youtube.com/channel/UC_BzFbxG2za3bp5NRRRXJSw

https://github.com/campoy/justforfunc

IPFS. I'm not a go developer, but I was easily able to dive into the code in many of their repos, understand the code and make fixes which were merged. https://github.com/multiformats/ https://github.com/ipfs

I've only read some of it, and not in depth, but nevertheless enough to highly recommend all of fogleman's Go projects:

https://github.com/fogleman?utf8=&tab=repositories&q=&type=&...

I really enjoyed studying the BoltDB source[1], which is Ben Johnson's popular key/value store library. I found it to be clear, idiomatically written, and well annotated.

[1]: https://github.com/boltdb/bolt

Several suggestions:

As others have suggested, looking through go's std library is a good idea as you will need to be familiar with it in any case. Use https://golang.org/pkg/ liberally to go from reading API documentation to code.

Look at https://golanglibs.com categories and pick the one that interests you or is something you already know a bunch about. Then pick something simple from that category.

There is a library that converts some python libs to go packages. This may also help as you can see how the mapping worked.

Finally, just pick up the Go book and start coding something you care about! Go's online documentation is quite good. I was able to bootstrap 3 high-school interns this way fairly well and they actually produced a fair bit of working code in 7 weeks. IMHO you should spend a significant part of your "study" time writing code. Read other code as and when you need it.

Upspin source is pretty good, since Golang creators are also working on it https://github.com/upspin/upspin

I second this one. They do a lot of things that I would consider best practices that are not incredibly obvious.

I'm surprised to not see Mattermost[0] mentioned yet.

[0] https://github.com/mattermost/platform

The standard library source code is pretty good.

https://github.com/golang/go

A bit of a plug but this is my "flagship" open source project and I have worked on the codebase since late 2015 (a lot of time) so hopefully it should be a good example to take a look at.

Also would like to hear feedback if you spot something which doesn't seem pretty/idiomatic in my code.

https://github.com/RichardKnop/machinery

This isn't a codebase, but it's a piece of knowledge that I've found to be quite useful:

https://commandcenter.blogspot.com/2014/01/self-referential-...

Standard lib 100%. And Dave Cheney's personal website/blog posts. You don't need anything else.

I know Go is mainly used for systems and infrastructure level software, but I also know that some companies are using it at a higher level of abstraction for their application servers as well.

I'd really like to see more examples of these types of codebases but can't seem to find examples anywhere.

http://exercism.io is pretty good, you solve programming exercise and then can look and see how other users solved the same problem.

I'll do a little self promotion here: https://pisc.junglecoder.com It's a stack-based concatenative scripting language implemented in Go. There are also Go implementations of LISP (zygomys) and Lua that may well be worth looking into, if you'd like to see how one builds a language in Go.

Otherwise, the standard library is always a good way to find out about things that you didn't know existed, or forgot after reading the standard.

Sqlx and go-kit are two libraries that helped while learning Go. Sqlx helped me understand the value of Go's implicit interfaces/package level structure, while go-kit helped me better understand good project structure.

https://github.com/jmoiron/sqlx https://github.com/go-kit/kit

https://github.com/btcsuite/btcd Learned a lot from reading this code

Agreed! It is crazy that this code even exists. They have since rolled it into their decred/altcoin project.

https://www.investitin.com/decred-altcoin-review/ https://github.com/decred/dcrd nice

Hugo is well maintained - https://github.com/spf13/hugo

Depends entirely on what you interested in building, but these are worked on fantastic Go engineers and architects:

https://github.com/hashicorp

https://github.com/coreos/

https://github.com/kubernetes

Caddy web server. Piece of art.

Link: https://github.com/mholt/caddy

I find anything made by tidwall super easy to read and informative. You could compare it to redis's code quality.

https://github.com/tidwall/buntdb https://github.com/tidwall/tile38

The Gogs codebase is pretty good, although it has many uses of package globals, which doesn't sit well with me. I mostly like what's in there: https://github.com/gogits/gogs

I wrote this blogging server in go a few weeks ago. It is very small, and should be helpful getting some basic concepts across for you

https://github.com/blhack/blogeyBlog

Camlistore is great: https://github.com/camlistore/camlistore

Many of what is now the Go standard lib came from Brad Fitzpatrick's work on Camlistore.

I'm not sure I agree. A lot of camlistore's idioms don't feel like "normal", idiomatic Go. Since it was still being written while the standard library and language was evolving, I get the feeling when reading the code that it's more like doing archeology of the Go language.

IIRC it also still has a non-standard build process (go run make.go) for the same reason (it was being written while the library/language were in their infancy).

A friend of mine helped create Pilosa [0], a bitmap index. I've been meaning to read through that at some point.

[0] https://github.com/pilosa/pilosa

Honorable mention of https://www.goinggo.net/

William is one of the Authors of "Go in Action" and a great teacher.

Macaron has helped me the most https://github.com/go-macaron/macaron

https://github.com/coredns/coredns/

The standard library is quite good and idiomatic. I particularly like net/* and io)

How about etcd? https://github.com/coreos/etcd

Reply on news.ycombinator.com