Go best practices, six years in (2016)
peter.bourgon.org
peter.bourgon.org
1. Don't put your internal libraries in /pkg. /pkg has special GOPATH meaning as "compilation cache". Its not an actual name conflict, but why bother risking it.
2. This is just wrong:
> fmt.Printf is self-contained and doesn’t affect or depend on global state; in functional terms, it has something like referential transparency. So it is not a dependency. Obviously, f.Bar is a dependency. And, interestingly, log.Printf acts on a package-global logger object, it’s just obscured behind the free function Printf. So it, too, is a dependency.
stdout (and the buffer, and mutex on it) are exactly the same as the global log object. In fact, `log.Printf` is more or less just an alias to `fmt.Printf`[0]
3. I wish it had mentioned the functional options pattern in the part about constructors[1]
0: https://github.com/golang/go/blob/b77aad089176ecab971d3a72f0...
1: https://dave.cheney.net/2014/10/17/functional-options-for-fr...
It's a little too late for this because it's pretty much an unofficial standard for library code :-) Just look at Kubernetes, Docker or many other major Go-based applications/tools.
Functional options are controversial in the Go community. I don't like them personally. The idea of having a bunch of functions that exist only to mutate internal state on init is... an odd choice. Google code is riddled with this sort of style.
It's annoying to write client code for another reason: It's hard to discover. For the "term" example in the article, you can't sit in your editor/IDE and type "term." and get a list of setter suggestions, since everything in a single namespace. "term.Speed()" does not sound like an option to me. Function names are generally verbs, and having one called Speed() doesn't read well. And it causes issues if you want to have a type called Speed. Then it has to be term.WithSpeed() or term.SetSpeed() or something. Neither of which is really self-explanatory.
The fact that options are functions has other downsides. The moment you pack a option into a function, they've lost their ability to be introspected. You can't dump the options to a debug log before you invoke the constructor function. You can never ask an option function what it contains.
I prefer representing options using actual data:
type (
Option interface {
isOption()
}
Speed int
RawMode bool
SomeOtherSetting struct {
A, B int
}
)
I come from a background in functional programming, and having used functional options in Go for a while now, I completely agree. I've never seen an implementation of them that felt natural or idiomatic to use. They're incredibly awkward, and I basically always prefer the alternative approaches to the functional options style.
NewServer(..., Config{})
It means that sometimes you'll have to use dummy values (-1 for "no value", for example) or pointers (nil means optional). It means that any boolean has to be phrased in such a way that false is the default (so if it feels natural to call the option UseKeychain and it should default to true, you have to invert it and call it DontUseKeychain).
Config structs also means it's easier for a caller to construct invalid configs:
type Config struct {
// Either stream or channel can be set, but not both
OutputStream io.Writer
OutputChan chan Event
}
// ...
pipelines.New(pipelines.Config{
OutputStream: myFile,
OutputChan: myChan,
})
// ...
pipelines.New(
pipelines.OutputStream(myFile),
pipelines.OutputChan(myChan))
(In this particular case, you can work around it by having a single Output field that uses an interface, but there are other situations where mutually incompatible fields can occur.)
For stuff that needs finer-grained control, or long-lived API stability, functional options give you a lot more flexibility.
I have a GitHub CI service that verifies the standard go tools pass on every commit.
The bot itself is naturally written in Go and executed in a Lambda function.
Everything is just so fast! You get feedback in seconds. Which keeps you in the flow of coding.
Maybe some other gophers will find it useful:
[0] http://devs.cloudimmunity.com/gotchas-and-common-mistakes-in...
It has a little more than Peter's post, but it's still not too opinionated :)
The code from go kit and [oklog](https://github.com/oklog/oklog) are great examples of idiomatic Go. Unfortunately the community at large doesn't really follow the "no init"/"no package global vars", which can sometimes lead to bad experiences importing opensource Go libs.
to be honest it strikes me as the sort of library that excites intermediate developers who tend to over-architect
I find this true about nearly all microservices in Go. Microservices are much more useful in something like Node that can only take advantage of 1 OS thread per instance. Without containerization and load balancing in Node you wouldn't be able to scale.
Go on the other hand can efficiently utilize a nearly unlimited amount of threads as necessary with its scheduler. You're much more likely to over-architect if you don't keep this capacity in mind.
I’ve used it and it works great (much better than glide, gb et al) but I don’t see it in the wild too often.
Go's dependency management story is starting to resemble JS's modules story :(
It's worse tbh. npm was actually always usable if you knew what you were doing. Complaints with npm were generally down to misunderstanding, people not being aware of features like 'npm shrinkwrap' (which has now been replaced with a lockfile-by-default strategy) and of course registry issues (allowing unpublishing) that weren't unique to npm, and were only an issue for people relying on public registries at deploy-time, a bad idea.
Dep will have a clean migration path to vgo, and the latter isn't really production-ready yet.
For now, I'd recommend using dep for daily use, until you have a compelling reason to switch to vgo, at which point the migration will probably be automatic.
This post sketches a proposal for doing exactly
that, along with a prototype demonstration ...
I intend this post to be the start of a productive
discussion about what works and what doesn't.
Based on that discussion, I will make adjustments
to both the proposal and the prototype, and then
I will submit an official Go proposal, for
integration into Go 1.11 as an opt-in feature.
Sam Boyer's follow-up is interesting reading [2]. I get the feeling that despite their ongoing discussions, the Dep team was/felt ambushed by this move.
I think it’s the right choice for now. It was going to be the one true solution and many people have stopped working on the alternatives.
We’ll see where vgo lands, but dep is very practical right now.
My only criticism is that "dep ensure" will actually parse the code to discover dependencies through import statements, which is also what Glide does. To me, this is antithetical to the purpose of a Gopkg.toml/lock file. In other words, Dep's full list of dependencies isn't actually in the Gopkg.toml file; it's a sum of Gopkg.toml and your code. That is confusing.
My desired behaviour:
* "dep ensure" should always used the lock file, nothing else, to install;
* "dep ensure -update" should update the lock file to what is specified in Gopkg.toml (and only that);
* "dep ensure -add" (which I think should be "dep add") should b required to add new dependencies to the Gopkg.toml file.
Aside: I wish Go projects weren't stuck with BSD style flags (-update instead of --update). GNU style is more common and arguably more practical. I applaud whenever a project (e.g. Prometheus, recently) finally sees sense and goes over to GNU flags.
Though supporting `-flag` does remove the very nice combining of short flags.
Another interesting recent project is Truss which allows you to autogen go-kit handlers from protobuf definitions
There are a few general ways to do this. The best way is to use the real underlying dependency. This should be the default.
If the dependency is too slow to use in the test (e.g. it has a long startup time, or is hosted remotely), the next best thing is to use a fake -- a local implementation of the dep that has the same behavior, but operates in memory on ephemeral data. The best libraries will provide these, and if they don't, you might be able to implement one and contribute it back. Even if you have to maintain it yourself, it often takes less code than describing mock behavior. The downside of fakes vs. the real implementation is that their behavior too can diverge. But this is not a net downside compared to mocks.
If you can't use a fake, the next best thing after that is a stub. See below.
Only if none of these options are viable should you use mocks.
(The position I'm describing is the one referred to as "classicist" in this post: https://martinfowler.com/articles/mocksArentStubs.html. I'm not by any means its originator.)
You don't test the http.Client as part of your business logic, even (especially) if your business logic makes HTTP requests. Instead, you trust that package http is already well-tested, mock out http.Client as an abstract HTTP Request Do-er, and you downscope your tests to your business logic exclusively.
I'm not so worried about http.Client behaving incorrectly, but I am worried about the server I'm targeting behaving in a way that's different than my expectations. I'm also worried about getting cookie, post-data, url, etc. formats wrong. Even if they match what I put in my mock, they won't necessarily match what's out there in production. I'm worried that I'll handle download resumption wrong, for example.
If I have the capability to bring up the target server locally, that's what I will do. If I can't, but it's straightforward to create a work-alike, I'll bring that up in process. Only in the most dire of circumstances will I use mocks to test my code.
Think about it this way. What harm is done by using the real object, under the circumstances I described? Sure, if the dependency code breaks, my tests will too. Is that bad? Is there any point in my tests passing when my code won't actually work in production? OK, so maybe the tests will run slower? This matters in the limit, but often times you can use the production dependency without materially increasing test time. OK but what about the CI system using more resources? That will only be a problem of significant economic import if you're running at a scale like Google or Facebook.
My opinion is there are a few main valid objections to my approach. One is an argument about the size of the test code. In some cases, bringing up the production dependency will require more code than a mock. Sometimes a lot more code. My response is that we should make it easier to bring up the production dependency! But if we can't, I can see the case for a mock or a fake. The other counterargument is for when we really are interested in testing the sequence of interactions the code under test has with the production system. In that event, I guess I'm ok with using a mock, but I think most of the time we're more interested in testing state. We can also use things like testvalue injection to test behavior sequences, which is normally a lighter-weight approach, and in some cases can allow you to reproduce paths that you'd otherwise have trouble hitting, even with a mock.
This is the responsibility of system or integration tests, not unit tests. (Unit tests are what's under discussion in the article.)
> What harm is done by using the real object, under the circumstances I described?
Time, energy, and effort, all of which inevitably lead to longer (slower) dev/test cycles.
My contention is that using real objects normally does not take materially more of any of these, and thus that most code is served best by being tested only by integration and regression tests. Where my contention is untrue, apply your best judgment, of course. :)
full-system testing, with an honest attempt to do fault injection, by definition exposes all the issues you might have in production. if you have a decent test suite you can honestly claim you can expect it to take client code correctly.
i don't know how thats true if you've only tested A against some stubbed out version of B that always returns success.
integrated testing is harder and requires more framework. its also kinda necessary, and if you only have the resources to do one or the other, the choice seems pretty clear.
i also heartily agree with your comment that since we're supposed to have frictionless and scalable deployment. there is no reason not to test this as part of integration testing and also exploit this function to make integration testing easier.
I work on search and indexing systems at Google, and have done for more than a decade. I'm by no means the best, nor an authority. But I have spent some time living in this problem space, and this is my conclusion from my years of experience.
FWIW a single run of the tests for the system I work on now takes . . . I don't know. I would guess hours or maybe even a day of computer time? But, computers are cheap, and defects are expensive. So you split the test suite into bits small enough that the whole thing runs in ten minutes, and run them all in parallel. And, yes, there have been cases where I have had to debug the slow startup of a dependency so that I can make it fast enough to run in my test. But, that's what they pay me for, I enjoy that type of work, and it has ancillary benefits to the dependency when it needs to start up in production.
OK, well I guess this would be another caveat for my advice. If bringing up a production simulacra would take more resources than you have, then you might have to lean more on various types of doubles. This may be more often in environments that have insane prioritization of dev time vs. machine resources.
If I may ask, what was the bottleneck in your test suite? Did it saturate the cpu, ram, disk?
I'm sorry, I'm making an argument about how people should write tests, not what terms they'll use to describe it. :) I promise you that there is no disingenuity going on in my use of certain terminology to describe how I think it's best to write tests.
(Yes, some do. On the whole I find most don't.)