Code coverage for Go integration tests

There was a technique that existed already where you could use `go test -cover` and the `-o` flag to produce a binary from `go test` rather than actually running tests. So you could build a binary that had coverage enabled. Then when you exercised code paths by running that binary with whatever inputs, coverage would be collected and dumped into some local file.

Here's an example of the build step: https://github.com/multiprocessio/datastation/blob/main/runn....

And here's how I'd analyze coverage after using the binary: https://github.com/multiprocessio/datastation/blob/main/runn.... This analysis actually combined both unit test coverage files with integration test coverage so it gave a pretty good overall indication of code paths covered by any tests.

I can't remember where I found this technique but it's been around for a while.

This new option is the same thing but a way to `go build` with `-cover` instead of `go test -cover -o $out`? Do I have that right?

Another example of doing this from 2019: https://stackoverflow.com/a/55640874/1507139. I'm sure I learned about this from somebody's blog but I can't remember where.

In case you have never used it, https://app.codecov.io/ can watch your GitHub repo and run tests + coverage. There's a nice UI to then see the coverage % for each folder and also whether a PR or commit is increasing or decreasing coverage.

If I recall correctly, there's zero configuration needed for Go projects. You can be up and running in minutes.

I think Codecov is free for open source projects and paid for private repos, someone more knowledgeable can chime in.

I find the value of code coverage goes down when the size of the instrumented codebase goes up. It's helpful for smaller tests as you can eyeball where you may be missing cases but on integration tests you end up getting a high coverage percentage just by bringing up the system and not even asserting anything (I guess it doesn't crash :) ).

If you want to methodically measure integration test coverage it seems more valuable to enumerate all your critical features or user flows and attribute test cases to them.

A bit more convoluted, but it's also possible in Rust

https://github.com/pldubouilh/rust-coverage-integration-test

This is a really cool feature, thanks very much for adding it. I'd love to see this in more languages. I'd love to use it in my Rust integration tests.

Go tooling keeps getting better

Could we hack this to better understand what your code is doing in prod?

Let me describe a scenario:

Let's say you just inherited some service that is already running in production but it don't have a great documentation nor a good test suite. To make matters worse you don't really know how this service is being used and what are the critical paths inside it.

What I'm thinking is if it would be possible to use this feature to collect some real world usage data for this application. And this data could give you a better understanding about what is actually important in the service.

For APIs we already have a part of this data, as most server in Go have the number of requests for each endpoint. But I think in many cases a more granular data could be really helpful.

Yes, I know that the number of times some function was called isn't a reliable indicator of the importance of said function. But it should be better than nothing.

And I also know that something like this would have a negative performance effect that would not be acceptable for some situations.

It is (non-sarcastically) good to see the Go team changing their opinion on the usefulness of it: https://github.com/golang/go/issues/16537

A colleague of mine wrote a tool to make the process of dealing with coverage data a bit simpler: https://github.com/gotesttools/limgo

Oh, interesting. Full on integration testing for Go CLI apps is an area in which I am sorely lacking.

Does anyone have recommendations of tooling to generally help instrument testing CLI tools? Hopefully better than what I’ve done thus far which is basically a shell script of

    ./mybinary <<< $INPUT > output.txt 2> error.txt
    diff expected-output.txt output.txt
    diff expected-error.txt error.txt

A few years ago, I used goc to collect coverage from multiple services at the same time when they were used in E2E tests. I was very happy with it. Check it out!

https://github.com/qiniu/goc

I guess this can be used to eliminate dead code from your projects as well. Cases that you thought were executed but, in reality, are not.

I am curious if the overhead of this feature allows for running it in production for some time, let’s say a day.

Cool. I definitely felt the lack of this.