The garbage collector has not been an issue, because I use memory the same way as if I were doing it in the C language: Allocate all memory used by the game at program launch. I find this scheme easier to work with in Go than in C, because Go's slice type is a natural fit for partitioning these launch-allocated blocks of memory (as they get used and reused during the game).
The garbage collector doesn't get called if you don't allocate. And for the most part¹, it's easy to reason about where Go allocates (if you know C). So my rule is: Always know where your memory is; never allocate.
This may sound like it's going against the grain of Go, but I find Go to be handily amenable to this style. I had my concerns going in, but it just hasn't been a problem. My game runs silky smooth.
At this point, my only concern about using Go to make a game is that bringing the game to consoles does not have a well-trodden path.
¹ So far, the only thing that has surprised me is that assigning a value type (like an int) to an interface allocates! (So another rule is to only assign pointers to interfaces [pointers to memory allocated at launch].) And while this was a surprise to me, I was pleased with how quick the debugging went: I noticed frames were dropping, so I ran go's pprof tool, which led me directly to the (freshly coded) interface assignment that was stealth allocating. (Also, this allocation becomes less surprising the more I think about how interfaces are implemented.)