Curious - what is frustrating about it? I found the process very easy and with no weird bugs, but I only made a thin layer over a supplied C SDK, so maybe my use of cgo is not representative?
Pure Go? don't need anything else beside the Go compiler.
If you can avoid CGO, Avoid it.
When cross compiling with cgo you need a C cross compiling toolchain for the target platform installed with the right libc, etc. It is not impossible and that is how C/C++ is always cross compiled, but it is much more hassle than compiling pure Go.