Ah, that's a good point. You just need to remember to set the right environment variables. Unless you need to link against MSVC stuff, although I don't know if Go supports that anyway.
Doesn't look like go works with msvc yet - the following issue was a fascinating read (partly because of the optimism and persistence displayed, partly in a ohmigoditsatrainwreckcantlookaway-way ;)
It isn't always as simple as setting some environment variables. None of the projects I work on will compile when told to compile for another OS_ARCH because of the dependencies they have.