Caching proxies for zip downloads sounds nice, but it's more than just "a bit more infrastructure". I think it would be a huge burden to package publishers if each of them has to manage their own dependency zip mirror as a separate piece of infrastructure. You need version control anyway; checking your dependencies into that same version control does not require a new piece of infrastructure.
Coming from Ruby, where rubygems.org is a very painful point of failure, in my eyes the fact that Go dependencies are not a separate download is a big plus.
In fact without a single blessed dependency repository such as rubygems.org, in the Go case you have as many points of failure at build time as there are different code hosting sites in your dependency graph.