You should be able to pull all your modules from a proxy, minus any private repos you control (which you can either setup our own proxy for OR set The GOPROXY var to go direct to vcs for it).
Also, not vendoring should give you a smaller repo and build speed shouldn’t suffer too much (proxy caching and such).