Building means downloading dependencies means an ever-growing module cache with no ability to prune it.
rm -rf ~/go
1. This will blow away any programs in ~/go/bin
2. Nothing in ~/go/pkg/mod has +w, so the rm -rf will not work anyway. Try
go clean -modcache