The problem is that if you clone the Git repository, or view it on GitHub, you have no assurance that you're seeing the same code that the go command or the Go module proxy saw. The author of a malicious module could change the Git tag to point to a different, benign, commit after the Go module proxy stores the malicious copy. There are other tricks an attacker can play as well: https://github.com/golang/go/issues/66653

Ultimately, if you're doing a code audit, you have to compute the checksum of the code that you're looking at, and compare it against the entry in go.sum or the checksum database to make sure you're auditing the right copy.