> nothing stops people with enough privileges from modifying that cached version

They may be able to modify it but with some kind of trusted hash at least I'll know about it. Making a change to a repo that keeps the same version and also keeps the same hash is a more difficult attack to pull off.