Even if you don't get the public key through a web of trust, you download it "once" not every time you download a file, then you keep using it until it expires.

You also typically download it from a different place than the storage location of the signed binary artifacts. This means that an adversary will have a hard time trying to replace a public key and remain undetected.