Yes it is a real concern. If you're running a Linux distribution, chances are you are downloading your packages from a mirror and not the primary mirror for your distribution. This is done to everyone's benefit bandwidth-wise.

However it opens up the possibility that the artefacts can be tampered with on the server. Signing confirms their authenticity. In a (cryptographically secure) hash, there is no 'key' and so anyone can create a valid one for their modified bundle.

Ditto for containers, Android Apps, and just about anything else we use.