If you're downloading random unsigned debs/rpms from the internet and installing them, then you're just as wrong as downloading/execing a binary.

If you're downloading rpms/debs, verifying the signatures, and you trust the signer, then you're probably fine. Those signature signing/checking mechanisms are already built into rpm/deb, which is why they're being argued for.