Actually, binary package managers are arguably strictly less safe than getting the binaries directly from upstream because you've just introduced a (literal) "man in the middle", who might have their own agendas and their own need to monetize. Like, for example ... SourceForge did. In a different context these MITM attacks can go wrong even when they're genuinely trying to be helpful, as we may remember when Debian accidentally patched out important code from a core encryption library and caused everyone to get the same SSH keys.
Generally for security you want to reduce the number of middlemen who can tamper with things, unless those middlemen are explicitly adding some sort of security value by pre-tasting the apps. So, Apple app reviewers: yes. Linux distro maintainers: sorta sometimes. Homebrew, winget etc: no. It's all automated.