Casks are the only things Homebrew does that some other package manager available on macOS doesn't reliably do better. Nix, Pkgsrc, MacPorts, and (and now Spack) all have better fundamental designs; sane, multi-user-friendly permissions; and enough isolation from the base system that they break neither each other nor manually-installed software.
I use Homebrew exclusively tucked away in isolated prefixes, only to install casks, and without ever putting any binaries it installs along the way on my PATH. I don't remember which programs it is, exactly, but I do use a few that are unsigned.
It also doesn't seem to me that the signing process is as vital in determining actual risk as the curation and moderation processes involved in maintaining "third-party" software distributions like Homebrew or Debian or whatever.
`--no-quarantine` in particular is one of the conveniences that makes Homebrew casks useful. If I have to give my consent anew for each app update, I might as well install the apps manually and live in the usual auto-update pop-up hell.