For all of these reasons it is not possible in the general case (the one that package management needs to target) to simply distribute binaries.
Linux distributions are not a working example here because the libraries are built for the use of the other packages in the same repository. Needs of third parties are never considered because finding library configurations that work for the packages in the distro is hard enough.
A platform independent package manager for C++ must effectively degrade into a build system that rebuilds all dependencies from source. And it must allow build parameters for each step to be defined by the dependent libraries or programs, so that the result is something that can be linked together successfully.