Yes, a distro package manager can partially solve this issue but there are some corner cases where this fails. Though I partially agree with you: my OS package manager can provide me with the C and C++ libs I need most of the time.
To name a C++-specific pain in the ass: Boost. The Boost libraries make minor, incompatible changes every now and then (which is good), requiring applications that use Boost to either specify a certain version that they can work with or bundle the Boost libraries with the application source (which is bad).
And "development" package manager should be able to deal with the situation above, but distro package managers don't really work here. There are several libboost-foo-1.xy.z packages available but it may be that none of those will work with a certain application you're trying to build (either too old or too new).
This situation is handled by development package managers by pinpointing the exact version (or range of versions) of a library an application depends on. This also allows specifying compiler/interpreter version(s) an application works with. Some of my projects depend on features that are only available in recent GCC versions (__builtin_shuffle), but that requirement can only be informally specified in the README, nothing enforces it (leading to unnecessary bug reports).
This is also what git submodules does but without a standard build system, it only solves half of the problem (and may create new problems).
I think you and I can both agree that this situation is less than ideal. Using just one package manager application should be enough, there's a lot of overlap between what a distro package manager and a development package manager do. And it's a huge waste of engineering resources to maintain language-specific package managers such as pip/gem/cabal/cpan.
"Next generation" package managers such as Nix and GNU Guix should be able to address these issues, but they are yet to be widely adopted. Unfortunately those applications are based on less-than-mainstream languages (Haskell and Guile Scheme, respectively) which might scare away potential adopters.