I tend to just repackage every Python package as an OS package, these days the tools to do that are reasonably good, and it forces you to inspect the contents for licensing and other junk/cruft/badness.
Have you tried out Spack? I used this quite a bit in academia to install the scientific sorts of libraries many Python packages depends on without needing to rely on the OS versions which were often out of date.
I'd probably just use the OS backports, or backport it myself rather than Spack.