Using anything other than distro packages is a pain. pip is an anti-pattern and one giant security risk.
The best way of using Python though, is to create a virtualenv (since Python 3, venv module is built in which makes it straight forward, python -m venv <directory>) and install packages inside of it. This gives you control to use exact versions of packages you need in your applications and makes your application not tied to the OS, so system upgrades and Python version updates are much easier to perform.
Is this true for drivers, too? like the coupling of cuda versions to tensorflow versions?
If you have any dependencies that depend on system libraries there are two options:
- the libraries can be compiled statically, they call them manylinux wheels, this generally works well, except if your dependencies are overlapping with python or other packages' dependencies. Most commonly this happen if package depends on openssl. If the compiled-in openssl is different than the version on the system in certain circumstances python might crash. This is for example why psycopg2 was initially distributed as manylinux but now they opt for the second method (they still provide psycopg2-binary but they discourage its use; most time it works fine but it is a problem on certain distros)
- make python create bindings on installation, this makes installation process do small compilation to create bindings between system library and python, this makes the packages robust, but tied to the OS. In my experience this is not an issue, but it can be annoying, because you still don't have 100% control of dependencies.
This issue is what made me investigate Nix[2] package manager, because it gives you full control of all dependencies down to libc making everything fully reproducible, so you can control the exact version of python to use and all system and python dependencies.
[1] in one of my previous jobs on a team that I joined they were running their applications on CentOS 5 that was already EOL, because rebuilding RPMs to CentOS 7 (at the time the newest version) was a lot of work, another issue was that they were bound to python package version that came with the system, they could create their own RPMs, but no one did it because maintaining that was adding more work. I spent time converting the python code to be packaged using setuptools. Once that was done switching the OS was trivial. We finally also could use the latest versions of many of our dependencies.
[2] This is IMO good article which describes tooling that makes Nix much more enjoyable to use: https://christine.website/blog/how-i-start-nix-2020-03-08