Yeah, you're right on the money. This is basically insane in ML, but _especially_ in corporate environments, think F500, where sudo is disabled in containers and experimentation (in terms of trying different package versions) is quite challenging due to security restrictions.
Frankly, it's one of the reasons that we're seeing a lot of excitement around stuff written in Rust, but with Python bindings[1][2]. Cargo makes it so easy to build across various platforms and then just produce statically compiled platform specific python wheels. If PyTorch and TF had been written using Rust, any other other ultra high perf language with a tool-chain as reliable as cargo, a lot of folks lives would be easier.
[2]: https://www.pola.rs/