Hacker News Reader

Comment by physicsguy | Hacker News Reader

`uv` is great but the biggest issue with Python packaging right now continues to be getting it right for scientific and ML packaging.

Want to install PyTorch? which one? CUDA? Oh, OK, then you have to get the wheel directly from them because there are 6 different versions for differnet CUDA versions, and the wheels are too large for PyPi anyway.

Conda offers only a partial resolution to this problem. Spack is great at being ultra configurable and having all the C/C++/Fortran dependencies and compiler toolchains you need, and so allowing ekeing out best performance, but doesn't integrate well with uv etc. so it's difficult to take an experimental ML project written by a researcher and take it through to productionisation with it.

View on HN

I had previously got around this by using Anaconda but I don't really like the amount of crap that brings in either, and also it leads to a dev environment that doesn't look anything like production so that sucks too... as a result of which I'm back in the boat you're describing.

It's still better than it was 10 years ago (compiling everything from source wasn't uncommon even then!) but it's painful. Comparisons to other interpreted languages aren't really fair though, if JS/TS had as many compiled C libraries, they'd be in exactly the same positions. Go doesn't have the same compiled library thing because of the focus of excluding C dependencies wherever possible.

Scipy maintainer here, the main issue with the wheels was the Fortran77 that was SciPy throwing wrenches into the mix. With C/C++ self compilation should be quite straightforward. We (all Scientific Python packages) really worked hard on that.

From version 1.19 of SciPy there will be no need for fortran compilers (because we translated everything to C https://github.com/scipy/scipy/issues/18566) and then all becomes much easier in all platforms due to the large availability of C compilers in all platforms. Together with the Stable API developments in CPython the wheel clash issues "hopefully" will decrease gradually.

Thanks for all your hard work! I already consider SciPy and NumPy to be best practice in this area but as I'm sure you know, it's the long tail of stuff built on top of these and other core packages that are really the issue!!

Ah, my bad. what I failed to emphasize is that many of the downstream issues are coming from the upstream restrictions so this is one of the major blocks that was causing some mayhem down the line. So indirectly we might have caused some heartburn for you, apologies in advance.

Reply on news.ycombinator.com