I will certainly mention Conda in contexts where it makes sense to do so.
The thing about conda is that it is not just a tool, it is a completely different ecosystem to handle what you describe in your post. You touched on the topic of building & packaging, distribution, environment management, etc. And all these are different in conda land. To me the answer to the problems you mentioned in the post is conda. But I think it is commonly misunderstood and some people hate to use it. Not to mention needing to maintain publishing in both PyPI and a conda channel is laborious.
This matches my understanding. But I also understand that Conda doesn't treat Python as anything special; it just provides its own kind of environments for multi-language computing.
Many people do swear by Conda and I should look into it eventually. (In some far future, maybe Paper can work in a Conda environment and avoid the problems that Pip encounters there.) If I'm going to fix problems (and educate about them) on the Python-first side, I need to have priorities, though.
> The genesis of Conda came after Guido van Rossum was invited to speak at the inaugural PyData meetup in 2012; in a Q&A on the subject of packaging difficulties, he told us that when it comes to packaging, "it really sounds like your needs are so unusual compared to the larger Python community that you're just better off building your own"
From https://jakevdp.github.io/blog/2016/08/25/conda-myths-and-mi...
This was already in my research pile. "Myth #2" there is what I meant about "not treating Python as anything special". The GvR quote is infamous, and seen as short-sighted, but it made sense in context.
The story is often told (though the link doesn't do this) as if the "main" Python community intended to ostracize the PyData folks. But 12 years later there is still much important dialog with them, and NumPy is one of the most commonly downloaded PyPI packages. My discussions on discuss.python.org exposed me to entire websites of analysis of the existing problems (like https://pypackaging-native.github.io/) and people like Henry Schreiner and Ralf Gommers are driving much of the discussion.
> in that non pure Python case, compiler is needed at some point, and that is going to cause problems if you have no control over it. Conda essentially achieve reproducibility, with the necessary compiler considered as part of the dependencies.
This is one of the major differences, yes: Conda has a system for ensuring you have the right compiler. It's hoped that PEP 725 "Specifying external dependencies in pyproject.toml" (https://peps.python.org/pep-0725/) will be a first step towards resolving this in the Pip world, or perhaps towards better interoperation with Conda. (Of course, the hard part is actually using that metadata to install compilers.)
One question I have in mind when dealing with PyPI vs conda is that why wouldn’t conda already solved all the problems with PyPI. My own answers to that includes 1) PyPI is still the de facto place to claim a namespace and it would be foolish to only release it in conda and not PyPI, and 2) there are many more packages available in PyPI and not conda, and the effort to just make all PyPI packages I used available in PyPI is too laborious.
Another issue is that many packages available on PyPI are not packaged well, from version constraints to setup.py madness to licensing issues. People managing the conda-forge channel is more gate keeping about those issues. I wonder even if the tooling around PyPI is improved, the community culture is much more difficult to be changed.