Hacker News Reader

Hacker News

Comment by awinter-py | Hacker News Reader

Full thread

awinter-py

have always wondered why pypi doesn't generate whl files for pure-python sdists

and why companies like travis / github aren't more active in language-level packaging work

github gives away so much free docker time -- faster installation would save them money directly

View on HN

Because there is no way to know if something is pure-python or not.

Even more so with pep517 and being able to use different build systems.

wait then how does bdist_wheel know to make platform-dependent vs platform-independent whl files?

Your hunch is right, setuptools is able to determine if it's pure python or not (https://packaging.python.org/guides/distributing-packages-us...)

Maybe it's not reliable enough?

In any case, I can see why PyPI wouldn't want to increase the scope of their work. Sometimes is super valuable to just be good at one thing (distribute what others give you).

I think the pypi folks just don't have the resources / capacity to implement this. With appropriate funding this could have been done (development efforts wouldn't be that big, but probably operational costs for the build farms). In the past I ran (devpi builder)[https://github.com/blue-yonder/devpi-builder/] with (devpi)[https://pypi.org/project/devpi/] to build wheels for all my dependencies and keep them stored in a place I have control over.

That's a specific implementation that relies on setuptools. An sdist doesn't have to use setuptools to be an sdist.

Wheels can be generated out of thin air, there is no requirement that setuptools or any tool is ever involved, if you wanted you could zip up a particular directory structure and it is a wheel.

are you saying pure-python detection is possible for standard packaging formats like setuptools and pyproject.toml, but not for sdist generally?

pyproject.toml specifies how to build a project using a builder. You'd have to execute the builder and let it build the project to know if the wheel it generates is pure python or not.

As for setuptools:

There's a class that setuptools relies on to do the actual building of the Python project, called distutils.dist.Distribution which has a function named is_pure():

https://github.com/python/cpython/blob/e42b705188271da108de4...

Which makes sure that it doesn't have any known external modules or c libraries.

So to do detection of whether an sdist is a pure python, you'd have to execute setuptools and setup.py because of course setup.py isn't just configuration, it's full on Python.

So just looking at an sdist statically you can not know if the sdist is a pure python project or not.

underrated comment.

pypi could probably generate a lot of wheels. In fact, `pip` generates wheels from packages it downloades before installing them, so wheels can be generated for a lot of packages already directly from sdist.

Why it isn't done already? Good question. I guess it has to do with a majority of python packages installing just fine from source (those that aren't a problem for packaging anyway), and then those packages that need wheels because they use native extensions kind of are high-maintenance to bundle and roll out, especially if upstream maintainers haven't invested in building their projects for that.

I would love to see auto-bundled wheels. but I guess for this to fly, package maintainers would need to start standardizing a bit better (that also involves tooling). Easy things are missing, for example in setuptools you cannot even define smoke tests that distributors could use to verify successful bundling.

Reply on news.ycombinator.com