Hacker News Reader

Comment by Galanwe | Hacker News Reader

The size of python packages is becoming an increasingly worrying issue.

A lot of maintainers don't pay attention to excluding test cases and artifacts from their packages, leading to ridiculous package size growth.

Last I checked numpy was close to 100MB, a huge chunk of that being test case artifacts.

Django packages all translations for all languages to ever exist.

Pandas bundles pre-built dynamic libraries with debug symbols.

Etc.

Most of my "production" virtualenvs are close to a GB nowadays, which is insane.

View on HN

This! Especially with pandas.

I did have a look at numpy and on my machine the tests did not bloat it as much as you made me believe. The `core/tests/` modules are 3MB and the `__pycache__` doubled it to 6MB. What do you refer to as "test case artifacts"? The modules, pycache, or both?

Also, wrt you statement on pandas; is it the debug symbols that account for the bloat, or the libs themselves?

I cannot seem to remember which version exactly that was. I remember there were some test input/output files that weighted multiple MBs.

Just installed a quick "data science" like virtualenv, here is the example things I talk about:

$ du -sh scipy 110M scipy $ find scipy -type f -name '.so' -exec strip -s {} \; $ du -sh scipy 76M scipy

$ du -sh django/contrib/admin/locale/ 5.4M django/contrib/admin/locale/

All of those cases sound like good reasons for optionals / extras.

I've had people report missing tests from my python packages as bugs. You can't please all the people all of the time.

Sounds like a bug should be filed towards numpy?

Apparently it was wontfixed last year, but maybe it could still be suggested to be an extra, not in the standard package? https://github.com/numpy/numpy/issues/19591

That was implied in the thread and subsequently wonfixed.

> Sure, but it's straightforward to install those [0]. A separate test suite would not be.

0: Those = Other packages you would have to download to make use of the testing components, i.e. pytest

What about pip install 'numpy[tests]' i.e using the extras feature?

Reply on news.ycombinator.com