Hacker News Reader

Comment by captainmuon | Hacker News Reader

It's a bit ridiculous that setup.py allows you to run custom code on install. Granted, even without this you would run the malware the latest when you import the module. But I've had all kinds of trouble from packages with too smart setup.py scripts. The best Python packages are just "xcopyable", meaning self-contained in a directory that you can just drop in the PYTHONPATH. Or if you really need a binary component, include binaries (plus a simple compilation step for other platforms) on PyPI.

View on HN

> It's a bit ridiculous that setup.py allows you to run custom code on install.

Agreed! The good news is that it's slowly getting better: just yesterday[1], setuptools added support for PEP 660[2], which removes one of the last remaining reasons to use a `setup.py` instead of a fully declarative `pyproject.toml`.

Edit: Also, just to qualify: this is a consistent problem across packaging ecosystems. Even system package managers (reasonably!) allow custom code during installation, e.g. to add service hooks and update locale files.

[1]: https://twitter.com/juanluisback/status/1557734536586625025

[2]: https://peps.python.org/pep-0660/

Hopefully next they'll incentivize package maintainers to migrate to the declarative way.

Perhaps spit a scary sounding security warning for all packages which use a setup.py?

> Perhaps spit a scary sounding security warning for all packages which use a setup.py?

This would be nice, but there's probably too many practical barriers to this. For example, you can't (easily) replace `setup.py`'s dynamism for packages that contain native packages: they need to be able to spawn compilers, etc. and do so in ways that are hard to describe declaratively.

We probably don't want to provide a security warning on every native non-wheel package, so that's a no-go for the time being.

Could we perhaps sandbox any such compilation steps within a container and then copy out the resulting artifact(s), with a hook that does some security scanning?

This is hard to do in the general case: the compilation steps often need system dependencies or to otherwise query system state to do the right thing, which means that the sandbox has to accurately reflect the host.

In terms of hooks, audit hooks and events[1] get us some of the way there. But they can be circumvented as well; I wouldn't rely on them for untrusted code.

[1]: https://docs.python.org/3/library/audit_events.html

Should start with a warning, and move after one year to users having confirm that they allow pqckqges to execute setup.py and ultimatrly not allow it any more.

This would probably break too much of the ecosystem, since many top Python packages include native extensions that can only be built with a dynamic `setup.py`.

Wheels are solving that problem, but it's likely going to be many years before warning about `setup.py` is reasonable and not disruptive to the majority of legitimate users.

Instead there's a scary deprecation warning when a package uses setup.cfg, which is another declarative replacement for setup.py.

And some pyproject.toml features are still experimental per the setuptools docs, so I'm not sure which method to choose for our internal projects.

I'm glad the setuptools/pip/Python devs are cleaning up the packaging situation though. pyproject.toml seems perfect, when it becomes ready.

I published my first Python code in a while using a pyproject.toml and it was amazingly simple. As someone who was doing Rust for a while I really feel like they're learning from the modern ecosystems very well.

Do you have a link to a good tutorial for this?

Follow the setuptools doc [1], it's very easy and there's an example you can start from. It links to PEP 621 [2] which seems to be the specification if that can help.

[1] https://setuptools.pypa.io/en/latest/userguide/pyproject_con...

[2] https://peps.python.org/pep-0621/

Is toml still not an included dependency in python?

There will be a `tomllib` in Python 3.11: https://docs.python.org/3.11/library/tomllib.html

Why the discrepancy between csv having the csv package but toml having the lib suffix

Not in 3.10. It's included in 3.11.

(But this doesn't matter for packaging purposes, since setuptools is also not included in the base distribution, but does include TOML support.)

Still doable with pyproject.

I actually wrote a Python tool for downloading binary releases from GitHub and Gitea and wrote a “builder” module that can be used in a pyproject file to tell it to download an arbitrary binary and run some commands.

Here’s an example in action: https://github.com/IamTheFij/release-gitter/tree/main/sample...

> It's a bit ridiculous that setup.py allows you to run custom code on install.

In Rust it's "worst", you can run custom code at compile time. But IMO the issue is that we are optimizing everything for convenience at the cost of security. The compile/install custom code wouldn't be an problem if each dependencies was pinned and audited.

PyPI requiring 2 factors authentication on "popular" projects will help, but this will not replace the needs for auditing, vetting and pinning.

I'm convinced that, while it's bad policy, there's no real effect - the first thing you do upon installing a new dependency is import it, and importing it... Runs arbitrary code.

yes, compiled Wheels really are the way to go, but anything.py is going to be able to run custom code, that's the point. And like you said, the code will likely be run on the machine anyway. Minor rant, but I think it's much worse that NPM installs can run custom code, since many(most?) npm modules are only run in a browser, it actually does open up more of a security risk.

I'm not so sure it's ridiculous.

What is ridiculous is that users have no way of restricting what packages can run on a fine-grained level, at both build time and runtime.

We should fix that problem. If we do, then we could make computing a lot safer.

Disclaimer: I'm working on that problem.

Many modules, especially a lot of specialized modules that are not available on PyPI and do not provide wheels, have complex installation steps, including compiling C libraries etc. There is nothing foundamentally wrong with this kind of use case.

Reply on news.ycombinator.com