Hacker News Reader

Comment by gary_0 | Hacker News Reader

(2022)

I remember this one because of this part:

> Unbeknownst to me, even with --dry-run pip will execute arbitrary code found in the package's setup.py. In fact, merely asking pip to download a package can execute arbitrary code (see pip issues 7325 and 1884 for more details)! So when I tried to dry-run install almost 400K Python packages, hilarity ensued. I spent a long time cleaning up the mess, and discovered some pretty poor setup.py practices along the way. But hey, at least I got two free pictures of anime catgirls, deposited directly into my home directory. Convenient!

View on HN

I've had to clean up after npm packages as well. Git hooks I didn't ask for. Shell modifications I didn't ask for. I hope that, one day, these package managers will utilize sandboxing/containerization to avoid messes like this.

Python and node software is something I always try to run in containers for convenience and security reasons

Here's the line that copies those pictures in: https://github.com/akazukin5151/koneko/blob/master/setup.py#...

Why is this ecosystem considered acceptable?

It’s because academia is a sort of anchor that preserves poor practices.

You'll have to ask every developer why it's acceptable, because they keep using this ecosystem. And then ask their managers. But this is what happens when you don't use package ecosystems with maintainers being run by adults.

If I wanted to compile a Go binary using an existing dynamically loaded C library, compiled with -ffast-math, what happens?

Well it's an ecosystem that is meant to be easy to use by people who don't know what they're doing.

> Why is this ecosystem considered acceptable?

It largely isn't, but python is a big ship and it takes time to turn. There's been a lot of movement in python packaging semi-recently and as far as I can tell using setup.py has been considered legacy for a while now.

Biggest problem is that the new way to do packaging is not documented very well. It's split over three or four different projects. It mostly amounts to "you need to make a pyproject.toml file" but it's somewhat tricky to find what such a file is even supposed to contain.

Again, why is this acceptable?

Okay, it's unacceptable. Now what? Sadly, we (you, I, the Python leadership) can't just visit every single Python package publisher/maintainer and ask politely but firmly ask them to fix their packaging issues. And even if we could, at least half of them would either ignore us or would mess up in some new, exciting ways while trying to adapt to the flavour du jour of Python packaging.

I think the answer is to not have a flavour du jour. Pick one, crown it the winner, document it well and give it some time. If the solution is always changing, nobody is going to want to put the effort in.

This is really a great idea. Once we finalize the choice, when can we expect the upload into the hivemind?

Right after the maintainers of libtelepathy.so release the version with Discord support.

> I think the answer is to not have a flavour du jour. Pick one, crown it the winner, document it well and give it some time.

It has been done with distutils: it has been around for more than a decade. The result is the ecosystem that you are complaining about.

This is exactly our problem. We develop a couple of packages (which we don’t put on PyPI so any security issue is affects nobody except us). But it’s not our job and we cannot justify someone spending too much time on this. I have a look every now and then to try to figure out what we’re supposed to do, but there are 154 solutions, some of them are outdated and unmaintained, some of them are broken in more or less subtle ways, and some of them have a bare minimum of documentation. So yeah. We’ll change when we have a clear, documented path forward.

Just put your package folder into a tarball as-is; to install, execute

    tar xzf our_custom_package.tar.gz -C "$(python3 -c 'import sysconfig; print(sysconfig.get_path("platlib"))')"

I'm only half-joking.

Not to mention that the Python packaging tutorial was missing a few bits, at least as of earlier this year: https://daveon.design/introducing-fontimize-subset-fonts-to-...

Speaking of poor programming practices, how about copy-pasting code without understanding it? Notice the author's mention of how many packages with -Ofast trace their lineage back through this comment:

    # Initially copied from
    # https://github.com/actions/starter-workflows/blob/main/ci/python-package.yml
    # And later based on the version jamadden updated at
    # gevent/gevent, and then at zodb/relstorage and zodb/perfmetrics

Although the linked config was modified 2 years ago to removed the unsafe math options[1], the copy-paste propagated before then.

Naturally anyone asking CoPilot, ChatGPT, or other modern LLM-based interface for config code will likely get something based on this, with the problematic -Ofast option included.

[1] https://github.com/zopefoundation/meta/commit/9c07520e90d9a1...

The same problems happen in any language that links C and C++ libraries largely.

It's less common for other languages because they just don't do it as much.

The "--only-binary=:all:" option should force Pip to only install packages that provide wheels, which doesn't run arbitrary setup.py code (right? I've been assuming it doesn't and really hope that is a valid assumption).

That will cause some installations to fail if wheels are not available. However, given that wheels are increasingly common (even for pure-python "source" packages), this can be used as a sort of bisect to enumerate and isolate/audit/file issues on/sandbox/etc. the remaining setup.py-based distributions.

Depends on the platform you're on, some platforms have very few wheels.

Naive question, does this have the same effect if I use virtual environments?

Yes. Virtual environments aren’t a full sandbox.

Yes, because the line that does it is literally:

  os.system('cp -r ./pics/ ~/.local/share/koneko/')

that's a feature. You can conveniently do

  pip install nukemydrive #no, this package doesn't exist (yet)

rather than than the harder to remember

  rm -rf --no-preserve-root /

I don't use pip but tbh this makes me much less likely to even try it. So many things about it sound unsafe.

Like dry-run not even working as you'd expect? I wonder if that got fixed after this article?

Probably the fix will be to rename the option --dank-run

Reply on news.ycombinator.com