Hacker News Reader

Comment by geofft | Hacker News Reader

Anyone can upload anything to PyPI. This is kind of like saying that you detected malicious packages on GitHub - the question is whether anyone actually ran it.

They say that the packages were downloaded 30,000 times, but automated processes like mirrors can easily inflate this. (As can people doing the exact sort of research they were doing - they themselves downloaded the files from PyPI!) Quoting PyPI maintainer Dustin Ingram https://twitter.com/di_codes/status/1421415135743254533 :

> *And here's your daily reminder that download statistics for PyPI are hugely inflated by mirrors & scrapers. Publish a new package today and you'll get 1000 'downloads' in 24 hours without even telling anyone about it.*

View on HN

Part of the issue is that FOSS, libraries, and independendent package managers and their specific repositories have exploded in about every domain. No longer are there a handful of places where software and libraries exist. Pick an ecosystem and there's probably a sub or additional levels of package/library management ecosystems below it. Developers have really bought into grabbing a package for everything and leveraging hundreds and thousands of packages, most of which have limited to no sort of vetting. We've had software complexity growing over the years, but the one benefit in previous years is that it was in fairly concentrated areas where many eyes were often watching. You could somewhat rely on the fact that someone had looked through and approved such additions to a package repo. It's a naive security but there were more professional eyes you could leverage, lowering overall risk.

Not anymore, it's more of this breakneck speed, leverage every package you can to save resources and glue them together without looking at them in detail, because the entire reason you're using them is because you don't have time. It's not all shops, plenty of teams vet or roll their own functionality to avoid this but there's a large world of software out there that just blindly trusts everything down the chain in an era where there should be less trust. Some software shops have never seen a package or library they didn't like and will use even trivial to implement packages (the benefit of your own implementation being you know it's secure and won't change under your feet unless an inside threat makes the change). There's a tradeoff to externalizing costs and tech debt for maintainance you pass on using these systems, the cost being you take on more risk in various forms.

> Anyone can upload anything to PyPI. This is kind of like saying that you detected malicious packages on GitHub - the question is whether anyone actually ran it.

There's a bigger social problem here. In many communities it has become completely normalized for any dependency to be just added from these types of "anyone can upload" repositories without any kind of due diligence as to provenance or security. It's as if these communities have just given up on that.

For example, if I suggest that a modern web app only use dependencies that ship in Debian (a project that does actually take this kind of thing seriously), many would laugh me out of the building.

The only practical alternative in many cases is to give up trying. It's now rare for projects to properly audit their dependencies because the community at large isn't rallying around doing it. It's a vicious circle.

This kind of incident serves as a valuable regular reminder of the risks that these communities are taking. Dismissing this by saying "anyone can upload" misses the point.

> It's now rare for projects to properly audit their dependencies

In the Python ecosystem, it is at least pretty easy to limit yourself to a handful of developers you trust (e.g. Django developers, Pallets developers, etc.).

In the npm ecosystem however, for instance I just ran `npx create-react-app` and got a node_modules with 1044 subdirectories, a 11407-line yarn.lock, and "207 vulnerabilities found" from `yarn audit`. Well what can you possibly do.

>> if I suggest that a modern web app only use dependencies that ship in Debian (a project that does actually take this kind of thing seriously), many would laugh me out of the building

And you are more right, and they are more wrong than they know.

Not only are malicious inserts in code a problem in themselves, if you have failed to properly vet your dependencies and it causes real losses for one of your users/customers, YOU have created a liability for yourself. Sure, many customers may never figure it out, and it might take them a while to prove it in court, but if it even gets to the point where someone is damaged and notices, and decides to do something about it, you have defense costs.

The "whatever" attitude has no place in serious engineering of any kind, and anyone with a "move fast and break things" attitude (unless these tests are properly sandboxed) shows that they are not engaged in any serious engineering.

If anyone wonders about the same for NPM, it is around 400. That what happened to my almost empty package.

https://i.imgur.com/Ryr2voN.png

Doesn't installing a python package from PyPI (optionally) run some of the code in the package? Like "setup.py" ? I'd take advantage of that if I were injecting malicious code in a module.

Downloading a Python package (as done by scrapers, mirrors and security analysts) does not run setup.py. Only if the module is installed is this run.

It's analagous to downloading vs. running an executable.

Ah, sure. Just making the distinction that you don't have to actually use a module within your code. That installing the module, even if you never use it in your own code, runs some of the code in that module.

Doing a pip install actually runs the setup.py of the package for source dist, which means running an executable.

It's not the case for wheels though, so you can protect yourself by restricting to binary : --only-binary.

Also doing a pip download is not sensible to this issue, but most people do pip install

Yep. In fact, I recently had to deal with this monstrosity https://pypi.org/project/awslambdaric whose setup.py invokes a shell script https://github.com/aws/aws-lambda-python-runtime-interface-c...

That shell script runs 'make && make install' on a couple of bundled dependencies, but in principle it could do anything https://github.com/aws/aws-lambda-python-runtime-interface-c...

For what it's worth, npm supports an option "ignore-scripts" for both "npm install" and "npm ci" (the latter of which ensures the installed packages match the integrity hashes from the package-lock.json file).

https://docs.npmjs.com/cli/v7/commands/npm-install/#ignore-s...

https://docs.npmjs.com/cli/v7/commands/npm-ci/#ignore-script...

Reply on news.ycombinator.com