In fact, most packages are now wheels, which are not sources: they are compressed, and may contain binaries for compiles extensions, something extremely popular (the scientific and AI stacks exist only because of this).
Some packages need to be compiled after the fact, something that setup.py will trigger, and some even embed a fallback compiler, like some cython based packages.
Also, remember there is very few people working on pypi, there is no moderation, anybody can publish anything, so you would need a bullet proof automated heuristic. That's either impractical, or too expensive.
If you want a secure package distribution platform, there are commercial ones, such as anaconda. You get what you pay for.
The most obvious (but not necessarily easiest) approach is to phase out setup.py and move everything to the declarative pyproject.toml approach. This is not just better for metadata (setup scripts make it really hard to statically infer what deps a lib has), it also allows for better control over what installers/toolchains run on install.
Attackers still have quite a lot of latitude during the build phase, but at least libraries have the option to specify declaratively what permissions they need (and presumably the user has the option to forbid them).
Also eval/exec are terrible and I wish there were a mode to disable their usage, but I don't know if the python runtime has some deep dependency on it. Maybe there's a way to restrict it so that only low level frames can call the eval opcode.
The hardening could mitigate on-compilation hacking.
Obviously, this leaves "compile in the backdoor and wait for the user to fall into it" but at least this isn't an issue of compiling on the user's computer and it isn't a issue of binary blobs. And possibly there's a greater chance of detection if actual source code has to be available to compile.
You can use audit hooks in the sys module (as long as you load it first) to disable eval/exec/process spawning or even arbitrary imports or network requests.
1. https://github.com/ossillate-inc/packj flags malicious/risky packages.
Even then, that only pushes the problem down a layer: you’re still fundamentally installing third party code, which can do whatever it pleases. The problem then becomes one of static analysis, for which precision is the major limitation (in effect, just continuing the cat-and-mouse game.)
Why will it be "useless". Explain your reasoning please.
That being said, I don’t think PyPI would be «useless» - this was the state a few years ago, and we had to compile all the libraries ourselves. I don’t want to go back.
It's a convenience/security trade-off, I see.
The only solution I've ever seen to that requires investing trust in an "authority" which then becomes corrupt and censorial. One simply expands the dilemma to a triad; security/freedom/convenience.
If I am not mistaken the PyPI "Cheese Shop" is owned by the Python Software Foundation, a 501(c3) nonprofit organisation which constitutionally values Software Freedom highly. It seems natural that convenience would be sacrificed if security is of concern.
But there is a tradeoff between having things thoroughly vetted and tested, and moving fast.
- security
- freedom
- convenience
- speed/newness
But none of that is a call to give up. We just need to think seriously about the problem we face.
At best I suppose we could put in place checks to get low hanging fruits. But we are, after all, allowing a turing complete and highly dynamic language to execute.
> or just disallow unfamiliar .pth files from being installed altogether
That would kill the entire plugin ecosystem.
Now, the next thing could be to have a permission system, requesting access to the network, fs, .pth, etc. It would not be a bad idea, given that we are, after all, installing things that are as powerful as apps.
But it would be a gigantic effort, and users still would just accept without reading, like they do with apps.
But I've made a ton of web apps and tools anyway, including a little process launcher that plays the role of poor man's Docker.
It'd be nice if those popular systems had a pure Python capability anyway, similar analogy being software rendered 3D back in the day.
It essentially bans binary blobs yet it is very useful.
Providing a build env for that would make it prohibitively complicated and expensive, and basically would mirror github CI.
That's the reason continuum is making money: they sell a python package distribution channel that is checked and locked.
The problem is a failure to understand security.