Pypi.org is running a survey on the state of Python packaging
pypi.org
pypi.org
With python, it feels like half the time I don't even have the right version of python installed, or it’s somehow not on the right path. And once I actually get to installing dependencies, there are often very opaque errors. (The last 2 years on M1 were really rough)
Setting up Pytorch or Tensorflow + CUDA is a nightmare I've experienced many times.
Having so many ways to manage packages is especially harmful for python because many of those writing python are not professional software engineers, but academics and researchers. If they write something that needs, for example, CUDA 10.2, Python 3.6, and a bunch of C audio drivers - good luck getting that code to work in less than a week. They aren’t writing install scripts, or testing their code on different platforms, and the python ecosystem makes the whole process worse by providing 15 ways of doing basically the same thing.
My proposal:
- Make poetry part of pip
- Make local installation the default (must pass -g for global)
- Provide official first party tooling for starting a new package
- Provide official first party tooling for migrating old dependency setups to the new standard
edit: fmt
managing npm and package.json which has dependency issues.
With python it always has been pip install package and we are done.
Although I do share your pain with versioning. I once spent a week debugging an issue only to find out that there's a fixed version available.
As a user of an application: `npm install` just works (same with `cargo build`). For Python, I’ll probably do `python -m venv env; . env/bin/activate` and then, well, it’s probably `pip install -r requirements.txt`, but sometimes it’ll be other things, and there are just too many options. I may well add --ignore-installed and use some packages installed locally of potentially different versions, e.g. for setting up Stable Diffusion recently (first time I’ve ever used the dGPU on my laptop) I wanted to use the Arch Linux packages for PyTorch and the likes.
For managing dependencies in an existing library or application (as distinct from starting from scratch, where you’ll have to make several extra choices in Python, and where npm is badly messed up for libraries), both npm and Python are generally fairly decent, but the whole you-can-only-have-one-version-of-a-library thing from Python lends itself more to insurmountable problems.
My personal background: many years of experience with both npm and Python, but I’ve never done all that much packaging in either. (Also many years of Rust, and I have done packaging there, and it’s so much easier than both.)
The worst I encountered was a somewhat lengthy manual to build a project (cannot remember what it was), and not only did you have to manually install all required dependencies -- npm automates this by having an npm-readable list in package.json -- but after running all those commands, the last note said something like "oh BTW, whenever we wrote "pip" above we actually meant "pip3", use that instead, and if you did it wrong, there's no way to undo it."
I agree with that, its pretty painful
Give, quite a bit. The egg format was introduced in 2004 [0], and even the newer wheel, that replaced it, is turning 10 later this month.
[0] https://packaging.python.org/en/latest/discussions/wheel-vs-...
Official dependency and build tooling is not properly geared for C extensions. You'll be looking at compiler errors to figure out what dependencies you are missing.
This is why we created Linux distributions in the first place. It is not the place of every language to reinvent the wheel - poorly.
> There should be one– and preferably only one –obvious way to do it.
Oh no…
> Oh no…
That sentence doesn't mean what you think you means. It is saying that there should be "only one way to do something".
Does not mean
> There should only ever be one way to do each thing.
But somehow extremely many people read it like that.
By default, yes. It is possible to install ML-package-1.2 into your 'base' python and then have virtual environments A,B and C all use that instead of installing their own copies. However this is generally not considered best practice.
I can see that as an option in tools like venv or poetry, possibly setting a "common" location to share across environments.
I completely agree with your feeling. I been working with js, php, ruby for years, and packaging is pretty straightforward in every one of them. And the lang versioning works for every one of them (php is the most annoying, yeah, but even on php is easier than in python). Ruby has several alternatives like rbenv or rvm, but any of them works has every feature 99,9% of developers needs, and they work just fine. I want that in python ):
I don't know how I'd feel about poetry being part of pip. Didn't poetry just push out a change where they deciding randomly failing 5% of the time while running in a CI env was a good idea?
The Python ecosystem is a bit of a disaster. There are many useful packages and programs written in Python, but they're legit scary to install. I always search for Go or JS alternatives if they're available to avoid all the python headaches.
Every time I need to interact with Python, I find myself banging my head trying to figure out eggs vs wheels vs distutils vs setuptools vs easyinstall vs pip vs pip3 vs /usr/bin/pip3 vs /opt/homebrew/bin/pip3 vs pip3.7 vs pip3.8 vs pipx vs pip-tools vs poetry vs pyenv vs pipenv vs virtualenv vs venv vs conda vs anaconda vs miniconda.
I’m curious because this gets repeated a lot by many people. What specific messes do you get into?
I’m asking because my experience with python these days is always just doing “python -m venv .venv” activating it and then using “pip install -r requirements.txt”
To update dependencies I do “pip install —-upgrade dep” run tests and then go “pip freeze > requirements.txt” this never fails me. Though sometimes updating dependencies does make tests fail of cause, but that’s the fault of the individual packages not the packaging system. Even so I’d say even that is rare for me these days.
I know the might only work for 95% of the workflows out there, but I’m very curious as to what specific messes the last 5% end up struggling with and what makes people like you feel that it’s always a mess and not just “sometimes it gets messy” etc.
Personally, it feels better in some ways and worse in others.
The whole pom.xml approach/format that Maven uses seems decent, all the way up to specifying which registries or mirrors you want to use (important if you have something like Nexus). Although publishing your own package needs additional configuration, which may mean putting credentials in text files, though thankfully this can be a temporary step in the CI pipeline.
That said, personally I almost prefer the node_modules approach to dependencies that Node uses (and I guess virtualenv to a degree), given that (at least usually) everything a particular project needs can easily be in a self-contained folder. The shared .m2 cache isn't a bad idea, it can just make cleaning it periodically/for particular projects kind of impossible, which is a shame.
I think one of the aspects that make dependencies better in JVM land is the fact that you oftentimes compile everything your app needs (perhaps without the JVM, though) into a .jar file or something similar, which can then be deployed. Personally, I think that that's one of the few good approaches, at least for business software, which is also why I like Go and .NET when similar deployments are possible. JVM already provides pretty much everything else you might need runtime-wise and you don't find yourself faffing about with system packages, like DB drivers.
That said, what I really dislike about the JVM ecosystem and frameworks like Spring, is the reliance on dynamic loading of classes and the huge amounts of reflection-related code that is put into apps. It's gotten to the point where even if your code has no warnings and actually compiles, it might still easily fail at runtime. Especially once you run into issues where dependencies have different versions of the same package that they need themselves, or alternatively your code doesn't run into the annotations/configuration that it needs.
Thankfully Spring Boot seems like a (decent) step forwards and helps you avoid some of the XML hell, but there is still definitely lots of historical baggage to deal with.
Personally, I like Python because of how easy it is to work with once things actually work and its relatively simplistic nature and rich ecosystem... but package management? I agree that it could definitely use some work. Then again, personally I just largely have stuck with the boring setup of something like pip/virtualenv inside of containers, or whatever is the most popular/widespread at any given moment.
Of course, Python is no exception here, trying to work with old Ruby or Node projects also sometimes has issues with getting things up and running. Personally I feel that the more dependencies you have, the harder it will be to keep your application up and running, and later update it (for example, even in regards to front end, consider React + lots of additional libraries vs something like Angular which has more functionality out of the box, even if more tightly coupled).
Next on the list would be pypi namespaces, but there are good reasons why that is very hard.
The mission statement they are proposing, “a packaging ecosystem for all”, completely misses the mark. How about a “packaging ecosystem that works” first?
I spent a bunch of time recently fixing our internal packaging repo (nexus) because the switch from md5 hashes to sha256 hashes broke everything, and re-locking a bajillion lock files would take literally months of man hours time.
I’ve been a Python user for the last 17 years, so I’m sympathetic of how we got to the current situation and aware that we’ve actually come quite far.
But every time I use Cargo I am insanely jealous, impressed and sad that we don’t have something like it. Poetry is closest, but it’s a far cry.
What alternative is there for me?
My package has a combination of hand-built C extensions and Cython extensions, as well as a code generation step during compilation. These are handled through a subclass of setuptools.command.build_ext.build_ext.
Furthermore, I have compile-time options to enable/disable certain configuration options, like enabling/disabling support for OpenMP, via environment variables so they can be passed through from pip.
OpenMP is a compile-time option because the default C compiler on macOS doesn't include OpenMP. You need to install it, using one of various approaches. Which is why I only have a source distribution for macOS, along with a description of the approaches.
I have not found a non-setup.py way to handle my configuration, nor to provide macOS wheels.
Even for the Linux wheels, I have to patch the manylinux Docker container to whitelist libomp (the OpenMP library), using something like this:
RUN perl -i -pe 's/"libresolv.so.2"/"libresolv.so.2", "libgomp.so.1"/'
/opt/_internal/pipx/venvs/auditwheel/lib/python3.9/site-packages/
auditwheel/policy/manylinux-policy.json
The non-setup.py packaging systems I've looked at are for Python-only code bases. Or, if I understand things correctly, I'm supposed to make a new specialized package which implements PEP 518, which I can then use to boot-strap my code.
Except, that's still going to use effectively arbitrary code during the compilation step (to run the codegen) and still use setup.py to build the extension. So it's not like the evil disappears.
I’m suggesting we find a better way to build them, something a bit more structured, and decouple that specific use case from setup.py.
It would be cool to be able to structure this in a way that means I can describe what system libraries I may need without having to execute setup.py and find out, and express compile time flags or options in a structured way.
Think of it like cargo.toml va build.rs.
But it appears to be such a hard problem that modern packaging tools ignore it, preferring to take on other challenges instead.
My own attempts at extracting Python configuration information to generate a Makefile for personal use (because Makefile understand dependencies better than setup.py) is a mess caused by my failure to understand what all the configuration options do.
Given that's the case, when do you think we'll be able to "Remove setup.py files and mandate wheels"?
I'm curious on what evils you're thinking of? I assume the need to run arbitrary Python code just to find metadata is one of them. But can't that be resolved with a pyproject.toml which uses setuptools only for the build backend? So you don't need to remove setup.py, only restrict when it's used, yes?
In my answers to the survey, I mentioned "nix" was the technology most likely to affect the future of Python packaging, in part because of reading that same article on Riff.
I think now I should have mentioned Riff too.
My specific use cases are adding custom CA certs to certifi after it is installed, and modifying the maximum version of a requirement listed for an abandoned library that works fine with a newer version.
I think the best solutions would be an official way to ignore dependencies for a specific package, and specify replacement packages in a project's dependencies. Something like this if it were a Pipfile:
public-package = {version = "~=1.0",replace_with='path/to/local-package'}
abandoned-package = {version = "~=*",ignore_dependencies=True}
I know your point is more about requiring static metadata than using wheels per se. I just believe that all things Python should be flexible and hack-able. There are other more rigid languages if you're into that sort of thing.
edit:
before anyone starts getting angry I know there are other ways to solve the problems I mentioned.
forking/vendoring is a bit of overkill for such a small change, and doesn't solve for when a dependency of a dependency needs to be modified.
monkeypatching works fine, however it would need to be done at all the launch points of the project, and even then if I open a repl and import a specific module to try something it won't have my modifications.
modifying an installed package at runtime works reasonably well, but it can cause a performance hit at launch, and while it only needs to be run once, it still needs to be run once. So if the first thing you do after recreating a virualenv is to try something with an existing module we have the same problem as monkey patching.
'just use docker' or maybe the more toned down version: 'create a real setup script for developers' are both valid solutions, and where I'll probably end up. It was just very useful to be able to modify things in a pinch.
https://setuptools.pypa.io/en/latest/userguide/pyproject_con...
Then orf dreamed of a day where you could "describe what system libraries I may need without having to execute setup.py and find out, and express compile time flags."
The link you pointed doesn't appear to handle what we were talking about. By specifying "gfortran", I hoped to highlight that difference.
riff, building on nix, seems an intriguing solution for this.
It is a better idea to do instruction selection at runtime in the code that currently uses AVX2. I recently wrote some docs for Debian contributors about the different ways to achieve this:
But if the compiler by default doesn't enable AVX2 then it will fail to compile the AVX2 intrinsics unless I add -mavx2.
Even worse was ~10 years ago when I had an SSSE3 code path, with one file using SSSE3 intrinsics.
I had to compile only that file for SSSE3, and not the rest of the package, as otherwise the compiler would issue SSSE3 instructions where it decided was appropriate. Including in code that wasn't behind a CPUID check.
Thus crash on hardware without SSSE3.
See https://stackoverflow.com/questions/15527611/how-do-i-specif... for more info about my solution. Someone last year contributed a solution for MS Windows.
Automatic multi-versioning doesn't handled what I needed, at least not when I started.
I needed a fast way to compute the popcount.
10 years ago, before most machines supported POPCNT, I implemented a variety of popcount algorithms (see https://jcheminf.biomedcentral.com/articles/10.1186/s13321-0... ) and found that the fastest version depended on more that just the CPU instruction set.
I ended up running some timings during startup to figure out the fastest version appropriate to the given hardware, with the option to override it (via environment variables) for things like benchmark comparisons. I used it to generate that table I linked to.
Function multi-versioning - which I only learned about a few month ago - isn't meant to handle that flexibility. To my understanding.
I still have one code path which uses __popcountll built-in intrinsics and another which has inline POPCNT assembly, so I can identify when it's no longer useful to have the inline assembly.
(Though I used AVX2 if available, I've also read that some of the AMD processors have several POPCNT execution ports, so may be faster than using AVX2 for my 1024-bit popcount case. I have the run-time option to choose which to use, if I ever have access to those processors.)
Furthermore, my code generation has one path for single-threaded use and one code path for OpenMP, because I found single-threaded-using-OpenMP was slower than single-threaded-without-OpenMP and it would crash on multithreaded macOS programs, due to conflicts between gcc's OpenMP implementation and Apple's POSIX threads implementation.
The AVX2 popcount is from Muła, Kurz, and Lemire, https://academic.oup.com/comjnl/article-abstract/61/1/111/38... , with manually added prefetch instructions (implemented by Kurz). It does not appear that SIMD Everywhere is the right route for me.
SIMD Everywhere is for a totally different situation; if you want to automatically port your AVX2 code to ARM NEON/etc without having to manually rewrite the AVX2 intrinsics to ARM ones.
I think at the point a programming language is going on about "mission statements" for a packaging tool, you know they've lost the plot
copy Maven from 2004 (possibly with less XML)
that's it, problem solved
Especially installing a tool chain to compile C code for python is no issue on Linux, but such a pain on Windows.
Every language tries to re-implement the package manager, but it ends up breaking down as soon as you need to interact with anything outside of that specific language's ecosystem. The only solution for interacting with the "outside" (other languages, toolchains, etc) is a system level, language agnostic package manager of some kind.
Linux distros package management is far from perfect but it's still miles ahead of the alternatives!
I very highly recommend people to learn how to write and create Linux packages if they need to distribute software. On Arch for example this would be creating PKGBUILDs, Gentoo has ebuilds, and other distros have something similar to these things.
It works OK in nuget on Windows, due to the different approach taken by the OS maintainer.
A DLL compiled 25 years ago for Windows NT stills work on a modern Windows, as long as the process is 32-bit. A DLL compiled 15 years ago for 64-bit Vista will still work on a modern Windows without issues at all.
People who need native code in their nuget packages are simply shipping native DLLs, often with static link to C runtime. Probably the most popular example of such package is SQLite.
> I very highly recommend people to learn how to write and create Linux packages if they need to distribute software.
I agree. When working on embedded Linux software where I control the environment and don’t care about compatibility with different Linuxes or different CPU architectures, I often package my software into *.deb packages.
When juggling as many transitive dependencies in C as folks do with node, python, etc., there's plenty of pain to deal with.
I make a tool/library that requires the C toolchain at runtime. That's even worse than build time, I need end users to have things like lld, objdump, ranlib, etc installed anywhere they use it. My options are essentially:
- Requiring users to just figure it out with their system package manager
- Building the C toolchain from source at build time and statically linking it (so you get to spend an hour or two recompiling all of LLVM each time you update or clear your package cache! Awesome!),
- Building just LLD/objdump/.. at build-time (but user still need to install LLVM. So you get both slow installs AND have to deal with finding a compatible copy of libLLVM),
- Pre-compiling all the C tools and putting them in a storage bucket somewhere, for all architectures and all OS versions. But then not have support when things like the M1 or new OS versions right away, or people on uncommon OSes. And now need to maintain a build machine for all of these myself.
- Pre-compile the whole C toolchain to WASM, build Wasmtime from source instead, and just eat the cost of Cranelift running LLVM 5-10x slower than natively...
I keep trying to work around the C toolchain, but I still can't see any very good solution that doesn't make my users have extra problems one way or another.
Hey RiiR evangelism people, anyone want to tackle all of LLVM? .. no? No one? :)
For comparison, pacman (the Arch Linux package manager) packages have fairly similar ".PKGINFO" file in them; but in order to support resolving dependencies without downloading the packages, the server's repository index includes not just a listing of the (name, version) tuple for each package, it also includes each package's full .PKGINFO.
Enhancing the PyPA "Simple repository API" to allow fetching the METADATA independently of the wheel would be a relatively simple enhancement that would make a big difference.
----
As I was writing this comment, I discovered that PyPA did this; they adopted PEP 658 in March of this year! https://github.com/pypa/packaging.python.org/commit/1ebb57b7...
Also, there's a difference between a pep being adopted and that pep being implemented (usually a bunch of elbow grease). That said there are a couple exciting steps towards 658 being implemented: https://github.com/pypa/pip/pull/11111 (just approved yesterday, not yet merged) https://github.com/pypi/warehouse/issues/8254 (been open for forever, but there has been incremental progress made. Warehouse seems to not attract the same amount of contribution as pip)
$ gzcat release_data/c/d/cdklabs.cdk-hyperledger-fabric-network.json.gz | jq '. | to_entries | .[].value.info.requires_dist' | head
[
"typeguard (~=2.13.3)",
"publication (>=0.0.3)",
"jsii (<2.0.0,>=1.63.2)",
"constructs (<11.0.0,>=10.0.5)",
"aws-cdk-lib (<3.0.0,>=2.33.0)"
]
I see I misunderstood korijn's comment. My earlier reply is off-topic, so I won't continue further off the track.
This would make most C extensions impossible to install on anything other than x86_64-pc-linux-gnu (or arm-linux-gnueabihf/aarch64-linux-gnu if you are lucky) because developers don't want to bother building wheels for them.
I view dependencies outside of the standard library as a kind of technical debt, not because I suffer from Not Invented Here and want to code it myself, no, I look and think, "Why isn't this in the standard library with a working set of idioms around it?"
I haven't developed anything with more than five digits of code to it, which is fine for me, but part of it is just ... avoidance of having to screw with libraries. Ran into a pip issue I won't go into (it requires a lot of justification to see how I got there) and just ... slumped over.
This has been a bad spot in Python for a long, long time. While people are busy cramming their favorite feature from their last language into Python, this sort of thing has languished.
Sadly, I have nothing to offer but encouragement, I don't know the complexities of packaging, it seems like a huge topic that perhaps nobody really dreamed Python would have to seriously deal with twenty years ago.
This is exactly why I prefer the larger opinionated web frameworks (Django, Vue.js) to the smaller more composable frameworks (Flask, React). I don’t what to make decisions every time I need a new feature, I want something that “just works”.
Python and Django just work, and brilliantly at that!
With Django there is a happy path answer for most everything. If I run into a problem, I know I'm not the first.
It seems ridiculous to me that there isn't a native method for something as simple and ubiquitous as putting a thread to sleep, or that there is an external library (underscore) that provides 100+ methods that seem to be staples in any modern language.
Python is nice in that way. It is also opinionated in a cohesive and community driven manner, e.g. PEP8.
Honestly I doubt see the package management being an issue for most end-users. Between the builtin venv, conda and Docker I feel that the use-cases for most is well covered.
The only focus area I really see is better documentation. Easier to read documentation more precisely. Perhaps a set of templates to help people getting start with something like pyproject.
It feels like the survey is looking for a specific answer, or maybe it’s just that surveys are really hard to do. In any case I find responses to be mostly: I have no opinion one way or the other.
Really good code in the standard library should reach a level of near perfection, then eventually transition into hopeful speed gains, after which you're really only changing that code because the language has changed or the specification has updated.
That's an interesting position. So are you suggesting that very specialised packages such as graph plotting, ML-packages, file formats, and image processing should be part of the standard library? What about very OS/hardware-specific packages, such as libraries for microcontrollers?
There are many areas that don't have a common agreed-upon set of idioms or functionality and that are way too specialised to be useful for most users. I really don't think putting those into the standard library would be a good idea.
OS-specific packages, quite possibly.
The thing about the standard library is that it is like high school: there's a lot of stuff you think you will never need, and you're right about most of it, but the stuff you do need you're glad you had something going, at least.
Including too much leads to a huge burden for the maintainers and consequently results in this: https://peps.python.org/pep-0594/
Quote:
> Times have changed. With the introduction of PyPI (née Cheeseshop), setuptools, and later pip, it became simple and straightforward to download and install packages. Nowadays Python has a rich and vibrant ecosystem of third-party packages. It’s pretty much standard to either install packages from PyPI or use one of the many Python or Linux distributions.
> On the other hand, Python’s standard library is piling up with cruft, unnecessary duplication of functionality, and dispensable features.
Minimal standard library languages let you pick and choose what needs to be run. Golang is a nice happy medium since it’s compiled.
If I own example.com, I should be able to have 'pypi.org/example.com/package'. The domain can be tied back to my (domain verified) GitHub profile and it opens up the possibility of using something like 'example.com/.well-known/pypi/' for self-managed signing keys, etc..
I could be using the same namespace for every package manager in existence if domain validated namespaces were common.
Then, in my perfect world, something like Sigstore could support code signing with domain validated identities. Domain validated signatures make a lot of sense. Domains are relatively inexpensive, inexhaustible, and globally unique.
For code signing, I recognize a lot of project names and developer handles while knowing zero real names for the companies / developers involved. If those were sitting under a recognizable organizational domain name (example.com/ryan29) I can do a significantly better job of judging something's trustworthiness than if it's attributed to 'Ryan Smith Inc.', right?
It's not without problems. One is that folks often don't control the domain (consider Go's charming habit of conflating version control with package namespacing). Another is what was noted below: resurrection attacks on domains can be quite trivial and already happen in other forms (eg registering lapsed domains for user accounts and performing a reset).
I don't know if registries would play ball with something like that, but it would be useful and should probably exist anyway. I would even argue that once a domain rolls through grace, redemption, etc. and gets dropped / re-registered, that should invalidate it as an account recovery method everywhere it's in use.
There's a bit of complexity when it comes to the actual validation because of stuff like that. I think you'd need buy in from at least one large company that could do the actual verification and attest to interested parties via something like OAuth. Think along the lines of "verify your domain by logging in with GitHub" and at GitHub an organization owner that's validated their domain would be allowed to grant OAuth permission to read the verified domain name.
Of course there are going to be complexities involving key-rollover and migrating to a different domain, but a sufficiently intelligent Sigstore client could handle the various messages and cryptographic proofs needed to secure that. The hard part is how to issue a new key if you lose the old one, since that probably requires social vouching and a reputation system.
A principal reason to use sigstore is to get out of the business of handling private keys entirely. It turns a key management problem into an identity problem, the latter being much easier to solve at scale.
That's a good point in terms of invalidation, but a new domain registrant should be able to claim the namespace and start using it.
I think one possible solution to that would be to assume namespaces can have their ownership changed and build something that works with that assumption.
Think along the lines of having 'pypi.org/example.com' be a redirect to an immutable organization; 'pypi.org/abcd1234'. If a new domain owner wants to take over the namespace they won't have access to the existing account and re-validating to take ownership would force them to use a different immutable organization; 'pypi.org/ef567890'.
If you have a package locking system (like NPM), it would lock to the immutable organization and any updates that resolve to a new organization could throw a warning and require explicit approval. If you think of it like an organization lock:
v1:
pypi.org/example.com --> pypi.org/abcd1234
pypi.org/example.com --> pypi.org/ef123456
Losing control of a domain would be recoverable because existing artifacts wouldn't be impacted and you could use the immutable organization to publish the change since that's technically the source of truth for the artifacts. Put another way, the immutable organization has a pointer back the current domain validated namespace:
v1:
pypi.org/abcd1234 --> example.com
pypi.org/abcd1234 --> example.net
I think that's a reasonably effective way of solving attacks that rely on registering expired domains to take over a namespace and it also makes it fairly trivial for namespace owners to point artifact consumers to a new domain if needed.
Think of the validated domain as more of a vanity pointer than an actual artifact repository. In fact, thinking about it like that, you don't actually need any cooperation or buy in from the domain registries.
> The hard part is how to issue a new key if you lose the old one, since that probably requires social vouching and a reputation system.
It's actually really hard because as you increase the value of a key, I think you decrease the security practices around handling them. For example, some people will simply drop their keys into OneDrive if there's any inconvenience associated with losing them.
I would really like to have something where I can use a key generated on a tamper proof device like a YubiKey and not have to worry about losing it. Ideally, I could register a new key without any friction.
* Dependencies are managed in a similar way to Go - where hashes of installed packages are stored and compared client side. This means that a hijacker could only serve up the valid versions of packages that I’ve already installed.
* This is still a “centralized” model where a certain level of trust is placed in PyPi - a mode of operation where the “fingerprint” of the TLS key is validated would assist here. However it comes with a few constraints.
Of course the above still comes with the caveat that you have to trust pypi. I’m not saying that this is an unreasonable ask. It’s just how it is.
The Google/trillian database which supports Google's CT logs uses Merkle trees but stores the records in a centralized data store - meaning there's at least one SPOF Single Point of Failure - which one party has root on and sole backup privileges for.
Keybase, for example, stores their root keys - at least - in a distributed, redundantly-backed-up blockchain that nobody has root on; and key creation and revocation events are publicly logged similarly to now-called "CT logs".
You can link your Keybase identity with your other online identities by proving control by posting a cryptographic proof; thus adding an edge to a WoT Web of Trust.
While you can add DNS record types like CERT, OPENPGPKEY, SSHFP, CAA, RRSIG, NSEC3; DNSSEC and DoH/DoT/DoQ cannot be considered to be universally deployed across all TLDs. Should/do e.g. ACME DNS challenges fail when a TLD doesn't support DNSSEC, or hasn't secured root nameservers to a sufficient baseline, or? DNS is not a trustless system.
EDNS (Ethereum DNS) is a trustless system. Reading EDNS records does not cost EDNS clients any gas/particles/opcodes/ops/money.
Blockcerts is designed to issue any sort of credential, and allow for signing of any RDF graph like JSON-LD.
List_of_DNS_record_types: https://en.wikipedia.org/wiki/List_of_DNS_record_types
Blockcerts: https://www.blockcerts.org/ https://github.com/blockchain-certificates :
> Blockcerts is an open standard for creating, issuing, viewing, and verifying blockchain-based certificates
W3C VC-DATA-MODEL: https://w3c.github.io/vc-data-model/ :
> Credentials are a part of our daily lives; driver's licenses are used to assert that we are capable of operating a motor vehicle, university degrees can be used to assert our level of education, and government-issued passports enable us to travel between countries. This specification provides a mechanism to express these sorts of credentials on the Web in a way that is cryptographically secure, privacy respecting, and machine-verifiable
W3C VC-DATA-INTEGRITY: "Verifiable Credential Data Integrity 1.0" https://w3c.github.io/vc-data-integrity/#introduction :
> This specification describes mechanisms for ensuring the authenticity and integrity of Verifiable Credentials and similar types of constrained digital documents using cryptography, especially through the use of digital signatures and related mathematical proofs. Cryptographic proofs enable functionality that is useful to implementors of distributed systems. For example, proofs can be used to: Make statements that can be shared without loss of trust,
W3C TR DID (Decentralized Identifiers) https://www.w3.org/TR/did-core/ :
> Decentralized identifiers (DIDs) are a new type of identifier that enables verifiable, decentralized digital identity. A DID refers to any subject (e.g., a person, organization, thing, data model, abstract entity, etc.) as determined by the controller of the DID. In contrast to typical, federated identifiers, DIDs have been designed so that they may be decoupled from centralized registries, identity providers, and certificate authorities. Specifically, while other parties might be used to help enable the discovery of information related to a DID, the design enables the controller of a DID to prove control over it without requiring permission from any other party. DIDs are URIs that associate a DID subject with a DID document allowing trustable interactions associated with that subject.
> Each DID document can express cryptographic material, verification methods, or services, which provide a set of mechanisms enabling a DID controller to prove control of the DID. Services enable trusted interactions associated with the DID subject. A DID might provide the means to return the DID subject itself, if the DID subject is an information resource such as a data model.
Abstract: https://link.springer.com/chapter/10.1007/978-3-662-53357-4_...
/? "Certificate Transparency" Blockchain https://scholar.google.com/scholar?q=%22Certificate+Transpar... https://scholar.google.com/scholar_alerts?view_op=list_alert...
- Some of these depend upon a private QKD [fiber,] line
- NIST PQ algos are only just now announced: https://news.ycombinator.com/item?id=32281357 : Kyber, NTRU, {FIPS-140-3}?
/? Ctrl-F "Certificate Transparency" https://westurner.github.io/hnlog/ :
"Google's Certificate Transparency Search page to be discontinued May 15th, 2022" https://news.ycombinator.com/item?id=30781698
- LetsEncrypt Oak is also powered by Google/trillian, which is a trustful centralized database
- e.g. Graph token (GRT) supports Indexing (search) and Curation of datasets
> And what about indexing and search queries at volume, again without replication?
My understanding is that the s Sigstore folks are now more open to the idea of a trustless DLT? "W3C Verifiable Credentials" is a future-proof standardized way to sign RDF (JSON-LD,) documents with DIDs.
Verifiable Credentials: https://en.wikipedia.org/wiki/Verifiable_credentials
# Reproducibile Science Publishing workflow procedures with Linked Data:
- Sign the git commits (GPG,)
- Sign the git tags (GPG+Sigstore, ORCID & DOI (-> W3C DIDs), FigShare, Zenodo,)
- Sign the package(s) and/or ScholarlyArticle & their metadata & manifest ( Sigstore, pkg_tool_xyz,CodeMeta RDF/JSON-LD, ),
- Sign the SBOM (CycloneDx, Sigstore,)
- Search for CVEs/vulns & Issues for everything in the SBOM (Dependabot, OSV,)
- Search for trusted package hashes for everything in the SBOM
- Sign the archive/VM/container image (Docker Notary TUF, Sigstore,)
- Archive & Upload & Restore & Verify (and then Upgrade Versions in the) from the dependency specifications, SBOM, and/or archive/VM/container image (VM/container tools, repo2docker (REES),)
- Upgrade Versions and run unit, functional, and integration tests ({pip-tools, pipenv, poetry, mamba}, pytest, CI, Dependabot,))
We need a way to configure an ordered list of indexes pip searches for packages. —extra-index-url or using a proxy index is not the solution.
Also namespaces and not based on a domain. So for example: pip install apache:parquet
Also some logic either in the pip client or index server to minimize typosquatting
Also pip should adopt a lock file similar to npm/yarn. Instead of requirements.txt
And also “pip list” should output a dependency tree like “npm list”
I should not have to compile source when I install. Every package should have wheels available for the most common arch+OS combos.
Also we need a way to download only what you need. Why does installing scipy or numpy install more dependencies than the conda version? For example pywin and scipy.
[[tool.poetry.source]] name = "my-pypi" url = "https://my-pypi-index.wherever" secondary = true
https://github.com/pypi/warehouse/pull/5001 - had to be reverted because it was too noisy
How are you going to name the file storing the wheel for that package? Using ":" on Windows is going to be problematic.
These are connected things.
I maintain a PyPA member project (and contribute to many others), and the latter is aided by the former: the mission statement keeps the community organized around shared goals (such as standardizing Python's packaging tooling), and inclusivity insures a healthy and steady flow of new contributors (and potential corporate funding sources).
THANK YOU!
> keeps the community organized around shared goals (such as standardizing Python's packaging tooling)
Personally I felt some disconnect between "package manager for all" and the need for "standardizing Python's packaging tooling." Yes, communities should be welcoming and friendly to everyone, AND the community should have clear expectations for best practices that members should follow. E.g., is an experienced female developer more likely to give up on contributing because she couldn't find a local meetup, or because she didn't know whether to create pyproject.toml vs requirements.txt? In some sense, the bigger and more diverse the community, the greater the need for a clear, solid foundation. IDK if that's remotely clear; it's just a feeling I had going through some of those questions.
The fiasco was that he was not the best at maintaining projects, and used his popularity from tablib and requests to get pipenv recommended by the PyPA well before it was ready for general use. Then around the same time there was a few scandals I don't remember the details of, something about a developer being mad that Kenneth kept the money being donated to requests. It ended up with all of his popular projects being maintained by others.
I wish the response from the PyPA was to go full in on pipenv and just keep improving it at the same rate it was in the beginning. Instead it stagnated. Poetry came out fragmenting the ecosystem even more. And quite a few developers gave up on virtual environments in favor of docker.
pipenv started having regular releases again and have kept it going for the past few years. I like it enough to not want to put any effort into switching, but in 2017/2018 it really felt like the next big thing. It makes me sad to think that we all lost out because of politics.
Ended up switching to Poetry because it had a superior dependency resolution algorithm.
The only question I have is, what grant application(s) is the survey data being used to support?
Remember not to just complain, but to offer specific problems/solutions--i.e. avoid statements like "virtualenvs suck, why can't it be like NPM?" and prefer instead feedback like "the difference between Python interpreter version and what virtualenv is being used causes confusion".
I think if this PEP makes it in then like 90% of people's pain with pip just completely goes away almost overnight. Love it or hate it the NPM/node_modules style of all dependencies dumped in a local directory solves a _ton_ of problems in the packaging world. It would go a long way towards making the experience much smoother for most python users.
It literally uses npm, the NPM registry, and node_modules for Python dependency management.
P.S. By the way https://peps.python.org/pep-0518/#other-file-formats
Python has had multiple legacy solutions going back a long time before nuget and npm existed, and before central registries of dependencies. Every new solution has to cope with all that compatibility/transitional baggage. Also a bunch of usecases .NET or JS never really had to deal much with - eg being a core system language for Linux distros, and supporting cross platform installs back in the download and run something days. The scope of areas Python gets used in means its packaging is pulled in more directions than most other languages who mostly stick to a main niche.
So the history and surface area of problems to solve in Python packaging is larger than what most other languages have had to deal with. It also takes years for the many 3rd party tools to try out new approaches, gain traction and then slowly get their best ideas synthesized and adapted into the much more conservative core Python stdlib.
Not saying it is great, just laying out some of the reasons it is what it is.
Absolutely agree about the rest of them.
It’s incredible how even Julia’s Pkg.jl supports better python packaging in combination with conda than the official python packaging tools.
This is very clearly a question of the culture of the core python developers (such as brett cannon) who seem to think the machine learning people with their compilers and JITs are not an important part of the community.
They have been peddling the dream of speeding up Python by neglecting the scientific ecosystem for a long time without having either the expertise or a clear plan how to achieve that.
All that while the primarily useful part of Python is its C extension capabilities. The web does not need Python, query/response applications are handled just fine and better by functional languages.
I believe this is the way forward. Existing packages are essentially tied to cpython because that’s what the python.org website offers for download, and people will just use that by default.
Once all major packages support the new c api, it will be easier to just use something other than cpython by default.
Software developers shouldn't need to figure out what build-time dependencies their libraries need...
My package uses C and Cython extensions. While I support macOS and Linux-based OSes, I don't know how to develop on or support MS Windows.
I've tried to be careful about the ILP32 vs LP64 differences, but I suspect there's going to be many places where I missed up.
I also use "/dev/stdin" to work-around my use of a third-party library that has no way to read from stdin. As far as I can tell, there's no equivalent in Windows, so I'll have to raise an exception for that case, and modify my tests cases.
Quoting the best comment, "echo test | type CON or echo test | type CONIN$ will read from the console, not from stdin."
But I'm usually on Windows if I need binary wheels; maybe the coverage is a bit different on Linux.
Pypi doesn't need users to manually to fill survey per se, they need to (optionally) prompt "install failed, submit traceback and error code to pypi for solutions" during setup.
Edit: Typo: reinstalling a version of package currently being used
Conda obviously offers a lot of value in sharing hairy compiled packages, but it does not play well with anything else. None of available the tooling really works with both conda and pip. It fragments the already lousy packaging story.
my main thing w/ conda is it's bananas figuring out how to make a new recipe, which is pretty surprising
Just as an explanation: a "version" in Python packaging can come from one of many potential distributions, including a local distribution (such as a path on disk) that might different from a canonical released distribution on PyPI. Having `pip install ...` always re-install based on its candidate selection rules is generally good (IMO), since an explicit run of `pip install` implies user intent to search for a potentially new or changed distribution.
This one I've seen when the dependency doesn't specify an exact version and you included "--upgrade".
There's a second case that I think was fixed in pip 21 or 22, where two transitive dependencies overlap - A and B depend on C, but with different version ranges. If A allows a newer version of C than B allows, C can get installed twice.
We ran into many unsolvable or 30m+ solvable envs with conda that mamba handled quickly.
The underlying solver can be used with conda directly as well, but I have not done that (https://www.anaconda.com/blog/a-faster-conda-for-a-growing-c...)
I think the answer to Python’s packaging woes has been in plain sight all along. There are so many competing package managers and systems that it is hard to keep track of them all. Of course; this is just what happens in open source, but the problem seems uniquely bad for Python. In Ruby, Swift, Rust or Golang I don’t encounter there being quite the multitude of options for each.
Work on cross comparability and roadmap to merge them all down to eventually one obvious way to install packages.
Lost an hour of time trying to get Stable Diffusion working on an M1 Mac yesterday
Namespace traversal that executes code is just so so bad.
Need to revive the MONTY Python spirit as a hedge against the general dreariness of the day.
lacking binary wheels on PyPI, problems with shipping project with dependencies, confusion about there being multiple "package managers" (pip, pip-tools, poetry, pipenv, conda) and multiple formats of dependency lists (setup.py, setup.cfg, requirements.txt, pyproject.toml, ...), sys.path associated confusion (global packages, user-level packages, and anything specified in PYTHONPATH, ...)
that is different from the french "pipi": pea + pea.
And wait until you learn what "bit" sounds like in French.