Quirks of Python package versioning
sethmlarson.dev
sethmlarson.dev
In my opinion, the best version comparison algorithm is that of the Debian package manager. Super simple, and very flexible. The description of the algorithm fits in just three paragraphs[3]. Most other algorithms implemented in the aforementioned library were actually implemented in terms of it. Prereleases, fourth version numbers, etc. anything you want basically fits nicely in that box.
PEP 440 was the only version comparison algorithm that couldn't be neatly implemented in terms of that of Debian's, beating out semver, maven, and rubygems for complexity.
1: http://djhaskin987.gitlab.io/serovers/index.html
2: https://peps.python.org/pep-0440/
3: https://www.debian.org/doc/debian-policy/ch-controlfields.ht...
In other words: Debian prioritizes unified versioning and has a "fan-out" topology of users; PyPI prioritizes facilitating dependency resolution for a "graph" topology of users.
It may have been designed for that use case, but so were these others. They also had simpler designs.
I would have agreed with this previously, but a year or so ago they changed their backend and now it just errors out on certain (maybe not common, but extant) version strings, such as cloudera’s spark release version.
> packaging.version.InvalidVersion: Invalid version: '2.4.0.cloudera2'
If I recall, it used to parse fine on their old backend parser. Sure, the maintainers can cast aspersions on uneducated devs from their ivory tower, amazed anyone would think this is a valid version. But things like this can and do show up in the wild, and the parsing package playing dumb (especially when it didn’t use to) doesn’t inspire a lot of confidence that this should be the tool you use.
Just parse the damn thing. It’s painfully obvious how to do so. I should have to write my own custom parser to wrap theirs. Sure, it’s nonstandard, but it should be handled as a best guess + a warning, not an exception. I didn’t choose this version string. Don’t punish me for trying to parse it.
(Nobody anywhere has called developers "dumb" for using legacy, non-PEP 440 versions. There's nothing else they even could have done prior to version standardization across the Python ecosystem.)
[1]: https://blog.yossarian.net/2022/05/09/A-most-vexing-parse-bu...
Further, there would be nothing wrong with returning something like a NonconformingVersion object, where we can best guess at things like major, minor, patch. Parse what you think you know, and when you run into something that is nonconforming, then you fail over to return an object that has what you think you know and what you can’t sort out. E.g: in the above example, major, minor, patch versions are apparent. I would even say the vast majority of nonconforming versions have at least one or more of major/minor/patch readily parseable, the problem is we aren’t sure exactly how of handle the rest of it. And that’s fine.
This approach is super pythonic in the BTAFTP philosophy, where I will let users decide what to do with the object returned. “You asked me to parse this mess. I can’t fully do that, but here’s a best guess.” This is worlds better than just failing completely.
a.b.c
one.2.3
1.pre.2.3.4
2004d
1.0beta5prerelease
0.1-charmander
0.1-bulbasaur
Whether or not it's preferable to attempt to parse unspecified inputs boils down to questions of predictability, reliability, and security and, IMO, security is an overriding factor when it comes to domains as critical as packaging.
That being said: you can attempt to parse legacy versions using this package[2].
I actually think it makes sense for the PyPI itself to be the one to dictate how to handle version policy via some callable or handler. Many companies have internal versions and mirrors of OSS code, and allowing them to preferentially shim these in (and reasonably version) these for their employees. Let’s say you fork the requests library. So now your versioning is requests=2.31.0.corporate-31. Sure, it makes no sense externally, but as the internal PyPI authority, you can enforce this versioning. Then your users just point to your index and get what they need (even if they just ask for requests=2.31.0).
In general, parsing and dependency resolution are two separate issues. The concerns of dependency resolution should not bleed into parsing. So it should be a valid use case to simply want to parse and compare two versions, and if it seems we have complete enough information, then make a truth estimation. E.g. in the above example, this should realistically not fail, but instead throw a NonconformingVersion warning:
> assert parse(‘2.4.0.coudera2’) >= parse(‘2.4’)
While we can’t parse the full string, we can make a really good, consistent guess at what the user is after. And if needed, parse could reasonably take an optional callable to support handling of nonconforming versions to convert them to conforming ones.
>>> from distlib.version import LegacyVersion, NormalizedVersion
>>> NormalizedVersion('2.4.0cloudera2')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/disk2/vinay/projects/scratch/distlib/distlib/version.py", line 33, in __init__
self._parts = parts = self.parse(s)
^^^^^^^^^^^^^
File "/disk2/vinay/projects/scratch/distlib/distlib/version.py", line 276, in parse
result = _normalized_key(s)
^^^^^^^^^^^^^^^^^^
File "/disk2/vinay/projects/scratch/distlib/distlib/version.py", line 188, in _pep_440_key
raise UnsupportedVersionError('Not a valid version: %s' % s)
distlib.version.UnsupportedVersionError: Not a valid version: 2.4.0cloudera2
>>> LegacyVersion('2.4.0cloudera2')
LegacyVersion('2.4.0cloudera2')
If in addition to the above you want to mutate Python versions, you may consider my parver package[1]. It also doesn't normalise the input unless you ask it to, so you can preserve your preferred format after mutating.
This is mentioned in the post, but the actual package was not identified.
Another Python versioning gotcha I've never see discussed is use of instructions like AVX2 in C extensions. There seems to be no way in Python versioning to say "this version is for CPUs with AVX2 instructions" so I guess public C extensions have to be compiled for the least common denominator instruction set. Does anyone know otherwise? How do other packaging systems like cargo deal with this?
That's unfortunate; I know Poetry's support for epochs has also been spotty (at least historically, I'm not sure about currently).
> There seems to be no way in Python versioning to say "this version is for CPUs with AVX2 instructions" so I guess public C extensions have to be compiled for the least common denominator instruction set. Does anyone know otherwise?
This is my understanding as well: x86_64 in wheel tags implies the lowest common denominator for AMD64. I think the best you can generally do here is either (1) accept the pain that comes with sdists, or (2) do runtime feature detection for things like AVX2.
> How do other packaging systems like cargo deal with this?
Cargo always builds from source, so they effectively punt on this problem.