Comment by appplication | Hacker News Reader

Comment by appplication | Hacker News Reader

> Instead use the official packaging.version.Version class to parse, compare, and reason about Python versions.

I would have agreed with this previously, but a year or so ago they changed their backend and now it just errors out on certain (maybe not common, but extant) version strings, such as cloudera’s spark release version.

> packaging.version.InvalidVersion: Invalid version: '2.4.0.cloudera2'

If I recall, it used to parse fine on their old backend parser. Sure, the maintainers can cast aspersions on uneducated devs from their ivory tower, amazed anyone would think this is a valid version. But things like this can and do show up in the wild, and the parsing package playing dumb (especially when it didn’t use to) doesn’t inspire a lot of confidence that this should be the tool you use.

Just parse the damn thing. It’s painfully obvious how to do so. I should have to write my own custom parser to wrap theirs. Sure, it’s nonstandard, but it should be handled as a best guess + a warning, not an exception. I didn’t choose this version string. Don’t punish me for trying to parse it.

"Just parse the damn thing" without a grammar leads to exploitable parser differentials[1]. Your "obvious" parse isn't everyone else's.

(Nobody anywhere has called developers "dumb" for using legacy, non-PEP 440 versions. There's nothing else they even could have done prior to version standardization across the Python ecosystem.)

[1]: https://blog.yossarian.net/2022/05/09/A-most-vexing-parse-bu...

I understand that the lack of standardization makes things difficult, but there are a lot of knowables.

Further, there would be nothing wrong with returning something like a NonconformingVersion object, where we can best guess at things like major, minor, patch. Parse what you think you know, and when you run into something that is nonconforming, then you fail over to return an object that has what you think you know and what you can’t sort out. E.g: in the above example, major, minor, patch versions are apparent. I would even say the vast majority of nonconforming versions have at least one or more of major/minor/patch readily parseable, the problem is we aren’t sure exactly how of handle the rest of it. And that’s fine.

This approach is super pythonic in the BTAFTP philosophy, where I will let users decide what to do with the object returned. “You asked me to parse this mess. I can’t fully do that, but here’s a best guess.” This is worlds better than just failing completely.

I understand the frustration here (I maintain a project that was directly broken by strict version checking), but I also think that reducing the problem to "best guess at major, minor, patch" understates the problem here. For example, what do you do with version strings like these?

    a.b.c
    one.2.3
    1.pre.2.3.4
    2004d
    1.0beta5prerelease
    0.1-charmander
    0.1-bulbasaur

Those last two are actual legacy Python versions[1] -- how should we compare between those two for the purposes of dependency resolution?

Whether or not it's preferable to attempt to parse unspecified inputs boils down to questions of predictability, reliability, and security and, IMO, security is an overriding factor when it comes to domains as critical as packaging.

That being said: you can attempt to parse legacy versions using this package[2].

[1]: https://pypi.org/project/paramiko/#history

[2]: https://pypi.org/project/packaging-legacy/

When it comes to dependency resolution, that’s where it’s up to the application how to handle NonconformingVersion objects. For vanilla pip, it makes sense for it to treat these as errors.

I actually think it makes sense for the PyPI itself to be the one to dictate how to handle version policy via some callable or handler. Many companies have internal versions and mirrors of OSS code, and allowing them to preferentially shim these in (and reasonably version) these for their employees. Let’s say you fork the requests library. So now your versioning is requests=2.31.0.corporate-31. Sure, it makes no sense externally, but as the internal PyPI authority, you can enforce this versioning. Then your users just point to your index and get what they need (even if they just ask for requests=2.31.0).

In general, parsing and dependency resolution are two separate issues. The concerns of dependency resolution should not bleed into parsing. So it should be a valid use case to simply want to parse and compare two versions, and if it seems we have complete enough information, then make a truth estimation. E.g. in the above example, this should realistically not fail, but instead throw a NonconformingVersion warning:

> assert parse(‘2.4.0.coudera2’) >= parse(‘2.4’)

While we can’t parse the full string, we can make a really good, consistent guess at what the user is after. And if needed, parse could reasonably take an optional callable to support handling of nonconforming versions to convert them to conforming ones.

Other libraries are available: distlib is even part of PyPA, but apparently its approach to trying out new things put some people off, so it's not "blessed". It supports legacy, standard and semver versions in its version module [0].

    >>> from distlib.version import LegacyVersion, NormalizedVersion
    >>> NormalizedVersion('2.4.0cloudera2')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/disk2/vinay/projects/scratch/distlib/distlib/version.py", line 33, in __init__
        self._parts = parts = self.parse(s)
                              ^^^^^^^^^^^^^
      File "/disk2/vinay/projects/scratch/distlib/distlib/version.py", line 276, in parse
        result = _normalized_key(s)
                 ^^^^^^^^^^^^^^^^^^
      File "/disk2/vinay/projects/scratch/distlib/distlib/version.py", line 188, in _pep_440_key
        raise UnsupportedVersionError('Not a valid version: %s' % s)
    distlib.version.UnsupportedVersionError: Not a valid version: 2.4.0cloudera2
    >>> LegacyVersion('2.4.0cloudera2')
    LegacyVersion('2.4.0cloudera2')

[0] https://distlib.readthedocs.io/en/latest/reference.html#the-...

Reply on news.ycombinator.com