Comment by laserharvest | Hacker News Reader

Comment by laserharvest | Hacker News Reader

Parent Full thread

Why is it necessarily Cryptography’s fault that Python Ansible was taken by surprise? Or any of the other affected parties along the chain? That Cryptography was starting to include Rust in the project was announced on the mailing list by Gaynor last summer. And that email said exactly what future (at that time) release would require a Rust toolchain if the project was to be built from source.

The reply to that by the Gentoo guy (who started the Github issue) was that package maintainers cannot follow every single mailing list of every dependency. That is debatable (e.g. maybe you should make an exception for security applications), but let’s take that as a given for now. In that case, what is Cryptography to do? Where should they announce such things in a way that orgs like Gentoo will see it? And also notice it and not just mentally gloss over it as some kind of “spam”? If the Gentoo guy didn’t see it half a year ago, would he have seen an announcement (or the reminders) if it was made five years ago?

I think making a change like this warrants a major version bump. It won't eliminate all the surprise for everyone, and I do have sympathy for the people who did go out of their way to talk about this on the mailing list and then surprised everyone. But it's common to pin yourself to a minor or maintenance release line to automatically pick up security fixes, etc. I expect breakage when changing the major (or even minor), and that's almost always a manual upgrade. And that's when I do read all the release notes, run tests, etc. before committing to the change.

What is a “major version bump”? Before you answer, consider that the library doesn’t use semantic versioning. Before this all blew up, the versioning scheme was this[1]:

> Given a version cryptography X.Y.Z,

> - X.Y is a decimal number that is incremented for potentially-backwards-incompatible releases.

> - - This increases like a standard decimal. In other words, 0.9 is the ninth release, and 1.0 is the tenth (not 0.10). The dividing decimal point can effectively be ignored.

> - Z is an integer that is incremented for backward-compatible releases.

The system has since changed, but it continues not to be semantic versioning. (It’s effectively the same, in fact, but protects against dependents who think it is semantic.)

By that scheme, it was already a “major” (signifying potential backwards-incompatibility) release.

[1]: https://cryptography.io/en/latest/api-stability.html#previou...

This reads to me like an argument for semantic versioning, because otherwise I need to internalize the rules of every package and know that some will break compatibility on the Y, some on the Z, ... Etc.

You know, I had that thought while writing it.

I wouldn’t want to force any particular versioning scheme on any particular developer, but maybe the “SemVer façade” versioning scheme they switched to is the best compromise. It has defensive value, at least.

Then again, PEP 440 has nothing to say about the semantics of versioning, only requiring:

  [N!]N(.N)*[{a|b|rc}N][.postN][.devN]

PyPA themselves describe various expected versioning schemes, but listing Semantic as preferred[1]. If I squint, I can fit `cryptography`’s previous scheme into “Hybrid”. The biggest lesson I take from this is that if your version scheme isn’t SemVer, work hard to make it look obviously different from SemVer.

[1]: https://packaging.python.org/guides/distributing-packages-us...

Would SemVer even strictly require a jump here? They didn't change what's usually thought of as the API of the library (i.e. if I don't compile it myself I don't really notice the change?), and that's what SemVer uses: "MAJOR version when they make incompatible API changes,"

Also a great point!

I can’t think of a clean alternative other than coming full-circle back to the “developer advocacy” solution, with its clear problems. Someone smarter than I am probably has it in the palm of their hand, though.

Yes, see https://news.ycombinator.com/item?id=26298805

On the other hand, from what I understand they do ship precompiled wheels for many platforms, just missed one lots of people use in their CI setups (Alpine, which uses musl and thus isn't compatible with other Linux wheels - personally I think that's an odd choice but whatever, people do it)? Easy to imagine that many more people compiled it than they expected.

Even SemVer doesn't help here, as changing the build toolchain isn't generally considered an API breakage (after all, the resulting binaries are API compatible)

Changing the build toolchain/requirements in such a significant way does seem like a major version break, as it could break downstream consumers attempting to install the package.

Because the build toolchain is "visible", that is, pip isn't just downloading a prebuilt binary every time, I think breaking changes that could cause CI systems or user installs to fail is part of the API contract. Think of what major distributions or software packages do when they want to deprecate support for certain platforms - those are major bumps that typically only occur on incrementing the most significant component of the version.

Hypothetical: suppose the the authors changed setup.py so that it only built on Red Hat Enterprise Linux(tm) version 6. Again, they could do that, it wouldn't change the runtime API. And on all other distributions or installers, it would error.

Would that be a major semver change? Of course it would be. The API contract has to include everything from packaging to use.

> The reply to that by the Gentoo guy (who started the Github issue) was that package maintainers cannot follow every single mailing list of every dependency.

It seems reasonable that, if you are the packager for critical packages, that you follow critical dependencies?

If the problem is that the distro is supporting so many things that the folks working on it can't keep up - well, that's precisely the author's point: stop pretending that you can support HPPA and MIPS or whatever as well as you can support x86_64. But you don't get to tell a million people that they have to have a less secure Python because 3 people have a toy in a closet they want treated as a first class citizen.

The number of people in the original thread who didn’t appear to be version pinning and then getting upset that a package that they directly relied upon automatically upgraded is eye-watering.

> stop pretending that you can support HPPA and MIPS or whatever as well as you can support x86_64

And then the corresponding uptightness will be “FOO_PROJECT is aligned with the Intel monopoly”, and just as many people will be unhappy. You can see this in many recent threads about Apple not providing free access to M1 documentation for alternative OSes they’re under no obligation to support.

That was, of course, a complaint about Linux back in the day. It turned out nobody cared enough to stop Linux development.

That’s pretty much verbatim the reply I had when people on here were prognosticating it for the M1, other than adding that they also promoted Linux virtualization in the announcement.

Alright, let's do some digging...

On 2013-03-21, urllib3 added an optional dependency to pyopenssl for SNI support on python2 - https://github.com/urllib3/urllib3/pull/156

On 2013-12-29, pyopenssl switched from opentls to cryptography - https://github.com/pyca/pyopenssl/commit/6037d073

On 2016-07-19, urllib3 started to depend on a new pyopenssl version that requires cryptography - https://github.com/urllib3/urllib3/commit/c5f393ae3

On 2016-11-15, requests started to depend on a new urllib3 version that now indirectly requires cryptography - https://github.com/psf/requests/commit/99fa7bec

On 2018-01-30, portage started to enable the +rsync-verify USE flag by default, which relies on the gemato python library maintained by mgorny himself, and gemato depended on requests. So 5-6 levels of indirection at this point? I lost count.

On 2020-01-01, python2 was sunset. A painful year to remember, and a painful migration to forget. And just when the year was about to end...

On 2020-12-22, cryptography started to integrate rust in the build process, and all hell broke loose - https://github.com/pyca/cryptography/commit/c84d6ee0

Ultimately, I think mgorny only has himself to blame here, by injecting his own library into the critical path of gentoo, without carefully taking care of its direct and indirect dependencies. (But of course it is also fair game to blame it on the 2to3 migration)

In comparison, few months before this, the librsvg package went through a similar change where it started to depend on rust, and it was swift and painless without much drama - https://bugs.gentoo.org/739820 and https://wiki.gentoo.org/wiki/Project:GNOME/3.36-notes

Folks with an eye towards backwards compatibility typically don't implement breaking changes without a year of deprecation warnings emitted by the software. Contrast that to a notice posted in a sub-basement 6 months before instituting a breaking change. The shock and alarm do seem warranted IMO.

> project mailing lists, Github issue tracker, project IRC channel, documentation (changelog, FAQ), and Twitter channels

It's hardly a notice posted in a sub-basement 6 months before instituting a breaking change. They communicated via numerous mechanisms.

It's a sub-basement from the perspective of folks for whom this is a second-order dependency. And it sounds like there are many more of them than there are folks that caught wind of this.

Regardless. The common practice is a year of emitting warnings from the software, that the end-user will see. That's prominent and allows package maintainers enough time to work around the upcoming breakage. Six months notice on a mailing list is simply not prominent enough, and it's half the standard year which puts undue strain downstream.

How do you do a deprecation warning for a new language being a build dependency? What does it look like? "your build environment is being deprecated"?

In C you can implement a "your build environment is being deprecated" message like this:

    #ifndef I_UNDERSTAND_THAT_SOON_RUST_WILL_BE_MANDATORY
    #error "Starting with version xxx this package will need Rust to compile. Recompile with -DI_UNDERSTAND_THAT_SOON_RUST_WILL_BE_MANDATORY to acknowledge that you understand this warning."
    #endif

Anyone building from source would get notified in a way that's impossible to miss but easy to turn off.

True, that's possible, but quite brutal and I'd expect that would merely have lead to shouting and people going wild over their builds breaking a year earlier.

And thousands upon thousands of automated CI jobs and docker container builds fail. You're basically causing massive developer stress to anyone who automatically compiles your package. Most would consider that a bad tradeoff to help a tiny fraction of your users who'd be impacted and also refuse to follow your mailing list.

You're going to do that a few versions down the line anyway. Why not ahead of time?

The only downstreams this affects more are the ones who decide to suppress the warning for now and ... put it off until you release the change that actually required breakage.

> And thousands upon thousands of automated CI jobs and docker container builds fail. You're basically causing massive developer stress to anyone who automatically compiles your package.

This isn’t a major source of stress: you include migration instructions and even tooling to automate it. The thing that’s stressful is when your build fails with completely unexpected errors and no indication of what went wrong.

Loudly announcing breaking changes is disruptive to some extent, but not doing so either means more disruption or nothing can ever change at all.

Isn't this basically equivalent to what they did? Their actual action was to include a dummy rust requirement just to break/warn people who wouldn't be prepared for it as an actual dependency.

These developers cause themselves stress by building against the latest library without a care in the world. Lock your dependencies, regularly upgrade them while looking at patchnotes, and it won't be a problem.

CI builds should be locked to exact versions anyways, for the sake of reproducibility.

If the change is transparent to all but a tiny fraction, you can of course guard the above with

    #if !(defined(__AMD64__) || defined(__AARCH64__))

(or whatever the accepted names of those platform macros are).

For whatever it's worth, if the C code in the cryptography package didn't already contain the above check, along with a some hurdle requiring the user to compile with -DNOT_OFFICIALLY_SUPPORTED_TARGET, then the article's point is false: If you don't prevent users from compiling your security-relevant, reliant-on-exact-memory-semantics software on System/390, then you are implicitly supporting System/390.

And for that matter, the Rust code should probably contain a similar check: Even if someone ported Rust to System/390, the cryptography library shouldn't magically start working there, unless the developers actually test there.

This is for all intents and purposes exactly what the cryptography maintainers did, except they did one better: they made sure this release is capable of validating the new configuration. They made rust an optional, but on-by-default, dependency. That is the equivalent of #error plus validation. They made it possible to turn this dependency off. That is the equivalent of -DI_UNDERSTAND.

Yes. It's pretty simple to insert a stub module which is imported in a try/catch, where failure to load results in a warning.

Hm yeah, that sounds like a bit of a hassle to set up (have it build where possible but not fail the build), but otherwise an elegant solution, since it would pass cleanly for people that have a suitable environment.

Reply on news.ycombinator.com