Hacker News Reader

Comment by macksd | Hacker News Reader

This is definitely the most reasonable case I've heard made from this argument, and it's changed my stance on the issue.

That said, I feel the cryptography project handled this poorly. I encountered the issue when one day, installing a Python Ansible module the same way I did the day before required a compiler for a different language that I've never used before and that is hardly used, if at all, in the rest of my stack. The Ansible project seemed to be taken by surprise, and eventually I found the root upstream issue, where the initial attitude was "too bad". Some people making those comments worked for Red Hat / IBM, Ansible is heavily backed by that. What company cares more about Linux on S390 than Red Hat / IBM? I would suggest none. So the fact that they had this attitude and community reaction to me suggests the problem is not one of expecting free work for a corporation on a non-free architecture. It was IMHO a combination of a lack of forethought, communication, and yes, perhaps a change that is overdue and will just be a little painful. The suggestion to maintain an LTS release while people figure out how to proceed is the right move.

View on HN

> Some people making those comments worked for Red Hat / IBM, Ansible is heavily backed by that.

Okay, so, I don't mean to pick on you here, but I've seen this sentiment cropping up a few times. Not everyone can be familiar with everything, but I would caution you against immediately assuming corporate politics are at the root here.

Anyone familiar with Alex Gaynor and his work over the last few years would know that he cares about Python, and memory safety. That has remained consistent regardless of his employer. Immediately assuming that this has something to do with company politics, rather than just a tireless open source contributor working to improve the things that he cares about, for years, is making a bit of a category error, in my opinion.

I agree - I don't feel picked on :D I guess my point is that it's overly dismissive to say "no free work for companies on niche architectures" when projects backed by the same company were a bit blind-sided by this and a bunch of people lost time chasing down what happened and why. To me that's a sign that this isn't just a mismatch of ideology or someone wanting free work: it was a failure in communication and mismatched expectations. If it had just been on a major version bump, I wonder if we'd even be having this discussion.

Glad to hear it :)

I think it gets really hard the larger the company you talk about. Before I worked at bigger places, I assumed a lot more coherence than is actually the case.

Sure, but the message - that if it's a problem for Red Hat's Ansible team or Red Hat's mainframe Linux team, they should be doing the work needed to make it not a problem.

Ansible Tower subscriptions aren't exactly cheap, and neither is RHEL s390x. If there's not the fat to be uplifting the core infrastructure needed to run RHEL or Ansible on Red Hat's products, that's most likely a choice.

I think corporate politics are at the root here, even if they're not Alex Gaynor's corporate politics. This blog post amounts to blessing amd64 and aarch64 as the only sustainable instruction set architectures.

Rust is currently the apex of a tall stack of hundreds of millions of lines of code, once you account for LLVM etc. Using it as the basis for other software means only processors with sufficient market penetration are 'worthy' candidates. In the long run, if Rust is as successful as lots of us hope it will be, this will kill what innovation is left in the hardware space. If a company is sufficiently motivated to care, it will almost certainly be cheaper to fork the code back to using C than it would be to forklift LLVM to a new architecture.

Is anyone aware of a direct Rust compiler project? Even discounting GCC, Go was bootstrapped from relatively simple (and naive) C compilers until it became self-hosting. I think a basic non-optimizing Rust compiler would go a long way toward leaving the door open for onboarding older -- and more importantly, novel -- architectures into the ecosystem.

I didn’t compare HEADs because I’m on my phone, but found reports that in 2019, LLVM was 7 million LOC, and gcc was 15. Both are C++ compiler projects. Why the double standard?

I also don’t know why re-writing an entire project and creating a compiler for it is somehow easier than writing an LLVM backend.

What double standard? I wouldn't recommend a language tie itself to GCC any more than I would recommend LLVM. The point is that the Rust compiler catalog is unhealthily small, and that introduces problems like this one.

To your other point, writing an LLVM backend is one thing. Getting it upstreamed is another, and maintaining it is another still. Then you have to navigate the politics of two foundations, both of whose boards of directors are basically the Who's Who of competing interests. I've watched more than one project fail to navigate those waters.

Anyway, the cryptography package going from python + c to python + C + C++ + Rust is a cambrian explosion of build time complexity, and in my work we found it simpler to just get rid of the python and the cryptography package, so it's mostly academic to me.

> I wouldn't recommend a language tie itself to GCC

Okay! I misunderstood you then, sorry. I think one of the hardest parts about this conversation is that there are so many different people with various, but overlapping, opinions. A lot of folks do think this way, and I thought that's what you were saying. My bad.

> Getting it upstreamed is another

You do not need it to be upstreamed in order to build Rust, we build with our own modified LLVM by default, so using it is quite easy.

The question has probably been asked a thousand times before, but wouldn't all those bootstrapping problems for niche platforms be solved if Rust had a C backend? Are there technical issues which would prevent compiling LLVM bytecode to a C blob, similar to what wasm2c does for WASM bytecode?

Having a C backend does not solve the hard issues. Because of undefined behavior in the C specification, sometimes there just is no way to write down a particular expression in a portable manner in C without treading through undefined territory. This may not be as big of a deal for boring application code, but we're talking about cryptographic code here, which needs to work hard to avoid memory corruption, integer overflows, timing side channels, etc.

I'd imagine the generated code could be hardened similar to the code generated by clang ASAN, UBSAN and TSAN, and also wouldn't generate code that depends on undefined behaviour in the first place. Or you could do a little detour through WASM:

https://kripken.github.io/talks/2020/universal.html#/

In any case, that's better than shrugging off esoteric platforms, IMHO.

I think, like majewsky says, it is a bit more complicated than it may initially seem. However, even if we assume that it is trivial, there's other problems. Sure, maybe it would. But who is going to do that work? We're an open source project. Effort is not fungible. On some level, we can only get stuff done when there's a sufficient need for it, and while there have been some folks talking about this in the last week or so, historically, it just hasn't been a massive issue. If it is a massive problem for someone, they should solve it! The Rust project's stance has been open to new platforms, and will continue to be so. But we need experts in those areas to help us help themselves.

Maybe this alternative Rust compiler?

https://github.com/thepowersgang/mrustc

The switch hasn't been sudden. It's just that many levels of disconnect between the project authors and downstream users meant it was basically impossible to communicate to all the affected users — nobody looks at their deps-of-deps until they break.

And they've released Rust as optional component you can disable, precisely because nobody paid attention until it actually shipped.

> The switch hasn't been sudden. It's just that many levels of disconnect between the project authors and downstream users meant it was basically impossible to communicate to all the affected users — nobody looks at their deps-of-deps until they break.

Exactly, from the original shitstorm issue:

> Rust bindings kicked off in July 2020 with #5357. Alex started a thread on the cryptography developer mailing list in December to get feedback from packagers and users. The FAQ also contains instrutions how you can disable Rust bindings.

> Do you have constructive suggestions how to communicate changes additionally to project mailing lists, Github issue tracker, project IRC channel, documentation (changelog, FAQ), and Twitter channels?

At one point there's sadly not much the project can do and still make progress.

> > Do you have constructive suggestions how to communicate changes

That one kinda got me, when python intentionally has a runtime developer-to-developer communication system, per say:

https://docs.python.org/3/library/warnings.html

Warnings are a funny thing, lots of projects like to turn all warnings into failing errors in CI, sometimes even at package build time, they think it's a best-practice ... but it means that nobody else can use warnings to communicate things, or else everything breaks, nullifying the utility of warnings.

The easy answer to this is "warnings should be warnings" but the hard question is "how do we get people to stop treating them as errors?"

The easy answer is to stop treating errors as warnings! I’m working on a project where for the first time in my career I’ve made my environment/workflow/tooling less shouty about problems.

I’m writing for my blog, so I installed a spell check extension. But its dictionary stinks. So I just turn off its warnings before a final post pass.

Most of the time when I see yellow in my editor it’s stuff I would expect to be red. Even from my primary language (TypeScript) which officially doesn’t even have warnings.

More things should be errors and treated as such! And more things that legitimately qualify as warnings should error in final checks to ensure they were addressed somehow, even with just an explicit dismissal.

"Sudden" is relative to the rate of propagation. Maybe we can say that the difficulty of communicating to all stakeholders of a package is egregious, and even endemic to the ecosystem as a whole, but it still sounds like a better communication effort could have been made, even if doing it perfectly is impossible

There was advocacy and pressure to adopt pyca/cryptography in other popular python packages. Don't roll your own crypto, don't use old unmaintained crypto libraries (pycrypto ... but these days there's pycryptodome) etc. If pyca/cryptography was instead advertised as "python crypto library for rust lovers" and other python developers knew that a _gigantic_ and _obnoxious_ rust toolchain dependency was involved, I think many would have avoided cryptography, or made it optional. This was a bait-and-switch.

This is obnoxious for people who want to start with a tiny standard distro image with just a C toolchain, and build their app and all dependencies from source. It is also obnoxious for distros that want to build all packages from source. rust depends on llvm, itself a huge complicated finicky package ... but not just that, it depends on a custom-patched llvm! ...but not just that, it also depends on an extremely recent version of rust! this problem is recursive! And there are many other niche cases ... and "niche cases" of "unauthorized" porting/modification/substitution is how much of the open source ecosystem we love got its start.

This is really about issues of paranoia and control in the "security" realm. They consider it irresponsible to enable users to do anything they can't guarantee "safe". "Your organization should be paying for a $$$ corporate support contract if you want to do anything we don't officialy support." Maybe you could have fixed a portability bug in some python or C code (I've fixed a couple when bringing up a little-endian mips64 system a decade ago), but no you'll need rust+llvm experts to port that whole crazy toolchain. It's for your own good.

Here you seem to be demanding that the authors and maintainers of their own cryptography library lighten up about security, because you find the way they manage their own package to be controlling. You see some irony in that, right?

I don't read /u/ploxiln as demanding anything. They seem frustrated with the changing dependency profile of the dependency, and a little frustrated with the way some people like to shout "security" as if it's a trump card, ignoring that security is a spectrum and that the only perfectly secure software does nothing (perfectly).

This is a security library. In fact, it's a security library that was created specifically to harden and user-proof less secure alternatives. If you don't care as much as they do about security, use a different library (or just keep using the pre-Rust version of this one).

> If pyca/cryptography was instead advertised as "python crypto library for rust lovers"

I feel like this really only makes sense if pyca/cryptography had planned on adding the Rust dependency from the very beginning (or from very early on). Is there any indication that was the case?

> but not just that, it depends on a custom-patched llvm!

This doesn't seem to be true [0, 1].

[0]: https://rustc-dev-guide.rust-lang.org/backend/updating-llvm....

[1]: https://news.ycombinator.com/item?id=26217182

>> Strongly prefer to upstream all patches to LLVM before including them in rustc.

> That is, this is already the case. We don't like maintaining a fork. We try to upstream as much as we can.

> But, at the same time, even when you do this, it takes tons of time. A contributor was talking about exactly this on Twitter earlier today, and estimated that, even if the patch was written and accepted as fast as possible, it would still take roughly a year for that patch to make it into the release used by Rust. This is the opposite side of the whole "move slow and only use old versions of things" tradeoff: that would take even longer to get into, say, the LLVM in Debian stable, as suggested in another comment chain.

> So our approach is, upstream the patches, keep them in our fork, and then remove them when they inevitably make it back downstream.

... so there are always some outstanding patches rust applies to the llvm codebase.

For Rust's LLVM fork, sure. But as Steve Klabnik noted in the first comment I linked, unmodified LLVM is supported.

In addition, later down in the comment chain:

cycloptic:

> Can't there be a build option to not use the LLVM submodule, and instead use the system LLVM?

steveklabnik:

> There is. We even test that it builds as part of our CI, to make sure it works, IIRC.

For a more concrete example, Fedora supports choosing between system and bundled LLVM when building Rust [0, 1].

[0]: https://news.ycombinator.com/item?id=26222190

[1]: https://src.fedoraproject.org/rpms/rust//blob/rawhide/f/rust...

> I feel like this really only makes sense if pyca/cryptography had planned on adding the Rust dependency from the very beginning (or from very early on). Is there any indication that was the case?

I am sure this idea surfaced several times in IRC or possibly in the mailing lists. Certainly, the authors have been toying with handling ASN.1 in rust since 2015 [1], which I guess will be the next logical step.

I do agree that this is mostly a political stance. pyca/cryptography is a wrapper sandwiched between a gigantic runtime written in C (CPython/PyPy) and a gigantic library written in C (openssl).

The addition of Rust as dependency enables the inclusion of just 90 lines of Rust [2] where the only part that really couldn't be implemented in pure Python is a line copied from OpenSSL [3] (i.e. it was already available), and which is purely algebraic, therefore not mitigating any real memory issue at all (the reason to use rust in the first place).

The change in this wrapper (pyca/cryptographic) does not move the needle of security in any significant way, and it is really only meant to send the signal that adding Rust in all other Python packages and especially in the runtime itself will now come at no (political) cost.

[1] https://github.com/alex/rust-asn1

[2] https://github.com/pyca/cryptography/blob/main/src/rust/src/...

[3] https://github.com/openssl/openssl/blob/OpenSSL_1_1_1i/inclu...

> The addition of Rust as dependency enables the inclusion of just 90 lines of Rust

My understanding is that this is just the beginning, and the whole reason it's only a small amount is precisely to do it in small steps, correctly, rather than re-writing the entire world in one go.

And why should they re-write the entire world? It's just a wrapper library to openssl and it was marketed heavily to the community (at the beginning) as one where maintainers would follow good practices and not try to write security-sensitive code as they are not security experts, but just rely on openssl so that all focus go into the same place.

So either that's still valid (and not too much of rust will come in to make a real difference security-wise) or they have revisited their position and will re-implement a bunch of openssl logic (in rust apparently though, and not in Python as it would be more logical, and as golang does successfully). And in the latter case, why just not focus on wrapping rustls instead?

In either case, the hand is being forced: it's a small amount, and I don't see how it would have been an excessive burden to maintain such a small logic as an opt in for a period. It makes much more sense to read this as a move to forcefully push rust into the python ecosystem.

> So either that's still valid (and not too much of rust will come in to make a real difference security-wise) or they have revisited their position and will re-implement a bunch of openssl logic (in rust apparently though

Looks like it's the former. From the initial (?) GitHub issue discussing the move to Rust [0]:

> We do not plan to move any of the crypto under the scope of FIPS-140 from OpenSSL to cryptography. We do expect to move a) our own code that's written in C (e.g. unpadding), b) ASN.1 parsing. Neither of those are in scope for FIPS-140.

> and not in Python as it would be more logical

Is Python actually suitable for cryptographic code, especially if constant-time operations are needed?

> I don't see how it would have been an excessive burden to maintain such a small logic as an opt in for a period.

And then when said logic stops being opt-in, why wouldn't the same problem arise?

[0]: https://github.com/pyca/cryptography/issues/5381#issuecommen...

> Is Python actually suitable for cryptographic code, especially if constant-time operations are needed?

No, but as seen in the first chunk of rust code added, the amount of logic that needs to be constant-time is a) very, very limited to a few primitives only b) algebraic in nature (put differently, it's not where memory bugs will pop up, so using rust over C doesn't even buy you much).

For instance, ASN.1 parsing doesn't need to done in constant time in the vast majority of cases (and in all cases I am aware of).

> And then when said logic stops being opt-in, why wouldn't the same problem arise?

Some friction will certainly remains, but it will be nothing compared to the current breakage.

> No, but as seen in the first chunk of rust code added, the amount of logic that needs to be constant-time is a) very, very limited to a few primitives only b) algebraic in nature (put differently, it's not where memory bugs will pop up, so using rust over C doesn't even buy you much).

That's fair, though if some other part of the codebase is ported to Rust anyways sticking to C doesn't save you, unfortunately.

> For instance, ASN.1 parsing doesn't need to done in constant time in the vast majority of cases (and in all cases I am aware of).

I'm curious why this has to be done in C/Rust. Performance?

> Some friction will certainly remains, but it will be nothing compared to the current breakage.

What would be different then compared to now that would reduce breakage to such an extent?

> That's fair, though if some other part of the codebase is ported to Rust anyways sticking to C doesn't save you, unfortunately.

This should be turned around. There is no evidence that part for the codebase really need to be ported and for which Rust makes a real difference.

> I'm curious why this has to be done in C/Rust. Performance?

ASN.1 is only used during handshakes and I/O of files (the chunk of rust added covers loading of PKCS#7 files which are typically quite small, and not typically dealt with in massive numbers). I doubt the performance hit would be so high. Also, the package wbond/asn1crypto has shown that doing ASN.1 in pure Python can be quite fast.

> What would be different then compared to now that would reduce breakage to such an extent?

The ability to try the feature out and still have a plan B if it doesn't work out, and the possibility to have a smooth, long, relaxed upgrade plan with good warnings and more time to prepare. But to be honest, while writing all this chain of comments, I really doubt there was a real need to add rust into the mix (other than the political angle I already mentioned).

> This should be turned around. There is no evidence that part for the codebase really need to be ported and for which Rust makes a real difference.

I was thinking that if the maintainers were planning on porting some other larger part of the codebase, then starting with something small/relatively inconsequential would be a good first step, and keeping it in C wouldn't provide much benefit once said other larger part were ported.

> Also, the package wbond/asn1crypto has shown that doing ASN.1 in pure Python can be quite fast.

Interesting! I'm curious why the maintainers didn't opt for that approach instead.

> The ability to try the feature out and still have a plan B if it doesn't work out

Would people have tried this feature out if it were made opt-in? It's clear that the initial announcements reached far fewer people than one might like, and I honestly have no idea how many people would see a build-time warning.

What you're saying makes sense in any other domain except cryptographical primitives. With those, the basic correctness of the code will depend on details of the compiler and processor architecture, and it is extremely likely that the code will be fundamentally incorrect when used on other architectures or compilers.

And this is not just a matter of fixing some little-endian assumption, it's a matter of understanding microoptimizations, instruction reordering, prefetch behavior, branch prediction etc. Ensuring code has predictable runtime in the presence of an optimizing compiler and out of order processor is extremely difficult.

It's probably just as smart to roll your own crypto instead of trying to use a known library on a new platform, because the hardest parts of rolling your own will have to be done anyway.

> rust depends on llvm, itself a huge complicated finicky package

gcc is also a "huge complicated finicky package"... Any decently sized compiler is.

> installing a Python Ansible module

It's funny that you mentioned having to install Rust to use some Python because, for us non-Python users, everything needs Python :). I don't use Python at all, but I need both versions installed.

Just pointing out that many people are already wearing that shoe.

Python is the Lingua Franca of getting shit done. Rust is seemingly becoming the Lingua Franca of [more] secure code.

Red Hat doesn't care at all about Linux on 32-bit s390, and neither does IBM as far as I know (except possibly the consulting group, which is interested in anything that makes them money).

s390x has no problem with switching to Rust/LLVM. Red Hat and IBM both employ engineers working on LLVM, specifically for s390x in IBM's case.

There's a phenomenon in lots of domains where "a change that is itself neutral or good can be bad if it happens too suddenly". I think that was the case here; we can say that this is a shift that reasonably can or should happen, but can still have caused needless disruption by catching a bunch of people off-guard and not giving them time to adapt to it

Why is it necessarily Cryptography’s fault that Python Ansible was taken by surprise? Or any of the other affected parties along the chain? That Cryptography was starting to include Rust in the project was announced on the mailing list by Gaynor last summer. And that email said exactly what future (at that time) release would require a Rust toolchain if the project was to be built from source.

The reply to that by the Gentoo guy (who started the Github issue) was that package maintainers cannot follow every single mailing list of every dependency. That is debatable (e.g. maybe you should make an exception for security applications), but let’s take that as a given for now. In that case, what is Cryptography to do? Where should they announce such things in a way that orgs like Gentoo will see it? And also notice it and not just mentally gloss over it as some kind of “spam”? If the Gentoo guy didn’t see it half a year ago, would he have seen an announcement (or the reminders) if it was made five years ago?

Folks with an eye towards backwards compatibility typically don't implement breaking changes without a year of deprecation warnings emitted by the software. Contrast that to a notice posted in a sub-basement 6 months before instituting a breaking change. The shock and alarm do seem warranted IMO.

> project mailing lists, Github issue tracker, project IRC channel, documentation (changelog, FAQ), and Twitter channels

It's hardly a notice posted in a sub-basement 6 months before instituting a breaking change. They communicated via numerous mechanisms.

It's a sub-basement from the perspective of folks for whom this is a second-order dependency. And it sounds like there are many more of them than there are folks that caught wind of this.

Regardless. The common practice is a year of emitting warnings from the software, that the end-user will see. That's prominent and allows package maintainers enough time to work around the upcoming breakage. Six months notice on a mailing list is simply not prominent enough, and it's half the standard year which puts undue strain downstream.

How do you do a deprecation warning for a new language being a build dependency? What does it look like? "your build environment is being deprecated"?

In C you can implement a "your build environment is being deprecated" message like this:

    #ifndef I_UNDERSTAND_THAT_SOON_RUST_WILL_BE_MANDATORY
    #error "Starting with version xxx this package will need Rust to compile. Recompile with -DI_UNDERSTAND_THAT_SOON_RUST_WILL_BE_MANDATORY to acknowledge that you understand this warning."
    #endif

Anyone building from source would get notified in a way that's impossible to miss but easy to turn off.

True, that's possible, but quite brutal and I'd expect that would merely have lead to shouting and people going wild over their builds breaking a year earlier.

And thousands upon thousands of automated CI jobs and docker container builds fail. You're basically causing massive developer stress to anyone who automatically compiles your package. Most would consider that a bad tradeoff to help a tiny fraction of your users who'd be impacted and also refuse to follow your mailing list.

You're going to do that a few versions down the line anyway. Why not ahead of time?

The only downstreams this affects more are the ones who decide to suppress the warning for now and ... put it off until you release the change that actually required breakage.

> And thousands upon thousands of automated CI jobs and docker container builds fail. You're basically causing massive developer stress to anyone who automatically compiles your package.

This isn’t a major source of stress: you include migration instructions and even tooling to automate it. The thing that’s stressful is when your build fails with completely unexpected errors and no indication of what went wrong.

Loudly announcing breaking changes is disruptive to some extent, but not doing so either means more disruption or nothing can ever change at all.

Isn't this basically equivalent to what they did? Their actual action was to include a dummy rust requirement just to break/warn people who wouldn't be prepared for it as an actual dependency.

These developers cause themselves stress by building against the latest library without a care in the world. Lock your dependencies, regularly upgrade them while looking at patchnotes, and it won't be a problem.

CI builds should be locked to exact versions anyways, for the sake of reproducibility.

If the change is transparent to all but a tiny fraction, you can of course guard the above with

    #if !(defined(__AMD64__) || defined(__AARCH64__))

(or whatever the accepted names of those platform macros are).

For whatever it's worth, if the C code in the cryptography package didn't already contain the above check, along with a some hurdle requiring the user to compile with -DNOT_OFFICIALLY_SUPPORTED_TARGET, then the article's point is false: If you don't prevent users from compiling your security-relevant, reliant-on-exact-memory-semantics software on System/390, then you are implicitly supporting System/390.

And for that matter, the Rust code should probably contain a similar check: Even if someone ported Rust to System/390, the cryptography library shouldn't magically start working there, unless the developers actually test there.

This is for all intents and purposes exactly what the cryptography maintainers did, except they did one better: they made sure this release is capable of validating the new configuration. They made rust an optional, but on-by-default, dependency. That is the equivalent of #error plus validation. They made it possible to turn this dependency off. That is the equivalent of -DI_UNDERSTAND.

Yes. It's pretty simple to insert a stub module which is imported in a try/catch, where failure to load results in a warning.

Hm yeah, that sounds like a bit of a hassle to set up (have it build where possible but not fail the build), but otherwise an elegant solution, since it would pass cleanly for people that have a suitable environment.

I think making a change like this warrants a major version bump. It won't eliminate all the surprise for everyone, and I do have sympathy for the people who did go out of their way to talk about this on the mailing list and then surprised everyone. But it's common to pin yourself to a minor or maintenance release line to automatically pick up security fixes, etc. I expect breakage when changing the major (or even minor), and that's almost always a manual upgrade. And that's when I do read all the release notes, run tests, etc. before committing to the change.

What is a “major version bump”? Before you answer, consider that the library doesn’t use semantic versioning. Before this all blew up, the versioning scheme was this[1]:

> Given a version cryptography X.Y.Z,

> - X.Y is a decimal number that is incremented for potentially-backwards-incompatible releases.

> - - This increases like a standard decimal. In other words, 0.9 is the ninth release, and 1.0 is the tenth (not 0.10). The dividing decimal point can effectively be ignored.

> - Z is an integer that is incremented for backward-compatible releases.

The system has since changed, but it continues not to be semantic versioning. (It’s effectively the same, in fact, but protects against dependents who think it is semantic.)

By that scheme, it was already a “major” (signifying potential backwards-incompatibility) release.

[1]: https://cryptography.io/en/latest/api-stability.html#previou...

This reads to me like an argument for semantic versioning, because otherwise I need to internalize the rules of every package and know that some will break compatibility on the Y, some on the Z, ... Etc.

You know, I had that thought while writing it.

I wouldn’t want to force any particular versioning scheme on any particular developer, but maybe the “SemVer façade” versioning scheme they switched to is the best compromise. It has defensive value, at least.

Then again, PEP 440 has nothing to say about the semantics of versioning, only requiring:

  [N!]N(.N)*[{a|b|rc}N][.postN][.devN]

PyPA themselves describe various expected versioning schemes, but listing Semantic as preferred[1]. If I squint, I can fit `cryptography`’s previous scheme into “Hybrid”. The biggest lesson I take from this is that if your version scheme isn’t SemVer, work hard to make it look obviously different from SemVer.

[1]: https://packaging.python.org/guides/distributing-packages-us...

Would SemVer even strictly require a jump here? They didn't change what's usually thought of as the API of the library (i.e. if I don't compile it myself I don't really notice the change?), and that's what SemVer uses: "MAJOR version when they make incompatible API changes,"

Also a great point!

I can’t think of a clean alternative other than coming full-circle back to the “developer advocacy” solution, with its clear problems. Someone smarter than I am probably has it in the palm of their hand, though.

Yes, see https://news.ycombinator.com/item?id=26298805

On the other hand, from what I understand they do ship precompiled wheels for many platforms, just missed one lots of people use in their CI setups (Alpine, which uses musl and thus isn't compatible with other Linux wheels - personally I think that's an odd choice but whatever, people do it)? Easy to imagine that many more people compiled it than they expected.

Even SemVer doesn't help here, as changing the build toolchain isn't generally considered an API breakage (after all, the resulting binaries are API compatible)

Changing the build toolchain/requirements in such a significant way does seem like a major version break, as it could break downstream consumers attempting to install the package.

Because the build toolchain is "visible", that is, pip isn't just downloading a prebuilt binary every time, I think breaking changes that could cause CI systems or user installs to fail is part of the API contract. Think of what major distributions or software packages do when they want to deprecate support for certain platforms - those are major bumps that typically only occur on incrementing the most significant component of the version.

Hypothetical: suppose the the authors changed setup.py so that it only built on Red Hat Enterprise Linux(tm) version 6. Again, they could do that, it wouldn't change the runtime API. And on all other distributions or installers, it would error.

Would that be a major semver change? Of course it would be. The API contract has to include everything from packaging to use.

> The reply to that by the Gentoo guy (who started the Github issue) was that package maintainers cannot follow every single mailing list of every dependency.

It seems reasonable that, if you are the packager for critical packages, that you follow critical dependencies?

If the problem is that the distro is supporting so many things that the folks working on it can't keep up - well, that's precisely the author's point: stop pretending that you can support HPPA and MIPS or whatever as well as you can support x86_64. But you don't get to tell a million people that they have to have a less secure Python because 3 people have a toy in a closet they want treated as a first class citizen.

The number of people in the original thread who didn’t appear to be version pinning and then getting upset that a package that they directly relied upon automatically upgraded is eye-watering.

> stop pretending that you can support HPPA and MIPS or whatever as well as you can support x86_64

And then the corresponding uptightness will be “FOO_PROJECT is aligned with the Intel monopoly”, and just as many people will be unhappy. You can see this in many recent threads about Apple not providing free access to M1 documentation for alternative OSes they’re under no obligation to support.

That was, of course, a complaint about Linux back in the day. It turned out nobody cared enough to stop Linux development.

That’s pretty much verbatim the reply I had when people on here were prognosticating it for the M1, other than adding that they also promoted Linux virtualization in the announcement.

Alright, let's do some digging...

On 2013-03-21, urllib3 added an optional dependency to pyopenssl for SNI support on python2 - https://github.com/urllib3/urllib3/pull/156

On 2013-12-29, pyopenssl switched from opentls to cryptography - https://github.com/pyca/pyopenssl/commit/6037d073

On 2016-07-19, urllib3 started to depend on a new pyopenssl version that requires cryptography - https://github.com/urllib3/urllib3/commit/c5f393ae3

On 2016-11-15, requests started to depend on a new urllib3 version that now indirectly requires cryptography - https://github.com/psf/requests/commit/99fa7bec

On 2018-01-30, portage started to enable the +rsync-verify USE flag by default, which relies on the gemato python library maintained by mgorny himself, and gemato depended on requests. So 5-6 levels of indirection at this point? I lost count.

On 2020-01-01, python2 was sunset. A painful year to remember, and a painful migration to forget. And just when the year was about to end...

On 2020-12-22, cryptography started to integrate rust in the build process, and all hell broke loose - https://github.com/pyca/cryptography/commit/c84d6ee0

Ultimately, I think mgorny only has himself to blame here, by injecting his own library into the critical path of gentoo, without carefully taking care of its direct and indirect dependencies. (But of course it is also fair game to blame it on the 2to3 migration)

In comparison, few months before this, the librsvg package went through a similar change where it started to depend on rust, and it was swift and painless without much drama - https://bugs.gentoo.org/739820 and https://wiki.gentoo.org/wiki/Project:GNOME/3.36-notes

Reply on news.ycombinator.com