Comment by ploxiln | Hacker News Reader

Comment by ploxiln | Hacker News Reader

Parent Full thread

There was advocacy and pressure to adopt pyca/cryptography in other popular python packages. Don't roll your own crypto, don't use old unmaintained crypto libraries (pycrypto ... but these days there's pycryptodome) etc. If pyca/cryptography was instead advertised as "python crypto library for rust lovers" and other python developers knew that a _gigantic_ and _obnoxious_ rust toolchain dependency was involved, I think many would have avoided cryptography, or made it optional. This was a bait-and-switch.

This is obnoxious for people who want to start with a tiny standard distro image with just a C toolchain, and build their app and all dependencies from source. It is also obnoxious for distros that want to build all packages from source. rust depends on llvm, itself a huge complicated finicky package ... but not just that, it depends on a custom-patched llvm! ...but not just that, it also depends on an extremely recent version of rust! this problem is recursive! And there are many other niche cases ... and "niche cases" of "unauthorized" porting/modification/substitution is how much of the open source ecosystem we love got its start.

This is really about issues of paranoia and control in the "security" realm. They consider it irresponsible to enable users to do anything they can't guarantee "safe". "Your organization should be paying for a $$$ corporate support contract if you want to do anything we don't officialy support." Maybe you could have fixed a portability bug in some python or C code (I've fixed a couple when bringing up a little-endian mips64 system a decade ago), but no you'll need rust+llvm experts to port that whole crazy toolchain. It's for your own good.

What you're saying makes sense in any other domain except cryptographical primitives. With those, the basic correctness of the code will depend on details of the compiler and processor architecture, and it is extremely likely that the code will be fundamentally incorrect when used on other architectures or compilers.

And this is not just a matter of fixing some little-endian assumption, it's a matter of understanding microoptimizations, instruction reordering, prefetch behavior, branch prediction etc. Ensuring code has predictable runtime in the presence of an optimizing compiler and out of order processor is extremely difficult.

It's probably just as smart to roll your own crypto instead of trying to use a known library on a new platform, because the hardest parts of rolling your own will have to be done anyway.

Here you seem to be demanding that the authors and maintainers of their own cryptography library lighten up about security, because you find the way they manage their own package to be controlling. You see some irony in that, right?

I don't read /u/ploxiln as demanding anything. They seem frustrated with the changing dependency profile of the dependency, and a little frustrated with the way some people like to shout "security" as if it's a trump card, ignoring that security is a spectrum and that the only perfectly secure software does nothing (perfectly).

This is a security library. In fact, it's a security library that was created specifically to harden and user-proof less secure alternatives. If you don't care as much as they do about security, use a different library (or just keep using the pre-Rust version of this one).

> If pyca/cryptography was instead advertised as "python crypto library for rust lovers"

I feel like this really only makes sense if pyca/cryptography had planned on adding the Rust dependency from the very beginning (or from very early on). Is there any indication that was the case?

> but not just that, it depends on a custom-patched llvm!

This doesn't seem to be true [0, 1].

[0]: https://rustc-dev-guide.rust-lang.org/backend/updating-llvm....

[1]: https://news.ycombinator.com/item?id=26217182

>> Strongly prefer to upstream all patches to LLVM before including them in rustc.

> That is, this is already the case. We don't like maintaining a fork. We try to upstream as much as we can.

> But, at the same time, even when you do this, it takes tons of time. A contributor was talking about exactly this on Twitter earlier today, and estimated that, even if the patch was written and accepted as fast as possible, it would still take roughly a year for that patch to make it into the release used by Rust. This is the opposite side of the whole "move slow and only use old versions of things" tradeoff: that would take even longer to get into, say, the LLVM in Debian stable, as suggested in another comment chain.

> So our approach is, upstream the patches, keep them in our fork, and then remove them when they inevitably make it back downstream.

... so there are always some outstanding patches rust applies to the llvm codebase.

For Rust's LLVM fork, sure. But as Steve Klabnik noted in the first comment I linked, unmodified LLVM is supported.

In addition, later down in the comment chain:

cycloptic:

> Can't there be a build option to not use the LLVM submodule, and instead use the system LLVM?

steveklabnik:

> There is. We even test that it builds as part of our CI, to make sure it works, IIRC.

For a more concrete example, Fedora supports choosing between system and bundled LLVM when building Rust [0, 1].

[0]: https://news.ycombinator.com/item?id=26222190

[1]: https://src.fedoraproject.org/rpms/rust//blob/rawhide/f/rust...

> I feel like this really only makes sense if pyca/cryptography had planned on adding the Rust dependency from the very beginning (or from very early on). Is there any indication that was the case?

I am sure this idea surfaced several times in IRC or possibly in the mailing lists. Certainly, the authors have been toying with handling ASN.1 in rust since 2015 [1], which I guess will be the next logical step.

I do agree that this is mostly a political stance. pyca/cryptography is a wrapper sandwiched between a gigantic runtime written in C (CPython/PyPy) and a gigantic library written in C (openssl).

The addition of Rust as dependency enables the inclusion of just 90 lines of Rust [2] where the only part that really couldn't be implemented in pure Python is a line copied from OpenSSL [3] (i.e. it was already available), and which is purely algebraic, therefore not mitigating any real memory issue at all (the reason to use rust in the first place).

The change in this wrapper (pyca/cryptographic) does not move the needle of security in any significant way, and it is really only meant to send the signal that adding Rust in all other Python packages and especially in the runtime itself will now come at no (political) cost.

[1] https://github.com/alex/rust-asn1

[2] https://github.com/pyca/cryptography/blob/main/src/rust/src/...

[3] https://github.com/openssl/openssl/blob/OpenSSL_1_1_1i/inclu...

> The addition of Rust as dependency enables the inclusion of just 90 lines of Rust

My understanding is that this is just the beginning, and the whole reason it's only a small amount is precisely to do it in small steps, correctly, rather than re-writing the entire world in one go.

And why should they re-write the entire world? It's just a wrapper library to openssl and it was marketed heavily to the community (at the beginning) as one where maintainers would follow good practices and not try to write security-sensitive code as they are not security experts, but just rely on openssl so that all focus go into the same place.

So either that's still valid (and not too much of rust will come in to make a real difference security-wise) or they have revisited their position and will re-implement a bunch of openssl logic (in rust apparently though, and not in Python as it would be more logical, and as golang does successfully). And in the latter case, why just not focus on wrapping rustls instead?

In either case, the hand is being forced: it's a small amount, and I don't see how it would have been an excessive burden to maintain such a small logic as an opt in for a period. It makes much more sense to read this as a move to forcefully push rust into the python ecosystem.

> So either that's still valid (and not too much of rust will come in to make a real difference security-wise) or they have revisited their position and will re-implement a bunch of openssl logic (in rust apparently though

Looks like it's the former. From the initial (?) GitHub issue discussing the move to Rust [0]:

> We do not plan to move any of the crypto under the scope of FIPS-140 from OpenSSL to cryptography. We do expect to move a) our own code that's written in C (e.g. unpadding), b) ASN.1 parsing. Neither of those are in scope for FIPS-140.

> and not in Python as it would be more logical

Is Python actually suitable for cryptographic code, especially if constant-time operations are needed?

> I don't see how it would have been an excessive burden to maintain such a small logic as an opt in for a period.

And then when said logic stops being opt-in, why wouldn't the same problem arise?

[0]: https://github.com/pyca/cryptography/issues/5381#issuecommen...

> Is Python actually suitable for cryptographic code, especially if constant-time operations are needed?

No, but as seen in the first chunk of rust code added, the amount of logic that needs to be constant-time is a) very, very limited to a few primitives only b) algebraic in nature (put differently, it's not where memory bugs will pop up, so using rust over C doesn't even buy you much).

For instance, ASN.1 parsing doesn't need to done in constant time in the vast majority of cases (and in all cases I am aware of).

> And then when said logic stops being opt-in, why wouldn't the same problem arise?

Some friction will certainly remains, but it will be nothing compared to the current breakage.

> No, but as seen in the first chunk of rust code added, the amount of logic that needs to be constant-time is a) very, very limited to a few primitives only b) algebraic in nature (put differently, it's not where memory bugs will pop up, so using rust over C doesn't even buy you much).

That's fair, though if some other part of the codebase is ported to Rust anyways sticking to C doesn't save you, unfortunately.

> For instance, ASN.1 parsing doesn't need to done in constant time in the vast majority of cases (and in all cases I am aware of).

I'm curious why this has to be done in C/Rust. Performance?

> Some friction will certainly remains, but it will be nothing compared to the current breakage.

What would be different then compared to now that would reduce breakage to such an extent?

> That's fair, though if some other part of the codebase is ported to Rust anyways sticking to C doesn't save you, unfortunately.

This should be turned around. There is no evidence that part for the codebase really need to be ported and for which Rust makes a real difference.

> I'm curious why this has to be done in C/Rust. Performance?

ASN.1 is only used during handshakes and I/O of files (the chunk of rust added covers loading of PKCS#7 files which are typically quite small, and not typically dealt with in massive numbers). I doubt the performance hit would be so high. Also, the package wbond/asn1crypto has shown that doing ASN.1 in pure Python can be quite fast.

> What would be different then compared to now that would reduce breakage to such an extent?

The ability to try the feature out and still have a plan B if it doesn't work out, and the possibility to have a smooth, long, relaxed upgrade plan with good warnings and more time to prepare. But to be honest, while writing all this chain of comments, I really doubt there was a real need to add rust into the mix (other than the political angle I already mentioned).

> This should be turned around. There is no evidence that part for the codebase really need to be ported and for which Rust makes a real difference.

I was thinking that if the maintainers were planning on porting some other larger part of the codebase, then starting with something small/relatively inconsequential would be a good first step, and keeping it in C wouldn't provide much benefit once said other larger part were ported.

> Also, the package wbond/asn1crypto has shown that doing ASN.1 in pure Python can be quite fast.

Interesting! I'm curious why the maintainers didn't opt for that approach instead.

> The ability to try the feature out and still have a plan B if it doesn't work out

Would people have tried this feature out if it were made opt-in? It's clear that the initial announcements reached far fewer people than one might like, and I honestly have no idea how many people would see a build-time warning.

> rust depends on llvm, itself a huge complicated finicky package

gcc is also a "huge complicated finicky package"... Any decently sized compiler is.

Reply on news.ycombinator.com