Python Cryptography, Rust, and Gentoo
lwn.net
lwn.net
> The LLVM developers have been somewhat leery of taking on new architectures, unless they can be convinced there will be long-term support for them, which is understandable, but makes the problem even worse.
I think, ideally, there would be a concerted effort to extend LLVM compatibility to those platforms. But I don't think it's unreasonable to assume that there simply won't be--I sometimes run into problems on aarch64, and most of these are far more niche than that. As the article mentioned, it seems likely that eventually the ability for certain platforms for get support on new compilers will be a make-or-break for their ability to stay current with upgrades for a lot of the more fundamental packages.
Alternatively, I guess, there could be effort to get a new gcc-backend for more langauges; I believe there's an ongoing effort for Rust[2], but who knows down the line what that might look like. Eventually, this will probably need to come to a head.
It's ok for Gentoo to drop support for these platforms IMO, the least used architectures need to be relegated to the museum sometime.
Alternately, python cryptography could have chosen a different memory safe language, like Go, which is supported by GCC. Why did they intentionally choose a poorly fit language like Rust? Use the best tool for the job, yes?
LLVM has changed the situation quite a bit though. GCC could make itself easier to hack on but it might be too late.
It's arguably true that RMS's zealotry actually backfired spectacularly.
I think it's probably for the best; having two competing solutions with different priorities is probably a win for everyone, gcc and clang users alike. gcc has improved much in the last few years in part due to LLVM/clang's pressure.
I've seen people bring this up a few years ago, but there wasn't much change in attitude on rms' end. I don't keep up on the day-to-day of gcc, so perhaps things changed in the last few years(?) I would be surprised if this would happen under rms' tenure though.
The main benefit of a GCC-based Rust compiler would be disentangling Rust from rustc, specification from implementation.
In some cases more than others. I feel the pain of people who want to keep running private unique architectures. I wish it would be easier for them.
But mentioning s390? Companies that bought mainframes from IBM don't get to complain about compatibility with open source projects in my opinion. They can deal with that issue themselves and contribute back. (Or fund a solution) Show me how much extra you spent on wages and contracts related to that choice and I'll show you my tiniest violin.
Put some of this upset-energy into building out gentoo+llvm. There's no way you're going to convince the entire programming world to stop using llvm at this point, even if you claim it's reasonable.
> https://github.com/M680x0/M680x0-mono-repo
Patches for the M68k backend are discussed on reviews.llvm.org.
I have also started a Rust port for m68k already that depends on the LLVM backend work above:
> https://github.com/glaubitz/rust/tree/m68k-linux
For other architectures, the goal is to get gccrs moving forward and merged upstream:
> https://github.com/philberty/gccrs/
For anyone interested in helping, there are Bountysource campaigns I started to support both efforts:
> https://www.bountysource.com/issues/90829856-llvm-complete-t...
> https://www.bountysource.com/issues/86138921-rfe-add-a-front...
I would much rather see rustc_codegen_gcc upstreamed: https://github.com/antoyo/rustc_codegen_gcc
A GCC backend for rustc would provide all the code generation of GCC without duplicating the Rust frontend.
FWIW, I think gccrs also tries to use the Rust frontend if I remember correctly.
FOSS isn't about providing other people whatever code they need for free.
If someone decides to do stuff in Rust they have every right to do so, and there's nothing their users can do except switch to something else or fork and maintain the code themselves.
The thing that also annoys me is that the platforms that are going to lose support are either so old or so obscure I sincerely doubt there are more than a handful Gentoo installations running on those.
Damn Alpha has been dead for more than 25 years, if you care about it so much you don't want to switch to a still supported architecture you should start invest real money on supporting the software stack yourself.
That's true. But FOSS is also about cooperation and Rust in particular is marketing themselves as open and inclusive.
> If someone decides to do stuff in Rust they have every right to do so, and there's nothing their users can do except switch to something else or fork and maintain the code themselves.
And what would Rust developers think if LLVM decided to make such a radical change to their codebase that Rust would no longer be able to use LLVM as a backend? Would also be in favor of an LLVM fork?
> The thing that also annoys me is that the platforms that are going to lose support are either so old or so obscure I sincerely doubt there are more than a handful Gentoo installations running on those.
There is also just a handful of users running Haiku, Hurd, NetBSD as compared to Linux. Yet I have not seen that argument made anywhere.
The nice thing about FOSS is that also smaller voices and groups are being heard. And I think that's great.
> Damn Alpha has been dead for more than 25 years, if you care about it so much you don't want to switch to a still supported architecture you should start invest real money on supporting the software stack yourself.
The nice thing about these old architectures is that they often help expose bugs that remain hidden on main targets. We have fixed a lot of bugs that only became visible on architectures like SPARC and M68k.
I dont see your point. Both Rust and LLVM are open for anyone that wants to implement and support new platforms. Espressif is submitting a new Xtensa backend for their architecture to LLVM, and the community has been using this to port Rust to the ESP32. If anyone wanted to step in and write a backend for Alpha or whatever, nobody would be there to stop them.
The real issue is that people expect everything to stay the same and don't change so to avoid them any extra work. If you care so much about a dead architecture you should be aware that things may break or stop working, or that someone could stop supporting it, especially if you got everything for free.
> users running Haiku, Hurd, NetBSD These platforms are run by determined people that often are ready to shoulder on themselves the task of porting stuff to their OS. I don't think they expect anyone to work and do stuff in their place. These also mostly run on platforms supported by LLVM (except netbsd, which has billions of ports), thus only requiring to adapt the OS part of the compiler (which is arguably simpler than writing an LLVM backend)
No package system, no module system, no memory safety. It only enjoys current popularity because of inertia.
That doesn't mean this isn't the right direction to go in, but some friction is to be expected, and ideally also reduced whenever possible in this "transitional period". I don't think this can just be dismissed with "C sucks".
C is the undefeated queen of compiler availability. This is a discussion very much about availability.
Just as an example, I came across a good blog of someone installing Gentoo on an Alpha workstation [0], which I find to be great, seeing as it is officially supported.
Maybe it's just my dogmatic, impractical preference that the software we write should try to work on older hardware, but seeing the entire LLVM ecosystem in this sort of situation makes me sad.
Portability is nice but at some point it has to stop. If literally no one is working on LLVM support for an architecture, it's dead.
Nah, m68k is one of the most actively developed architectures. It has very active kernel developers and we're soon seeing an official LLVM backend for the architecture coming (search reviews.llvm.org).
Alpha and HPPA are also actively maintained. IA64 is currently hanging a bit on a string, but we're most likely going to save it for the near future.
Admittedly I haven't looked at the Cryptography code base to see how much it depends on native code but I wish people made a conscious effort to consider the downstream ramifications of including a particular dependency. I am the maintainer of a quasi-popular JavaScript library and I deliberately wrote my own helper functions for things like base64/UTF-8 conversion just to avoid potential issues (it's been around since 2008 so JavaScript was also a hot mess at the time).
Having once upon a time preferred pycryptodome for some reason or other (various additional support to the standard at the time), perhaps I'm out of date or confused as to what the best standard python cryptography library should be for someone who might occasionally play in the "hazmat" space (to use the terms of the article's library).
Cryptography on the other hand while providing the low-level functionality if required, provides a higher level interface on top of those primitives that makes it hard to misuse and thereby potentially make cryptographic mistakes.
We got a RISC-V and WASM backend into LLVM very quickly because people gave a f*ck about these. Stuff like IA64 or Alpha have been dead for so long I'm surprised there's still support in the kernel for those.
It just has no maintainer at the moment. But that has been true for other architectures such as SuperH which have gained new maintainers in the meantime.
Alpha is still well maintained in the kernel and in Debian (and Gentoo) which is why it's still being around.
The same is true for HPPA, M68k and SuperH. IA64 has been a little dormant but we're trying to keep it alive although its chances are worse than for other architectures.
"Back in Gentoo-land, it turned out that the cryptography dependency for Portage came because it was using urllib3 and requests. Those two packages in Gentoo are dependent on cryptography, but it turns out that they do not actually need it. A pull request to fix that was merged, so the problem for Portage, which is pretty fundamental to the operation of a Gentoo system, was averted."
Hence, pip now depends on software that is not tested on some platforms officially supported by Python…
Sources: - https://doc.rust-lang.org/nightly/rustc/platform-support.htm... - https://buildbot.python.org/all/#/grid
No offense to be taken, but by that I mean the kind of software where they have some personal principes that they impose on everyone as being the one truth and considering users as kids that need to be framed.
I liked so much more the pycryptodome project that took care of covering the most important things for users like retro-compatibility.
I have worked in the embedded world and there, that kind of sudden change, with extra heavy dependency at compile time is a real pain in the ass!
So, the conclusion is probably like for Gentoo to avoid using the cryptography lib, that would certainly avoid further other unilateral hard sudden changes in the future...