Dependency on rust removes support for a number of platforms | Hacker News Reader

Dependency on rust removes support for a number of platforms

github.com

49 pointsallanbreyes56 comments

It's unfortunate that some platforms got dropped, but OTOH all these platforms seem to be 30 years old, niche, and discontinued by their original manufacturers.

If someone has antique computer that still works, that's great, but they shouldn't expect to have the latest software for it.

> Only users on Alpine (musl), BSD, other hardware platforms, and distro packagers are affected.

Not just niche or antique!

Rust supports musl and a bunch of BSDs, so if something broke with their packages, it should be only a matter of fixing the builds. That's a much smaller problem than adding a new old architecture to LLVM and rustc.

Indeed, a commenter demonstrates that resolving the issue on Alpine Linux was as simple as adding an `apk add rustc cargo` to their Dockerfile.

It's not just about adding new architectures to LLVM and rustc, it's also about bootstrapping supported architectures on all architectures supported by a distribution.

This isn't fully done in Alpine yet, and took a long time to do in Debian.

This is the sort of opinion that evaporates once you become the victim of what’s happening here.

IMO Rust should have a C backend, then the maintainers could drop it as a build dependency and not break existing platforms. Though I expect there to be political reasons for Rust’s conspicuous lack of a C backend.

LLVM had a C backend which went unmaintained until the Julia folks picked it up and updated it for LLVM 10. I'm not sure about its current state.

There's mrustc which compiles Rust to C. Rustc itself supports two back-ends now (LLVM and cranelift), so more should be possible if someone wanted to contribute and maintain them.

I don't think there's any ideological opposition to having more back-ends. It's just a huge amount of work, and is lacking volunteers, especially for niche platforms.

The amount of work doesn’t seem to be the limiting factor. Rewriting large amounts of C code into Rust is a lot of work too but that doesn’t stop Rust justice warriors from doing that.

I think there are alternative implementations that are (much) less mature than rustc. And I wonder if using those implementations would be good for a project, that requires security...

For one, respected community member pcwalton has argued against compiling to C, citing the difficulty in producing performant, memory-safe output.

For another, from personal experience, I can comment that compiling to C in such a way that doesn't leak abstractions left and right is quite challenging. It's pretty hard to produce memory-safe C, so it's a ton of work, and the payoff is pretty marginal, as the most important platforms already are supported by LLVM.

Automatically generating “memory safe” C is no harder than automatically generating “memory safe” LLVM bitcode.

I think the issue is all of C's undefined behavior

For the most part, that can be worked around as well, with careful use of unsigned arithmetic. See https://git.yzena.com/Yzena/Yc/src/branch/master/include/yc/... for some examples. All arithmetic is unsigned, but I use it to simulate 2's complement signed.

That is not difficult to avoid if you are generating short primitives. You can for the most part just convert the LLVM bitcode that the Rust compiler would output to the equivalent C snippet. Since each snippet is short, you can trivially check if it invokes undefined behavior. LLVM bitcode also can exhibit undefined behavior.

> You can for the most part just convert the LLVM bitcode that the Rust compiler would output to the equivalent C snippet.

Sounds like a good argument to resurrect the LLVM C backend. As it stands, the Rust core team has no desire to implement a C backend, as it would be a ton of work for not much gain.

>> IMO Rust should have a C backend, then the maintainers could drop it as a build dependency and not break existing platforms.

That's a lot of work you expect from people developing a language you don't even want to use.

I expect nothing, it is a casual opinion. You’re projecting.

Indeed. If the platforms are business critical then an impacted company should put money on the table to do the necessary work.

If they are not critical then well, why should other people care?

Also, Python packaging is pure hell.

What's more, the Gentoo maintainer that filed the issue lists a number of platforms that CPython itself doesn't even support.

If you are using open source packages in a corporate setting you really should be a) pinning versions b) maintaining secure, internal mirrors instead of always pulling from Github etc. That would prevent breaking your builds without being intentional about it, regardless of what changes upstream introduces.

There is a lot of "right side of history" / "get with the program" smugness coming from the Rust camp, this bug report and other discussions. Claims of improved security are used as an ultimate, unbeatable-by-definition, trump card.

This might prove right eventually - or might very well end up just like Java, for which similar claims were made. The smugness around Java was moderated a bit by the more corportey image of it, but the gist was quite similar nonetheless. Let's just say openly: the smugness is using up a lot of natural goodwill, and generating its own pushback. As there's no widely acknowledged "equal competitor" to Rust (why?), something feels "off" about the whole situation.

One of they devs did say "C is a bad language to implement parsers for e.g. ASN.1". Unfortunately, there are enough examples out there [0] to prove him right. Not all of them will be mitigated by Rust (e.g. something like CVE-2019-17359), but it's probably worth it long-term (idk, i don't feel the pain of security issues day-to-day, only occasionally).

[0] https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=asn.1

To add data on the Rust side:

https://github.com/rust-fuzz/trophy-case

there are a few Rust ASN implementations. They've been caught running out of memory and having arithmetic overflows, but no segfaults or use-after-frees. Rust doesn't prevent all problems, but things that slip through tend to be less severe.

One could ask whether python would have been an option for writing the parser. There is for example the asn1crypto library, which is pure python.

I'd have to assume the issue there is really speed.

Edit: @alex made a much better comment here:

https://github.com/pyca/cryptography/issues/5771#issuecommen...

One point that I didn't think of, but makes a ton of sense:

>There's no way to implement constant-time code in pure Python securely

Claims of improved security are used as an ultimate, unbeatable-by-definition, trump card.

Having an authoritarian tone seems to be a prerequisite for being in the security industry. Unfortunately.

Many users care much more about working software than they do secure software until suddenly they have a major live exploit on their stuff. This causes many people in the security world to get very tired of the whiplash cycle of "No we don't want to fix this issue it's probably minor" until "Oh my god, why didn't you fix this it caused us millions of dollars in losses" hits.

Especially in the security hardware world, getting customers to upgrade is like pulling teeth.

> there's no widely acknowledged "equal competitor" to Rust

this is largely a myth

Please tell me who this "equal competitor" is. I would like to compare it to Rust.

It would be hard to find a thing you can do in Rust that cannot be equally well or better done in C++.

Any safety that can be provided by a library+Rust can be provided by another library+C++, with only a different apportionment of responsibility between library and language.

pyca/cryptography replaced C code with Rust code without bumping the major semver. This broke people's CI because it removed support for certain platforms which Rust barely supports or doesn't support at all. People are arguing over this change in the github comments, but the problem from my perspective is just with semver. The maintainers didn't do semver right.

Edit: the maintainers didn't do semver right because the maintainers didn't do semver

I think the maintainers mentioned that they don't use semver. https://github.com/pyca/cryptography/issues/5771#issuecommen...

First, they don't use semver [0]

Python is all over the place with versioning, making it hard to predict what scheme anyone is using. People either falsely assume semver or just don't do any version constraints, both leading to problems. Whats unfortunate, is the Poetry project is putting their head in the sand on this and not letting you patch transitive dependency versions [1]

Even if they did use semver, its still a contentious topic within projects using semver of what all is "included". Some people take an idealistic perspective of "if it might break me" but any change can break them [2], making it impractical. I've seen others take this as a sign that semver is impractical and shouldn't be used. I feel its a limited but useful communication tool; we have to accept imperfections in its use.

[0] https://cryptography.io/en/latest/api-stability.html#version...

[1] https://github.com/python-poetry/poetry/issues/697

[2] https://xkcd.com/1172/

I did not realize how inconsistent python versioning is, or that compatible release (`~=`) doesn't use the semver way. https://www.python.org/dev/peps/pep-0440/#compatible-release

Say what you will about the JavaScript ecosystem, but this is something it executes really well on

One of the perfect example of many toxic comment of entitled users...

I bet you will be just as pissed off when others break your environment too...

It's for this reason I've never liked the sprawling dependency-of-dependency mess that a lot of software projects seem to have.

Yes; it’s particularly bad when you mix languages like this.

I maintain the nodejs foundationdb libraries. Fdb doesn’t have a published wire protocol - bindings are expected to wrap a dynamically linked C library which contains the protocol implementation and net code. And that library is very “clever” - it leans heavily on codegen (via mono) and hand written asm to support a high performance actor model internally. The downstream effect is that FreeBSD support for database clients has only been added recently. There’s no native Apple M1 client support. There’s been issues with code signing in Java ... and so on. Generally you can only talk to a foundationdb server from an x86_64 client that runs Linux/macos/windows. Which is arbitrary and frustrating.

I hope wasm/wasi eventually becomes the standard for libraries like this. Then the binaries/packages we distribute can work in different languages, different OSes and different architectures without needing all this drama. Wasm bundles fit perfectly in pip/npm/etc. Write them in any language you want and they run fast & run everywhere.

I'm torn on this, In the past, I've been broken by changes breaking me.

On the other hand, you have you have an open source project that has to deal with the schedule/quality/cost trade off and people are expecting support for particular platforms without there being funding coming in to support those efforts. How much free work do they do so other people's company's continue to function?

Yeahhh, I understand the frustration of a breaking change to the build system, but I don't understand the vitriol at the maintainers.

Ultimately, projects upgrade/move. If you are dependent, that's something you need to deal with. If you're not ready to deal with it, then pin your dependencies until you are.

Many of the issues in the thread aren't necessarily problems with cryptography - for example pip not being able to deliver wheels to Alpine.

The authors aren't exactly cloaking themselves in glory either, to be fair.

I get that the maintainers do what they want, but following semantic versioning should prevent this kind of problems.

Its not clear cut what all is included with semver. Is switching from a C89 to a C99 a semver-breaking change? Is using a Python 3.1-specific function a semver-breaking change? Is fixing a bug or adding a new feature a semver breaking change [1]?

There is no clear cut answer in all of these situations and applying purity tests to semver instead just push another crowd of developers to abandon semver completely, making it hard to get any information communicated from your dependencies up to you.

[1] https://xkcd.com/1172/

Furthermore, semver states:

> MAJOR version when you make incompatible API changes,[0]

A build-time change isn't an API change, at least in my understanding.

[0] https://semver.org/

.. and to pin dependencies, or expect the occasional CI breakage. (you could argue a major version bump would've been nice instead of a minor one. but i don't even want to open that can of worms.)

Honestly, even with minor versions, I'd prefer to use something like dependabot, or for a bot to open a pull request bumping versions. Tons of authors mess up semver in subtle ways, it's just much easier to avoid problems if you just pin dependencies.

I've started doing this with Nix for my own Rust projects, using the technique described here[1]. Planning on setting up a GitHub workflow to automatically open pull requests with bumped versions of nixpkgs/rust.

[1]: https://christine.website/blog/how-i-start-nix-2020-03-08

I’m not discounting the pains that some people are going through because of this change, but this response [1] in the issue says

> The new Rust code adds exactly 0 (zero) runtime packages to Cryptography. Rust, Cargo, pyo3, its dependencies, and setuptools_rust are build-time dependencies only.

Aren’t there tools available to build this on a supported platform and integrate the binaries in the systems in use? It is a bit convoluted, but seems like a solution at least for some (?) cases at additional cost.

[1]: https://github.com/pyca/cryptography/issues/5771#issuecommen...

Unfortunately no, python cannot distribute pre-compiled dependencies for Alpine Linux yet.

Why don't they pin their dependency version? Or is that hard to do with transitive dependencies in Python? (I'm not familiar with Python)

They can pin, they just didn't.

It depends on how you want to pin. You probably always want the latest API compatible version for a package like cryptography. But as I read from that thread, it seems cryptography doesn't use semver, thus you doing something like `>=3,<4` is not really feasible. In addition, this change was introduced at 3.4.

Pinning is specifying the exact version, so that wouldn't have failed.

I disagree. If the library follows semver, or has a predictable way of managing its versions, I would personally set it as broad as possible without breaking compatibility. Especially with something like cryptography. If a critical bug is found, you don't wanna be stuck at a version from a few years ago. Setting a broad pinned version allows you to update this as part of your day to day development.

I mean, literally the term "pinning" is setting the exact version.

If you're setting just constraints but not a specific version, then you're not pinning.

There is no standard in the Python community for versioning. From this, a lot of projects just never constrained their versions while a lot of other projects assume semver when specifying their version constraints, both wrong.

Then there is the problem of there not being a standard dependency management system. Your `setup.py` can specify version constraints. You either over-constrain in there or have to add a whole separate process for locking your constraint.

- You could use `pip-compile` to get a platform-specific locking of constraints, requiring you to run this for every platform and python version you support. At least, since you duplicate `setup.py` into a `requirements.in`, you can override transitive dependencies.

- You could use `pipenv` and just capture what you happened to install, from constraints or directly, from a specific machine

- You could use Poetry which solves most of these problems except they've put their heads in the sand regarding how bad versioning is within the Python ecosystem and refuse to support overriding transitive dependencies despite being modeled off of Rust's Cargo which does support it despite the Rust ecosystem being good with versioning [1]

[1] https://github.com/python-poetry/poetry/issues/697

Reply on news.ycombinator.com