Hacker News Reader

Comment by weinzierl | Hacker News Reader

"rust is _almost_ as complex as C++."

I don't agree, but even if it were true, C++ has so much accidental complexity (because of its legacy) that you cannot meaningfully compare it with Rust, which by and large is very well thought out.

In general, if something is complex in Rust that is because it's in the nature of the problem and not because of some quirkiness in the solution.

Borrowing is the prime example. People find it complex, but just because the C++ compiler doesn't check the rules for you, doesn't mean you have follow them in C++ as well (which is tedious and error-prone) or know exactly why you don't have to follow them (which is even harder).

I admit though, that the Leakpocalypse (which the article talks about, even if it doesn't call it like that) is an area where accidental complexity lurks.

View on HN

Pin is another one. I still can’t wrap my head around it. I’m sure it’s simple and the docs are great, but having read them multiple times I’m not really more clear. I probably need to see a bunch of hands on examples. I wish they’d gone the Move route instead of rushing to 1.0 and then retrofitting in Pin.

But I agree, in general things in rust are a lot more thought out and experiments are relegated to unstable which is fine with breaking changes.

I do wish though that they had a better versioning story in the language for making backward incompatible changes more a matter of course. It’s great already for “user space” code via editions, but as the author notes making certain kinds of changes brings up risk and uncertainty because there’s not a well trodden path to support changing core semantics in the language and having it work gracefully without needing a migration en masse.

> I wish they’d gone the Move route instead of rushing to 1.0 and then retrofitting in Pin.

That is easy to say, and I don't disagree with the sentiment, but there are valid reasons for setting a 1.0 release date and cutting corners to get there. If they had not done that, there probably was a real risk of running an infinite loop of "let's wait for Move to be implemented", "great, now let's wait for Leak to be implemented", "great, now let's wait for..." and a 1.0 never gets put out. Rust may not be perfect, but at least it's stable and people are actually using it.

Pin used to confuse me, but now it's clearer. It's right in the name: Pin this thing to where it is in memory, never try to move it.

Rust can and does move things around as it feels the need. So you need something like Pin if what you've built can't move.

It's confusing if you're coming from a C++ background because in C++ something only "moves" if you explicitly std::move it. In Rust its effectively the reverse. A crude way of thinking about Rust's call pattern is that it's basically the opposite: as if every call was a std::move operation.

So Pin is basically just saying: you can't move this, don't try. The memory can't be moved. And it's mostly used in corner cases where some kind of explicit memory management is involved.

Most Rust users should not have to worry about it. It's one of those things that becomes more of a concern if you're writing library code for low level datastructures, or interfacing with C/C++ libraries.

> Most Rust users should not have to worry about it. It's one of those things that becomes more of a concern if you're writing library code for low level datastructures, or interfacing with C/C++ libraries.

Don’t all Rust users writing async code have to worry about Pin?

Not really, the async libraries hide the implementation details from you. It's an issue for library developers.

I've never had to and I have stuff that uses async all over. The only time I've seriously had to deal with Pin is for some pretty low level stuff: manual memory/heap management for database datastructures (paged btree) and for some work stuff related to working with a C++ library that uses zeromq, etc. The point being that in both cases these were frobby dark corners where the mechanics of the pinning ends up being hidden from the user.

People writing application code in Rust should not encounter it often.

You get to avoid pin until you can’t, and then it’s because of some terribly unintuitive advanced usage corner case.

For instance, the async_stream crate would be much nicer if you didn’t need to pun_mut! the stream instances it provides:

https://docs.rs/async-stream/latest/async_stream/

Conceptually I understand the motivation. But trying to figure out how to express it frustrates me so much that eventually I just gave up.

Fair. The documentation made my eyes glaze over and I got confused. It was only once I needed it that it made sense to me. And I still don't understand some uses of it.

One advantage C++ has over Rust is that there are multiple fairly compliant C++ implementations, and the process of making them created some shared understanding of how the language actually works from an implementation perspective. There are several aspects of the current Rust language that were effectively defined by arbitrary decisions made by the implementors, which aren't really spelled out anywhere besides the code.

Do you have examples?

I think in many ways Rust is doing a better job of tying down the tricky details than C++ has.

There is no effort to explain how pointer provenance works in C++ at all. The state of the art is if you write a provenance bug, someone from a compiler vendor will gesture at DR/260 which says provenance exists but WG14 (so C, not even C++) declines to explain how it works, and that's all you get. In contrast Aria's "Experiment" (https://doc.rust-lang.org/std/ptr/index.html#strict-provenan...) has gone rather well. The vast majority of people who want to do acrobatics with pointers can do what Aria says you have to do to get coherent behaviour, and by following her rules they can make code which works in Rust.

WG14 produced a document clarifying provenance in C (which Aria knew): https://open-std.org/JTC1/SC22/WG14/www/docs/n3005.pdf

Given your username, I assume I'm addressing Martin Uecker and I want to commend you for putting the work in on this problem over a long period. While a draft TS is closer to actually resolving a 20+ year old problem it's not there yet and this draft is, if you pardon me, in my opinion more than three months work from publishable.

In contrast, Aria went from "We should see whether strict rules are workable" to the current set of Rust nightly features relatively quickly.

If I want to attempt pointer stunts in C, N3005 is probably worth reading first, but it isn't actionable from what I can tell. Contrast Aria's changes, I can go try that (in nightly Rust) and see it works or why I can't / mustn't do what I wanted under Strict Provenance and who I ought to talk to about that.

I wish you luck for C 29 (?) but that's a distant future, and we're contrasting C++ here, so as with #embed you might take this obvious must-have for C29 and find that somehow WG21 are still so far behind they can't do the same for C++ 29.

You realize that this experiment is basically our proposal translated to Rust? Also I wonder why it is an issue that N3005 is still a draft? Rust also does not have an ISO standard.

In what sense do you believe it's "basically our proposal translated to Rust" ?

N3005 explains four provenance variants, but settles on specifically PNVI-ue-udi for the future of C. But Aria's experiment focuses on roughly what you'd call PNVI-plain, reasoning that Rust largely does not need the extra headache of exposure.

N3005 doesn't provide an implementation, so Aria can't have merely "translated" that. It doesn't provide an API design (so not that). Which leaves basically the fact that you've extensively discussed the semantics of these variants.

Let's try specifics: Take a common feature of Rust's design, ptr::map_addr. Where is equivalent functionality described for C in N3005 ? In Rust it's perfectly natural to write a lambda here, in C perhaps you'd spell this very differently, but N3005 doesn't seem to offer such a thing.

It's an issue that N3005 is still a draft because it goes to how long this wound has festered in C comparatively and will still be there. If this was merely a theoretical problem I wouldn't be bothered, but the whole reason DR260 was raised is that it's not a theoretical problem.

The "expose" mechanism as described in the document you linked to corresponds exactly to PVNI-ae-udi (including the "expose" terminology which was taken from N3005 or an earlier draft). Aria's document is very vague (is there a precise version?), but the explanation using angelic determinism ("guessing the right provenance") corresponds to PVNI-ae-udi. "Strict-provenance" does not need the "ae-udi" part but the doc seems to acknowledge that this is not enough and if you ignore the uintptr_t conversions which make us of this in C, you also have strict provenance. N3005 doesn't provide an implementation, but there are - of course - implementations. First, because it captures what most C compilers do (although there are still some differences for some). Then there is a precise mathematical model in N3005 and there is Cerberos which you can run code and analyze. You are right that we do not have the convenience tools such as ptr::map_addr. Maybe someone should build a header for C that provides similar convenience macros. But the hard part is getting all the bugs in the compiler backends fixed. Last time I checked, Rust was also still affected by the same bugs in LLVM as C (but those may be fixed by now).

I should also give a bit of background information, because I find it funny that some surface-level API for Rust is portrait as "solving" the problem in three months which C struggled for years. (And by that I do not want to downplay Arias impressive work, it is a very nice and clean interface). The fundamental problem is that compiler writers build optimizations without a clear mathematical understanding of the rules. One could argue that this is the fault of the C standard by not being a mathematically precise specification, but this was also never really the goal of ISO C. The main issue is that compilers implemented rules that were not consistent even between different optimizer passes. WG14's main mission is to standardize existing practice. DR260 (failed) attempt to clarify some of this came out this. I agree that this should have been done better, but the important thing to understand is that WG14 as a consenus-based standardization committee is not really equipped to do this kind of work. The basic assumption is that compilers implement sound models, which can then later be standardized, only harmonizing differences between different compilers. But here we had to clean up the mess compiler vendors created. Not at all what WG14 was meant for. To be able to start fixing compilers, one had to find a common and consistent formulation for provenance. For C this meant also considering what all existing code needs and what optimization different compilers implement, and deciding which version to pick and what specific behavior is a useful optimization, what is a compiler bug, and what optimization can be scarified to come to a consistent formulation. Only with this can one decide what an optimizer bug is and what not. To the extend these bugs are still there they affect Rust too and to they extend they are fixed, Rust benefits. (vice-versa C benefits from Rust's push towards safety which also puts pressure on compilers to fix such bugs). But in any case, the API on top is not at all the most difficult part and Aria's document is not precise enough to even differentiate between subtle points of provenance.

I think you're correct that it's a big problem that the serious compilers don't actually have coherent internal behaviour, but I think that's a distinct problem which has been masked by the provenance problem in C and thus C++. It meant that real bugs in their software can be argued as "Works as intended" pointing to DR260 rather than anybody needing to fix the compiler.

For LLVM the effect of Rust has been to drag some of these issues into the sunlight, as it did for aliasing problems previously. https://github.com/llvm/llvm-project/issues/45725

Once the GCC work is closer to finished, I expect we'll see the same there.

I disagree that you need to solve the C problem first to solve the compiler problem, and I think it was misguided to start there. You seem to have focused on the fact that exposure is even an option in Aria's implementation, but let me quote: "The goal of the Strict Provenance experiment is to determine whether it is possible to use Rust without expose_addr and from_exposed_addr". Setting PNVI-ae-udi as "the" provenance rule is your end goal for C, but there's a reason it's called the "Strict Provenance" experiment in Rust, the goal is something like what you call PNVI-plain.

APIs like map are key to that goal, Rust has them and N3005 does not.

So like I said, rather than just being a "translation" I think the most that can be said is you've got the same problem albeit in a very different context, and your solutions are related in the way we'd expect when competent people attack similar problems.

We have no "end goal". PVNI-ae-udi is intended to capture the semantics of most existing C code. I mention terminology such as "exposure" etc. just because this makes it obvious that this work builds on C's provenance model (the terminology did not exist before as far as I know). PVNI plain is a stricter subset. You can already use it in C and a lot of C code would just work fine with it. Note that some compilers people and formal semantics people were pushing for PVI.

If you think that "map_addr" etc. is an important API, then I agree that this is an innovation on top of what we did. But I personally do not quite see the importance of this API. Yes, it allows some things in the scope of strict provenance which in C would now require PVNI-ae-udi. We envisioned future extensions that prevent exposure of pointers for certain operations, but somehow this seems more academic at this point in time. If you are not using hardware such as CHERI, this does not matter. On the other hand, PVNI-ae-udi makes most existing C code follow a precise provenance model, which I think is a huge step forward.

Reply on news.ycombinator.com