Do we need a “Rust Standard”?
blog.m-ou.se
blog.m-ou.se
I don't think Rust needs a standard. I don't think it needs a stable ABI. I don't think it needs a second or third implementation. My argument is pretty simple: there are dozens of other language projects that have none of these things and have seen a lot of success. Yet these criticisms are rare of other languages - I don't think I've ever seen someone lament that Swift or Go or Typescript or Dart or $ANY_LANG don't have an ISO spec. That doesn't mean myself and others can't see the value in these features but it does seem to be a weird critique.
The better question is to state more explicitly the preconditions: C and C++ have a specification. Does Rust need one before it can replace C and C++ in projects? And I think the answer is "it depends but probably not for most projects."
Rust's current reference docs, more than seven years after Rust 1.0 was released, are much weaker than the first version of the Java specification (which was published seven _months_ after JDK 1.0).
The place where it seems to me that there's a strange inconsistency is inside the Rust community: when discussing package management or compiler error messages or tutorial documentation or pretty much anything else I can think of, the there seems to be general agreement that Rust should aim to be best in class.
But when it comes to complete-and-correct documentation, I hear voices saying "are we sure we need this?", and "we're no worse than C was in 1985", and "we shouldn't let writing documentation slow down releasing new features".
Topic matters. Take the Guaranteed Niche Optimisation. Many users rely on Rust's promise that Option<&T> is the same size as &T for example, that's the "Guaranteed Niche Optimisation" (the niche in &T is that it can't be a NULL pointer) If the specification didn't mention this at all, or said eh, we can't promise anything, that would be a problem. I've no doubt serious Rust users would demand this get written down properly.
But the exact details of the Guarantee aren't so important. Far more people care that Option<&T> is the same size as &T, than care about, say, Option<NonZeroU32> let alone more obscure types that are in practice optimised. If it turned out the guarantee was narrower than you expected, but only for some type you never use, is that important?
What if Option<Infallible> wasn't a ZST due to a compiler bug? I'd guess Rust internal tests would notice, but if not I'm not sure how much real software would care.
If somebody can spend next week either carefully documenting exactly what the Guaranteed Niche Optimisation is, all the fine details, or implementing a cute niche optimiser trick which turns out to make the most popular JSON parser use 1% less RAM and 5% less CPU, the latter is just clearly the better deal.
To the extent that people are relying on the common cases of the niche optimisation taking place: yes, the Rust description should be documenting what people may rely on, and should have been documenting it for years.
In order to do that there's absolutely no need to describe the fine details of exactly where the optimisation currently applies.
There are many random details of a language that are "important" and they get documented all over the place. If it was all under "THE REFERENCE" and was a 500 page PDF, would the information actually be more discoverable? How would you even know to look up what you don't know? Are you going to read the reference cover to cover before writing hello world?
But those thousands of people, however enthusiastic, can't produce authoritative documentation, because that needs the maintainers' "sign-off".
I'm speaking out of ignorance here but, is there a chance of overflowing the stack of a small microcontroller even by one byte with this? Or other side effects on platforms that are not even remotely capable of parsing JSON?
If that's the case and my ECU crashes and people die, should I rely on the fact that an user on Github promised me that the code generated should be correct?
Who ensures this?
Under these conditions, how Ferrocene or the next Greenhills can achieve certifications?
Shouldn't be an authoritative document that says "'this thing' must have 'this size'", or "depends on implementation", "depends on platform" or even "behavior is undefined"?
Maybe? If Option<Infallible> was mistakenly not a ZST I guess it might cost one byte to store the None/ Some discriminator (which would always be None in this example).
> Shouldn't be an authoritative document that says "'this thing' must have 'this size'", or "depends on implementation", "depends on platform" or even "behavior is undefined"?
Well, wait, the current situation is that if the specification doesn't say, Rust isn't promising what size it is. It does say that Option<&T> is the same size as &T, and various other useful things, including an outline of the Guaranteed Niche Optimization, but there are definitely edge cases that are hard to reason about from the human text and it's reasonable to say these are not specified.
I think if your ECU kills people if the size is wrong, you should probably have demanded more than "It wasn't clear in the specification so I asked on GitHub" and so should your regulator if you're in a regulated industry.
If you're happy with "it depends" as an answer then congratulations, Rust can meet that today. But then it seems like you should be more careful with that ECU. I recommend adding hardware failsafe systems so that when, not if, you screw up, you don't kill anybody.
If this is true then the spec should say "using Option<&T> in certain cases carries allocations of undefined size".
Other than forcing me to not use Option<&T> in my embedded systems (which IIRC is widely used), what is the solution? MISRA Rust specs mandating "NO Option<&T>"?
Saying "You used Option<&T> with an edge case. The compiler is not consistent with edge cases sometimes. Don't do that" is shifting the blame on the programmer once again and this is bad for Rust for the reasons everybody knows.
Or I am misintepreting "edge cases that are hard to reason about from the human text".
> you should probably have demanded more
Demanded who? how?
See how difficult is to integrate Rust in an automotive/industrial company like the one I work for?
Who should I sue if I found code generated "out of specs" (specs which don't exist in the first place)?
> ... and so should your regulator if you're in a regulated industry.
But this can't happen right now. I guess regulators will want to rely on ISO or other formal specs for language specifications.
And where is this so-called specification? Do you have a link? Where can I get a hard copy?
Is it this? https://doc.rust-lang.org/reference/
You've smooshed together unrelated phrases with an ellipsis. You're unlikely to learn anything by doing that whether from a specification or really even basic API documentation.
> Or I am misintepreting "edge cases that are hard to reason about from the human text".
Given it isn't anywhere close to the phrase you've decided it's about yes, I'd generously say you are "misinterpreting" it.
> Demanded who? how?
You can explicitly assert claims about the world in your code if you believe they are true but there seems not to be official specifications saying so. Then the code doesn't compile if the assertions are wrong.
In some cases you may find the Rust language team can explicitly clarify something which troubles you, especially if you believe it isn't documented and yet would concern other people too.
And in many cases if you can't see why X is true, you should stop assuming X is true in your safety critical systems.
> Where can I get a hard copy?
This is especially hilarious when people ramble on about ISO because they're always imagining there actually is a "hard copy" of the modern standards. ISO tries really hard to persuade you give them a lot of money to download a PDF of for example 14882:2020 (C++) but you may be able to find a local standards agency who insist they actually have bound copies they can post to you for $$$.
In practice you still won't get a book. Such outfits tend to have a Print On Demand service and ISO 14882 is about 2000 pages so their POD system will reject the print. Once you put your order in, and wait a few days or weeks, either you get a refund and an apology or you'll receive... a CD with the PDF on it. If you have a big University or other technical library nearby they might have an earlier ISO version e.g. C++ 98 and C89 are things some really big libraries actually bought on paper. The standards were smaller and fewer people used PDF readers back then. But they won't have the current versions because it's a waste of money and paper.
Is "Option<&T>" possibly overflowing the stack by at least one byte? You said "maybe?".
And also you said: "the specification (or Rust?) it does say that Option<&T> is the same size as &T".
So???
Is "Option<&T> is the same size as &T" ALWAYS or not???
If it's not, then what I said in my previous comment applies. Otherwise, please be clearer.
> You can explicitly assert claims about the world in your code if you believe they are true but there seems not to be official specifications saying so. Then the code doesn't compile if the assertions are wrong.
"if you believe they are true" is kind-of shifting the blame on me.
> In some cases you may find the Rust language team can explicitly clarify something which troubles you, especially if you believe it isn't documented and yet would concern other people too.
Yes, this is the Github user(s) I mentioned earlier. Is not good enough for an automotive/aerospace/industrial/railway/white goods/etc industry.
> This is especially hilarious
Fine. Where can I get the official PDF with the Rust specifications?
No, although I see where you maybe got confused. I had proposed - as an example of a hypothetical Rust compiler bug - a case where Option<Infallible> might mistakenly not be a ZST even though that's what you'd expect.
It is apparent that you weren't really following, so lets explain a little slower, this will take a few paragraphs and it may require a little extra mental flexibility: In C++ your types all have minimum size 1 byte, Rust goes two sizes smaller.
Consider Rust's generic TryInto trait and its try_into() function:
a = b.try_into().expect("We give up, it's impossible");
Now, what if we tried to turn an unsigned 16-bit integer into a 32-bit unsigned integer? Well that can't fail. We could have the same arrangement, but it's wasteful because there are actually no errors. So we use Infallible here, an Empty Type, making a return type of Result<u32, Infallible>. No values of the Infallible type exist, signifying that although this is a generic function, and other functions of the same kind might fail, this one just cannot.
This Infallible type is pretty useful for such things. But although it's pretty special it is just a type. What happens if we wrap it in Option<> ? Option<Infallible> only has one possible value: None. The Some kind of Option isn't possible because there are no values of type Infallible, and the Some kind needs a value inside it.
In Rust, types with only one possible value are called ZSTs: Zero Size Types. Unlike the Empty type they can exist - there is a value for them, but only one value, so there's no need to store it anywhere, it has Zero Size.
So in my hypothetical the Rust compiler is erroneously not treating Option<Infallible> as a ZST. That seems very unlikely, feels like it just naturally falls out of how the types are defined - but I don't know for sure that the specification says it mustn't happen.
In contrast Rust does specify that Option<&T> and &T are the same size. If that stopped working that would be a major compiler bug, (and it would set off like a bajillion alarms if it broke).
I now realise maybe you were so unfamiliar with Rust that you thought Option<Infallible> was somehow an example of Option<&T>. It isn't at all, and if I'd realised that had confused you I'd have chosen something different. Option<&T> is in human words "Maybe this is a reference to a T, but maybe not", whereas Option<Infallible> is "Maybe this is a failure which can't happen, or maybe not".
> Yes, this is the Github user(s) I mentioned earlier. Is not good enough for an automotive/aerospace/industrial/railway/white goods/etc industry.
I don't believe you that somehow the possibly erroneous text in an ISO document trumps reality, whereas the same isn't true for @m_ou_se or other Rust people's documentation. These are both just words, and the ISO document isn't even describing a concrete system which actually exists. All actual C++ compilers have various deviations from the document, on top of which they have bugs too.
Rust's documentation is taken very seriously and often praised! But we focus on documenting actual APIs (with examples that we actually run in CI to make sure they don't break!) (and links to the actual implementation to dig deeper!).
We also gave everyone builtin tools to write their own docs to the same standard and automatically build and host them. Example: https://docs.rs/tracing/0.1.37/tracing/
What Rust has eternally lagged on is properly documenting little fiddly nubs like "this is an integer literal expression which decays..." and "the grammar for a pattern expression is...".
For whatever reason some people think these details are of THE UTMOST IMPORTANCE, instead of just being a thing you try out and see what happens. Or a thing you discover by reading existing code, books, examples, posts, etc.
As a language Rust is extremely tuned for "fuck around and find out". I can rattle off some spooky interactions in Rust but I always end up concluding with "... and in practice this doesn't really matter because if you run afoul of it you just get a compilation error (or a lint)".
In this regard knowing fiddly details about Rust... tends to not matter. Unless you're writing Unsafe Code. Then you read the book I wrote, and eventually you drill down the rabbit hole deep enough to learn that "oh actually C doesn't even know what this means. UHHH... WELL. FINE?"
But it often feels as if the Rust community doesn't believe that people who prefer to learn by reading accurate documentation exist.
I commend the following page to you to get an idea of the mindset involved: https://blog.nelhage.com/post/computers-can-be-understood/
(No doubt it will gladden your heart to know that I updated the Reference to document the somewhat surprising ways in which integer literal expressions decay earlier this year.)
Your confusing two radically different types of documentation.
Specifications, such as the ISO standards for C and C++, are the source of truth of what determines what the language actually is.
Writing tutorials with example code ran in some implementation might help people onboard onto it but it does not, in any way, shape, or form, specify what's the language.
It's like comparing blueprints of a construction site with random videos and photos of the same building.
A detailed specification only matters if you're trying to construct another building to the same spec (write another compiler), want to do heavy renovation work (implement a major new feature on rustc), or if the technical details are insufficiently "abstracted out" (say, there are exposed live wires that you have to know to avoid in daily use – cf. undefined behavior in C and C++).
My point is that this line of argument makes no sense.
It's like claiming that dictionaries and English grammar books are not needed because kids learning their ABCs almost never require reading those to learn how to speak English.
Rust standardization opponents insist in these moot strawmen arguments. They completely miss the whole point of a standard. Standards are critical because how things work need to be specified in precise terms that are set in stone. Standards are read to gain deeper understanding of a language. They are used to clarify corner cases and obscure issues and behavior. You do not need a ISO standard because you want to write a "hello world". You need an ISO standard because you are a mature professional that understands that "it works on my machine" is not an acceptable answer to any question you have on how a programming language supposedly works.
I don't think this makes the case you were hoping for?
Both Dictionaries and actual "English grammar books" (not pedantry written by non-experts like Lynne Truss) are descriptive because English is a natural language. I don't own the full Cambridge Grammar of the English Language but here's an extract from the Student's Introduction which I do own. (The Introduction is intended to be suitable as an undergraduate text)
"... it isn't sensible to call a construction grammatically incorrect when people whose status as fully competent speakers of the standard language is unassailable use it nearly all the time".
We don't use these books to teach children or ESL (English as a 2nd language) learners. My employer runs "pre-sessionals" which are short language courses for overseas students, they have a conditional visa, they fly in a month or two before term starts and they take our intensive ESL courses, if they do OK they're signed off as able to successfully use English, they start their degree course (say, Electronics) and the visa conditions check off, if they can't hack it the visa conditions terminate their stay and they are sent back. This allows us to recruit foreign students who could, if they apply themselves, be successful - without them needing to somehow find local English tutors in their home country, squeeze in learning a foreign language and convince immigration they've learned enough English to follow their degree course.
It is not the goal of pre-sessionals to teach a Chinese teenager to say "It is I" rather than "It's me" because that's a weird hyper-formal English. It is the goal that they should understand phrases their tutors and peers might use, which are often going to be informal or even somewhat non-standard, and be able to express themselves in both formal and informal contexts. They should be able to read say an Economist article, write a 1000 word summary of the article and talk about it in conversation.
Books like CGEL and dictionaries represent our understanding of how a very complicated system is actually used by real people. They are not guides to how English should work, such a document would be very silly.
> Standards are critical because how things work need to be specified in precise terms that are set in stone.
This is a mythologized version of what's going on. If a language is spending huge amounts of money and personal effort on the ISO process I can understand they'd want to project this myth, but it isn't true.
For C++ in particular - the most obvious comparison from the ISO standard languages to Rust - huge swathes of important stuff is "ill-formed, no diagnostic required" in the standard. If your C++ program falls foul of any such clauses, anywhere, even once in some obscure sub-routine you don't use, too bad your entire C++ program has no defined meaning and the compiler is not expected to report this as an error or warn you about the trouble you're in.
Let's use my favourite modern example. Suppose we've got a std::vector of float, and we sort that using std::sort but in some cases, due to an oversight, one element of the vector can be NaN. In C++ 20 Our entire program is undefined and anything might happen because of clauses introduced by the C++ 20 Concepts feature. Our program probably works fine in practice, but no thanks to your "critical" standard.
Yes. And the issue is, that's not an acceptable stance for a language seeking to be a systems programming language.
But that's not what a language specification looks like. You will rarely turn to the ISO spec of C or C++ to learn something about the language unless you are trying to build a greenfield compiler. And while a lot of people want a second implementation of Rust for technical reasons, it's not as compelling as improving language documentation.
For example language documentation would look like: the `for x in y {}` syntax is called a "for" loop, which iterates over the expression `y` by calling the `next` method until it returns `None`. (this is bad, but you get the idea)
Language specification would be something like `for x in y { body }` is equivalent to
let y = y.into_iter();
loop {
match y.next() {
Some(x) => { body },
None => break;
}
}
Meanwhile, for unsafe Rust, if you look at the Rust Reference Book, there is a single page that lists the possible situations for undefined behavior, with an added warning message:
“Warning: The following list is not exhaustive. There is no formal model of Rust's semantics for what is and is not allowed in unsafe code, so there may be more behavior considered unsafe. The following list is just what we know for sure is undefined behavior. Please read the Rustonomicon before writing unsafe code.”
Then when I look at the Rustonomicon, it says:
“Warning: This book is incomplete. Documenting everything and rewriting outdated parts take a while. See the issue tracker to check what's missing/outdated, and if there are any mistakes or ideas that haven't been reported, feel free to open a new issue there.”
So maybe that should be more prioritized if Rust really wants to get the reputation of a safe, stable language. (Safe Rust is good, but Unsafe is where the real devils are all hidden…)
It's not. Rust lags way behind the standards help for decades by languages which the Rust community repeatedly claims Rust is supposed to replace. Rust has in no way, shape or form anything resembling an effort to specify it's core language and standard library in an international standard. Standards are critical because they are objective, clear, and well thought through specifications of how the language works now and anywhere, including and specially clean room implementations.
> I don't think I've ever seen someone lament that Swift or Go or Typescript or Dart or $ANY_LANG don't have an ISO spec.
Most of the top programming languages already are specified in international standards. Including JavaScript.
https://en.wikipedia.org/wiki/Category:Programming_languages...
Also, all of your examples, arguably with the exception of Go, are still in very early and unstable development stages, with projects repeatedly introducing breaking changes. If even the language designers don't make up their minds over what goes into it then we can't really talk about stability.
> Most of the top programming languages already are specified in international standards. Including JavaScript.
> https://en.wikipedia.org/wiki/Category:Programming_languages...
> Also, all of your examples, arguably with the exception of Go, are still in very early and unstable development stages, with projects repeatedly introducing breaking changes. If even the language designers don't make up their minds over what goes into it then we can't really talk about stability.
Java, Python, and PHP are all widely used and do not have ISO standards.
One thing I think is worth more reflection is: Are there specific things such a Specification must cover, or is it enough, as it seems C and C++ felt, to just specify some of the language and leave the rest unacknowledged ?
For example, ANSI and ISO C significantly pre-date the memory model we're all now comfortable C implements. They don't offer a different memory model, the specification just never mentions what's going on at all in this respect. If you wrote concurrent software in C in, say, 1996 (which I did) then eh, it does whatever it is that it happens to do. The ISO standard document has no view.
Right now, Rust "pretty blatantly just inherits the memory model for atomics from C++20. This is not due to this model being particularly excellent or easy to understand. Indeed, this model is quite complex and known to have several flaws. Rather, it is a pragmatic concession" (quoting the Nomicon). Is that good enough? If Aria's Strict Provenance experiment goes well, how much of that should be specified? As much as possible? As little as possible?
I think a spec is different in a way that allows it to document expected behaviour vs implemented behaviour
But you have a point that a traditional spec can underspecify behaviour in a way that is not possible in Rust itself. But I content that underspecified behavior in a programming language should be avoided.
In practice, for a programming language, even something relatively simple like C, the specification is too woolly to be enough, and so the fact of what compilers actually do matters. So Rust having a specification would not magically prevent this problem.
Rust's Crater runs allow sunlight from the real world to intrude into decision making. Suppose it is discovered that although the docs say X doesn't happen, actually the compiler does X. Instead of "I bet lots of people rely on X, we must change the specs to match" versus "I bet everybody trusted our spec, which says X doesn't happen, fix the compiler" we have real facts. "Popular crates cool_stuff and jims_other_stuff both rely on X in versions from the last 18 months. Their authors both said they tried this and assumed the documentation was wrong. We should update the documentation to match reality" vs "There's only one crate which uses X, it's a test case in a sub-component of obscure_doodad 0.0.1 from 2017, it appears to receive no usage, and two of the other test cases in obscure_doodad fail on current Rust, so perhaps it is just broken anyway? We should just fix the compiler"
The C++ people couldn't have done this twenty years ago. Today they probably could attempt this with various Free Software C++ codebases out there (e.g. GitHub's C++ projects), but as well as considerable technical effort to attempt this, it would be a political burden because lots of non-Free stuff believes it has secret sauce and so vague and unsourced claims about what does or doesn't exist in proprietary C++ are a big problem for compatibility. If you're on team "Don't change X" of course you'll claim huge industrial partners you've seen rely on X in code you can't show anybody. If you're on team "X is obsolete garbage no-uses" you'll claim it's apparent nobody relies on X.
This is much less true for Rust. There obviously is proprietary Rust, including in-house projects and stuff that's deliberately proprietary rather than just nobody cared, but Crate covers a lot of what's out there.
If Rust wants to supplant C in embedded, a specification is required. I'm honestly surprised Linux is accepting unspecified code into the kernel with the addition of Rust.
https://lwn.net/Articles/718628/
The Linux model and the C++ 11 (as subsequently modified) model are similar in the sense that they respect how the CPU actually works, necessarily in the case of Linux and for practical low level C or C++, but in details they differ, sometimes for good reason and sometimes just because Linux was here first.
Red herring. If I recall correctly, Linus publicly stated a number of times that C's memory model is relevant for userspace code but OS development has its own requirements.
Amusingly enough, I recall that this discussion was about Rust, and the way that Rust proponents advocated pushing Rust's memory model onto Linux.
No, there's will still only be one official compiler – the one written and provided by The Rust Project.
> If Rust wants to supplant C in embedded, a specification is required.
You're getting your history backwards.
C did not become successful because of its specification, it got a specification because it was successful.
Similarly, if Rust becomes successful enough, it will probably get a specification. But the popularity will drive the work on a specification, not the other way around.
I disagree. The need for a standard specification of what the C programming language is sprung from the need to ensure that multiple independent compiler implementations were actually compatible and supported the same sets of features.
If a nonstandard C ecosystem had to endure the pain of either targeting products from specific vendors or waste time maintaining a caniuse database of C features, the amount of wasted effort to just keep code building would make things too challenging and troublesome to maintain. Consequently, we would see gradual fragmentation both within the C-like ecosystem and in other potential C-like successors that ink offered minor changes.