Rust solves the problem of incomplete Kernel Linux API docs
vt.social
vt.social
They remove a lot of ambiguity in how something should be held.
Is this data type or method thread safe? Well I don’t need to go look up the docs only to find it’s not mentioned anywhere but in some community discussion. The compiler tells me.
Reviewing code? I don’t need to verify every use of a pointer is safe because the code tells me itself at that exact local point.
This isn’t unique to the Linux kernel. This is every codebase that doesn’t use a memory safe language.
With memory safe languages you can focus so much more on the implementation of your business logic than making sure all your codebases invariants are in your head at a given time.
To a first approximation, Arc<Mutex<T>> is what you want when you want a blob of mutable stuff to just work between threads, but you still have to worry about terrible performance if you hold the mutexes too long, etc. (Not to mention mutexes are just slower in general than having types which are natively Send+Sync.)
Without knowing the implementation, how would you know this statement is true?
I could just stick the Arc<Mutex<T>> within my abstraction and say it is send+sync and it would have the exact same performance, and that might be the most efficient solution possible on the given hardware.
A race condition (which safe Rust can have) is a normal phenemenon in our world. If we put the cat out the front door, then walk to the kitchen and close that door by the time we reached the back door maybe the cat had run around the outside of the building and come inside again, closing the doors in the wrong order introduced the opportunity for a race - be very careful.
A data race (which safe Rust does not have) is a weird thing caused by the mismatch between how you think the computer works and how it actually works. What happens if Alice tries the put the cat out the front door at the same time Bob tries to put it out the back door? Well, in the real world that cannot happen, either Alice has the cat or Bob does. But for many languages† this sort of nonsense can happen and when it does that's a disaster
† In safe Rust it can't happen, in Java, OCaml and Go it can happen, but it may not be a disaster (the details are different for each, it is a bug though).
https://doc.rust-lang.org/nomicon/send-and-sync.html
So therefore a function can’t do things that invalidate that type level contract.
Many of the same problems — the mentioned underdocumented API assumptions — exist in memory-safe languages like Java and C#. Memory safety is just one among many aspects in interface contracts, and is actually often one of the better documented ones in non-memory-safe languages.
Type-based checking is of course very beneficial, but TFA specifically focuses on the documentation aspect, not on the enforcement, which are in principle orthogonal to each other. TFA draws attention to the fact that the enforcement here also entails better self-documentation. However, it’s limited to whatever aspects the type system enforces.
If you’re pedantically holding on to one sentence as a gotcha, because I didn’t caveat everything, then fine. But seems a bit much to then say I’m forcing a definition just because you ignore the rest.
Yes I know it’s "faster-development" to have type instead of additional definition attributes, but I think the problem would be also solved.
If you have code that deals e.g. with pounds and kilograms, Dollars and Euros, screen coordinates and window coordinates, plain text and HTML and so on, those values are usually encapsulated in safe wrapper structs instead of being passed as raw ints or floats.
This prevents you from accidentally passing the wrong kind of value into a function, and potentially blowing up your $125 million spacecraft[1].
I also find that such wrappers also make the code far more readable, as there's no confusion exactly what kind of value is expected.
[1] https://www.simscale.com/blog/nasa-mars-climate-orbiter-metr...
But both the mp-units and this lighter weight Rust approach would at least make you write that down in the software. There's some chance the mistake is realised, whereas when it's just an organisational assumption that just doesn't happen.
Serialize a dict containing a value with uncertainties and/or Pint (or astropy.units) and complex values to JSON, then read it from JSON back to the same types. Handle datetimes, complex values, and categoricals
"CSVW: CSV on the Web" https://github.com/jazzband/tablib/issues/305
7 Columnar metadata header rows for a CSV: column label, property URI, datatype, quantity/unit, accuracy, precision, significant figures https://wrdrd.github.io/docs/consulting/linkedreproducibilit...
CSV on the Web: A Primer > 6. Advanced Use > 6.1 How do you support units of measure? https://www.w3.org/TR/tabular-data-primer/#units-of-measure
You can specify units in a CSVW file with QUDT, an RDFS schema and vocabulary for Quantities, Units, Dimensions, and Types
Schema.org has StructuredValue and rdfs:subPropertyOf like QuantitativeValue and QuantitativeValueDistribution: https://schema.org/StructuredValue
There are linked data schema for units, and there are various in-code in-RAM typed primitive and compound type serialization libraries for various programming languages; but they're not integrated, so we're unable to share data with units between apps and fall back to CSVW.
There are zero-copy solutions for sharing variables between cells of e.g. a polyglot notebook with input cells written in more than one programing language without reshaping or serializing.
You can do this in other languages such as Pascal with subtypes and something functionally equivalent in C# and other object oriented languages.
There is a space of all possible programs that can be written. Every time you add a restriction like the borrow checker, more expressive data types, or immutability, you are shrinking the space of all possible programs you can write. Sure, some of those programs are correct but 99.9999% repeating are incorrect.
WUFFS is an example of this, the WUFFS JPEG decoder decodes JPEGs, duh, but even if you deliberately sabotaged it, your sabotaged decoder just... decodes JPEGs wrong, maybe now they all decode to the Goatse image. It not only can't escape to execute arbitrary code, it can't do much of anything outside of the task we intended, all the resulting runtime bugs (for which you need unit tests) are things like oops the grass is blue, or the decoder rejects this 1x1 image even though it's valid. No "Segfault" no "Ooh it truncated the file instead of reading it", those aren't things we want, so no need to be able to express them.
For example I think a lot of big data work could use such a language. Data goes in, processing happens, data comes out. No "Oops, it wiped the hard disk on nodes 6, 18 and 24, I wonder why". No weird interactions with the OS that weren't catered for, just data in, processing, data out, which is what the people using these things always want.
If any one API of a struct/enum has '&mut self' then its instance cannot be shared across threads without using a mutex (which brings its own friend along i.e Arc). This led me to structure the code in my mind much in advance to prevent borrow checker issues.
Another use of this is while embedding an object in a struct. When I realized that '&mut self' infects everything up the call chain it was a big learning moment. That is the embedding struct also needs a mutable borrow as well.
No, Rust doesn't solve the problem of incomplete API docs. Developers being diligent enough to document the APIs solves the problems, and that solution is entirely language-independent.
The argument is that properly utilising a stronger type system is more self-documenting. And quite often to achieve safe abstractions, you have to lean into this paradigm more. It of course still requires discipline, but in my experience it works well in practice.
Since C has no generics/type templates, you cannot achieve the same level of type based abstractions, thus you cannot rely on the types semantics alone to suggest it's own use.
I'm personally of the opinion that crate author should provide explanations, examples, and take into account the kind of compiler errors people will get when they inevitably try to misuse the APIs.
Sure you can over engineer Rust, for dubious gains. But that's true for any language. *cough* MetaAbstractFactoryProxyFactoryFacadeBuilderProxySingletonWebViewFactory *cough*
> More self-documenting for whom
Anyone using basic Rust types.
Is type `Sync`? It's safe to share between threads.
Is type `Option` its value can be missing i.e. None.
Is type moved? Yes, if it isn't borrowed, and not Copy.
Or a C function that either returns an existing object, or a new object that is incompletely initialized. If it is an existing object, you are required to call a second function to finish the initialization. And you’re just supposed to know about that requirement; it’s not documented anywhere. On the Rust side they decided to return an Either<EmptyObject, InitializedObject> instead, so that the programmer can directly see what is going on.
They’re not saying that all Rust APIs are immediately easy to understand, only that they require less manually–written documentation because the constraints can be encoded into the type system.
https://www.reddit.com/r/programming/comments/1f5qbhv/rust_s...
> This isn't a great title for the submission. Rust doesn't solve incomplete/missing docs in general (that is still a major problem when it comes to things like how subsystems are engineered and designed, and how they're meant to be used, including rules and patterns that are not encodable in the Rust type system and not related to soundness but rather correctness in other ways). What I meant is that kernel docs are specifically very often (almost always) incomplete in ways that relate to lifetimes, safety, borrowing, object states, error handling, optionality, etc., and Rust solves that.
For this I have to bring up that there are indeed rust API's that I have to Google around a bit because I have no way of knowing just by looking at the function signature how to create the type that it expects. Because some types you can't just make from scratch and have to use a combination of other functions to create.
In C, it's all MysteryObject* and you have no clue if that is a Box, Rc/Arc, &mut, Cow, MutexGuard<MysteryObject>, maybe a slice of &[MysteryObject].
And if you're unlucky you get functions taking or returning void*.
https://gist.github.com/shmup/d6290988c876c303663d56563bc570...
This also came up in the video referenced in the step-down announcement, specifically just before Ted jumped in all hot, they were in the middle of about to propose to take some of what had been learned and perhaps rename or even document the C side once it was understood. Amusingly/disappointingly that suggestion which was coming was also counter to what Ted assumed they were saying.
A recent personal experience example down this same path happened in the GSO paths in the network stack, where a well intentioned incremental linux change made its way through review, landed not only in head but also into stable and LTS before anyone noticed that it was broken. It had actually broken kselftests (noted by a maintainer after the report it was broken), but kselftests is quite a mess to use on any given day, so it's hard to "spot" regressions and demonstrably isn't used as a tool to guard against regressions consistently. Reviewers missed the semantics, and i wouldn't blame them because it turns out the semantics in question aren't explicitly or clearly written down anywhere. Last year we wrote against this interface at work, and gained substantial performance improvements, but to do so we had to read the docs, read the source, read the review threads, make assumptions, try stuff, and ultimately settle on the observed semantics more than anything else. This is a userspace interface, so it should be well defined and stable following the mantra, and it's mostly ossified now due to having real users, but it's not well documented or well understood. We follow along some of our fellow offload implementors and have seen a number struggle trying to work solely off of the docs and looking at our implementation, as they miss minor semantics - you can only get those by reading the source and patches very closely. The point of this story is that these problems of understandability cause regressions inside the kernel too, all the time. Reaching for tools to avoid this is a good idea.
This sort of thing also comes up a lot when you write Python extensions in C. You have to know the "calling convention" when you pass object refs in and out of functions. This entire category of problems goes away when you write extensions in Rust with PyO3, and the barrier to entry for new folks is much lower. C++ can do similar things (std::shared_ptr), but it tends to introduce new footguns in a way that Rust really doesn't.
In most languages, you have a lock on the side that you need to remember to take to protect some data. In Rust, the lock wraps the guarded data, you can't access the data without holding the lock, you can't forget to unlock, and you can't keep a reference alive while releasing the lock.
"More readable" is self-documenting! Doesn't matter how you get there (though it's a bummer you've lost faith), just matters that you did.
> No type system will replace docs or even comments. No type signature will give you the why, the big picture or the context.
Types are just one small part of the picture: did you read even the first full message, much less the full post? Are you aware of the other non-type-related benefits Rust provides?
OP helpfully gave an illustrative list I'll reproduce here for you since I'm not sure you saw it:
> When a callback is called are any locks held or do you need to acquire your own? What about free callbacks, are they special? What's the intended locking order? Are there special cases where some operations might take locks in some cases but not others?
> Is a NULL argument allowed and valid usage, or not? What happens to reference counts in the error case? Is a returned ref counted pointer already incremented, or is it an implied borrow from a reference owned by a passed argument?
> Is the return value always a valid pointer? Can it be NULL? Or maybe it's an ERR_PTR? Maybe both? What about pointers returned via indirect arguments, are those cleared to NULL on error or left alone? Is it valid to pass a NULL * if you don't need that return pointer?
Only a few of those are concerned with type safety. You really should read the whole thing.
What bothers me is exactly this attitude that all you need is Rust:
> The solution is called Rust. Encode all the rules in the code and type system once, and never have to worry about them again.
My point is that you can encode all important information in a formal language. It does not solve the documentation problems, it can mitigate them. That's not nitpicking, because I've met plenty of people thinking the former and their code is usually difficult to read as a consequence.