A pair of Rust kernel modules | Hacker News Reader

A pair of Rust kernel modules

lwn.net

123 pointssigna1130 comments

Asahi Lina has been developing in Rust a new Linux driver to support Apple M1/M2 GPUs for Asahi Linux … almost completely during 12 hour+ livestreams. Expecting to see triangles rendered late tonight (California time). (Her first Rust code btw)

https://www.youtube.com/watch?v=Jjut6E31dBY

She mentions some challenges with Rust here:

https://twitter.com/LinaAsahi/status/1570119306461204481

>Her first Rust code btw

>She mentions some challenges with Rust here:

Did not expect a 100 line macro to hack on placement new onto Rust.

Yea, that’s really impressive.

Rust really doesn't have placement new? Sounds…sus.

I'm no C++ god, but I use placement new regularly. It's a standard, normal, typical feature in C++ development.

You can absolutely use local allocators in Rust, but the APIs around allocation are rough around the edges. Rust 1.0 shipped in 2015, it's certainly no where near as mature as C++ (though it's gaining ground quickly, and can obviously benefit from lessons learned in C++).

Also note that Rust's approach is to have many things that would be core/stdlib features in other languages implemented in community supported packages, where a consensus can form safely without risking shipping a language feature that might not serve the needs of the community, and the package can evolve on it's own schedule & not be tied to a compiler release schedule. On a cursory glance, there are numerous libraries providing this functionality: https://crates.io/search?q=placement%20new

I'm curious, are you a game developer or something? I've written less than 1k lines of C++, so I really don't know much about it, but I was under the impression this was a fairly niche feature.

(ETA: I peaked at your profile & saw it looks like you work at a brokerage - yeah, that makes sense why you'd care deeply about avoiding allocations, fair enough.)

Placement new is kind of niche but it's not as rare as you might think, especially if you consider cases where you're using it through some other abstraction. For example, in C++ std::shared_ptr is implemented as the the object you're pointing to, plus another data structure called a control block that contains the reference count for the pointed-to data. The control block has to be heap allocated. If you're allocating a shared object, you could allocate the object being pointed to, and then also allocate the control block... but that requires two allocations. That's why the standard library has std::make_shared, which does a single allocation for both the object being pointed to as well as the control block (there's also a secondary benefit which is that the control block is adjacent in memory to the data it points to, which improves memory locality and caching). To implement std::make_shared you need to use placement new, as you're doing one allocation and two initializations.

I write C++ as my day job and while I wouldn't say I have to write placement new code often (I don't think I've used it once in the past year), it's definitely an essential tool for writing certain abstractions.

For sure. That's interesting, thank you for sharing. In Rust an Arc<T>, the equivalent to shared_ptr, would also require a single contiguous allocation, because it's size is the size of T plus the size of the Arc's internal state. But I must be missing something because C++ has had templates since more then a decade before Rust 1.0 released. I suspect the difference is in how object lifetimes are handled & Rust's lack of a language-level concept of a constructor or a new keyword.

Skimming the source code for Arc<T>, it looks like T will likely be on the stack & copied to the heap, adding a copy C++ is probably able to avoid with it's strategy, but I'm not sure if the compiler is able to optimize that out or not.

> I suspect the difference is in how object lifetimes are handled & Rust's lack of a language-level concept of a constructor or a new keyword.

Object lifetimes certainly play a role. The data structure used by the shared pointer can be kept alive by a weak pointer, but you want to call the destructor of the embedded object when no more shared pointers point to it. If you just used a templated member the auto generated cleanup code would try to call the destructor when the data structure itself is disposed. That is too late if weak_ptrs where involved and you can't call the destructor manually before that without causing loads of FUN as the automated cleanup will try to destroy the object again later. With placement new the member object can be created and destroyed as needed without the language imposing any assumptions on the objects lifetime.

There's an open RFC for a version of it: https://github.com/rust-lang/rfcs/pull/2884

The proposal has received a lot of new discussion in the past few days, as in addition to being wanted by kernel developers, it's a potential path to support dynamic dispatch in `async` code.

Rust can do it with "unsafe" pointers, but the safe subset of the language lacks ergonomic and foolproof interface for it. Ability to witness uninitialized memory is "unsafe" in Rust. A safe mutable reference is required to point to valid initialized object at all times, no cheating.

In practice Rust usually gets away with not having placement new and letting LLVM eliminate copies.

That macro reminds me a lot of the macros of old times from before the availability of proc macros, which would manually parse Rust code to rely on their features. It's probably done that way because they don't want proc macros to be used for the Rust code. I guess this is because of the fear of the heavy dependency tree of stuff like syn, but you don't even need syn for simple macros, you can also parse them manually instead.

so, maybe this is a dumb question, but is Asahi Lina just marcan with a pitch modulated voice and an anime avatar?

I believe it's someone else, but I don't think she's been explicit about the connection.

It might be a bit different across the community, but it's usually considered polite not to "dox" (however obvious in some cases) VTubers who have public personas under a different name. There are only so many Asahi developers though.

Assuming they want to upstream their changes, that kind of runs in conflict with the Linux kernel community's "contribute under your real name" policy. Of course, given the general queering of hacker and open source culture we've seen over the past half-decade or more, it may be that demands for use of "real" (legal/dead) names is now considered outdated and marginalizing.

They might end up offering the changes for merge under a name other than "Asahi Lina". Who knows? I don't see any particular conflict with the immediate development happening under their VTuber identity, but I'm not much of a kernel hacker, either.

At any rate, it's kind of their problem to worry about, if they even choose to worry about it. I think the work stands on its own merits.

The work is good, but they need to submit it under a real name. The Linux kernel will lose trust if anonymous contributors with hidden interest start contributing code to it, regardless of its quality.

Not really, though.

You can judge the quality of a contribution by looking at the code. Who wrote it is completely irrelevant. The trust in the Linux kernel never came from knowing the names of all the people that contributed to it.

Is that a new(-ish) policy because I can swear that I've seen a few l33t handles among the contributors? I haven't really done any kernel development for a decade, so I'm not following it closely.

It shouldn't be a problem anyway, because someone else can contribute it, it's not like it's that one guy sending the pull requests who writes all of AMD's giant blob of GPU code.

That's what I thought after the first stream since it was April 1st. But after a while, I don't think so. Then again, who cares - let's enjoy the show :-) For the kernel contribution issue, we'll see. If you can reverse engineer a GPU, you can figure out a solution to that issue too.

IIUC Linux upstream requires contributions have the real name of the author in the signed-off-by, and Asahi Lina seems a lot like a pseudonym (otherwise a very extremely fortunate coincidence as a name for someone working on Asahi Linux), so I wonder if the patches will get a new signed-off-by before going upstream or if they’ll be able to be merged as is.

https://www.kernel.org/doc/html/latest/process/submitting-pa...

I remember seeing patches from Solar Designer so I wonder when the policy changed.

I never thought I'd see vtuber doing a live stream coding in a leading edge language...I think if my professors were vtubers, I would have done much better.

Why do you think you would have done much better? I find the format distracting and the speech sometimes inaudible from the post-processing.

Does Rust have a special interest group that's working with the kernel engineers to figure out how best to go about this integration, or is it only kernel devs that are working on this?

I'd like to read more about what the Rust people are working on to help this effort, and if there's any particular nightly feature the kernel requires that will soon reach stable for everybody's benefit.

Rust for Linux tracks the list of "Required" / "Good to have" features from Rust itself. The set of things from that first category which aren't in Stable Rust has been shrinking but isn't zero.

https://github.com/Rust-for-Linux/linux/issues/2

I don't believe there is a "Special interest group" but since the only comparable team to Rust for Linux would be a hypothetical "NT kernel with Rust" group, to some extent it's hard to argue with what they say they need when they come to Rust teams, e.g. library. Of course just because Rust for Linux wants X doesn't magically mean X is the top priority for the people working in that area, or even in some cases that Rust is going to ever do X. Sometimes the answer will be, as it has been for alloc, well, too bad then, make your own.

Linus really doesn't like implicit allocation, this is one thing he didn't like about C++. But Rust's own alloc has lots of implicit allocation. For example its String can be appended to, even using an operator like in Java or C++ which means the heap allocation just grows. But what if there's no more RAM? So, in Rust for Linux such features don't exist.

In both your desktop Rust and Rust for Linux, we can have a variable of type String, like &str it's UTF-8 encoded text, but unlike &str we can have mutable Strings. In desktop Rust you can just write my_string += " and then we ate a whole cake,"; and Rust will grow the String if necessary to fit on this extra text. But that's an implied allocation, so in Rust for Linux, this won't compile, the Rust for Linux String type doesn't have that += operator.

Whoa, after about 7 years of writing Rust, I only now learned that Rust's strings can be extended with `+=`.

In any case, a Rust string is inherently a growable data structure. It has a capacity and a size. If the size exceeds the capacity, it is reallocated.

Does Rust for Linux have "mystring.push_str(" and then we ate a whole cake,")"? If yes, why is "+=" implicit and "push_str" is not?

Rust's standard library String implements AddAssign, the effect of which is that the += operator exists, in effect a += b is sugar for AddAssign::add_assign(&mut a, b)

Personally I would rather it didn't, but the use of + as a concatenate operator is widespread and I imagine if String didn't do this a popular BetterString would.

No, the Rust for Linux String doesn't have push_str(), see for yourself: https://rust-for-linux.github.io/docs/alloc/string/struct.St... that String also has no way to do infallible reserve(), and doesn't implement Write among other differences.

10 days ago: https://news.ycombinator.com/item?id=32822082

I am mote excited about the 9P server than Rust kernel modules.

Reply on news.ycombinator.com