Rust in QEMU Roadmap
lore.kernel.org
lore.kernel.org
(By the way, I did not originally start the project, though I've worked quite a bit on the safe abstractions that are mentioned in the roadmap).
For drivers, it's already happening, especially for graphics but not limited to that. 6.13 has some very important changes. A lot of Linux is drivers so that's already a reason to be bullish.
Answering for QEMU instead: it depends on the community being willing to share the burden of writing the FFI code. Despite Rust being low level, there is still a substantial amount of work to do. Replies to the roadmap pointed out tracepoints as an area where I know nothing and therefore I would like someone else to do the work (I am working mostly on the object and threading models, which is also where a lot of the impedance mismatch between Rust and C lies).
This is probably a little different from tracepoints in the kernel space but I'm somewhat interested in going deeper and into the kernel side of things. Let me know if you have any pointers as to where I might be of concrete assistance to you!
I also found https://github.com/cuviper/probe-rs/tree/master/src/platform which seems interesting.
This crate gives you a nifty macro that translates Rust probe definitions into the correct ELF sections and other related assembly. Other ways to declare probes are also supported, though.
Their support is currently only for illumos and OS X but I managed to get Linux SystemTap SDTs working as well, and a PR for FreeBSD exists as well but has been languishing in review purgatory for two years: I imagine QEMU might enjoy this wide support (if they land).
Pin is what it is, but it is mostly okay since I haven't needed projection so far. Initialization using Linux's "impl PinInit<Self>" approach seems to be working very well in my early experiments, I contributed changes to use the crate without unstable features.
In the FFI area: Bindgen support for toml configuration (https://github.com/rust-lang/rust-bindgen/pull/2917 but it could also be response files on the command line), and easier passing of closures from Rust to C (though I found a very nice way to do it for ZSTs that implement Fn, which is by far the common case).
The "data structure interoperability" part of the roadmap is something I should present to someone in the Rust community for impressions. Some kind of standardization of core FFI traits would be nice.
Outside the Rust core proper, Meson support needs to mature a bit for ease of use, but it is getting there. Linux obviously doesn't need that.
BTW, saw your comment in the dead thread, you're too nice. I have been curious about Rust for some time and with Linux maturing, and Linaro doing the first contribution of build system integration + sample device code, it was time to give it a try.
As far as I know, const traits are still on track.
> easier passing of closures from Rust to C
As in, turning a Rust closure into a C function-pointer-plus-context-parameter?
> The "data structure interoperability" part of the roadmap is something I should present to someone in the Rust community for impressions. Some kind of standardization of core FFI traits would be nice.
Would be happy to work with you to get something on the calendar.
Yes they are. My use case is something like the bitflags crate, there are lots of bit flags in emulated devices of course. In the meanwhile I guess it would be possible to use macros to turn something like "bit_const!(Type:A|B)" to "Type(A.0|B.0)" or something like that.
> As in, turning a Rust closure into a C function-pointer-plus-context-parameter?
Yes, more in general everything related to callbacks is doable but very verbose. We might do (procedural?) macro magic later on to avoid the verbosity but for now I prefer to stick to pure Rust until there's an idea of which patterns recur.
Let me know by email about any occasions to present what I have.
This is unfair to C. With a little work when defining all classes (or non-leaf classes if you're willing for a hairier implementation), you can do this there too.
There are probably other ways, but the way that seems most "obvious" to me is to define the base via union of all bases in the chain, rather than only the immediate base class. So:
/*
class Foo {int f;};
class Bar extends Foo {int b;};
class Qux extends Bar {int q;};
class Leaf extends Qux {int l;};
*/
struct Foo { int f; };
struct Bar { union { struct Foo base, base_Foo}; int b; };
struct Qux { union { struct Bar base, base_Bar; struct Foo base_Foo; }; int q; };
struct Leaf { struct Qux base; int l; }
1. check if we're already the right class.
2. check if `base` is the right class.
3. unconditionally try to use the appropriately-named `base_Foo` member as calculated from the type.
Though I admit that Rust also has to include the full list of base classes (https://github.com/rust-lang/rfcs/pull/1268 would fix it).
(that said, the C approach is also nice for manually accessing fields of distant base classes)
When you later say "you have IsA<Device> for all types, but only if they implement IsA<I2CDevice>" the two conditions conflict.
It might be possible to avoid this issue with a different implementation but this is the simplest one that works and QEMU's hierarchy is generally shallow. If a better implementation came along, it would be a matter of search and repeat.
Can you please show a concrete example? Thanks.
Will it work on most tool chains? Yes. But when it doesn't, it is going to be fun.
This one I wasn't familiar with, and I believe it absolves the comment I initially replied to in this thread.
> 6.5.2.3.6 states:
> One special guarantee is made in order to simplify the use of unions: if a union contains several structures that share a common initial sequence (see below), and if the union object currently contains one of these structures, it is permitted to inspect the common initial part of any of them anywhere that a declaration of the completed type of the union is visible
Outside of that special case, the general semantics of unions are specified as follows. I think that means you are right?
> 6.2.6.1.7 states
> When a value is stored in a member of an object of union type, the bytes of the object representation that do not correspond to that member but do correspond to other members take unspecified values
99) If the member used to read the contents of a union object is not the same as the member last used to store a value in the object, the appropriate part of the object representation of the value is reinterpreted as an object representation in the new type as described in 6.2.6 (a process sometimes called "type punning"). This might be a trap representation.
...it all makes sense until that last sentence, maybe that's for really weird bit casts like trying to reinterpret an integer as float. FWIW the only place I used union type punning was in an Z80 emulator for the 16-bit register pairs, but that's no longer necessary, since modern compilers will optimize two consequitive byte reads into a single word read:
All this to say: if Rust can make it in, that's great, because I'm tired of dealing with C's simplicity for more complicated tasks. I'm not a rust user but if it lets me use classes and templates, I'll switch over
Yer not switching any time soon then. Rust does have methods but not classes (QOM’s inheritance is specifically called as an issue in TFA) and it uses Haskell-style generics rather than C++-style templates.
You can though, unless I totally misunderstand your syntax
use std::{marker::PhantomData, ops::{Add, Mul}};
pub struct Foo<T> {
_p: PhantomData<T>,
}
impl<N, M> Mul<Foo<M>> for Foo<N>
where
N: Add<M>
{
type Output = Foo<<N as Add<M>>::Output>;
fn mul(self, rhs: Foo<M>) -> Self::Output {
todo!()
}
}Rust decided that it's important to not have instantiation-time compiler errors, but this makes computation with const generics complicated: you're not allowed to write Foo<{N+M}> because N+M might overflow, even if it never actually overflows for the generic arguments used in the program.
It's so much harder to debug things when function calls are secretly macros, and if I didn't have the vscode cpp language server for goto definition, I'd be completely lost. I'd wager that only 5% of the #defines aren't auto generated by occasionally recursive macros. Maybe hyperbole. Makes it really hard to figure out how existing code works.
Of Simula (& descendants) proto-OOP objects, not as it was envisioned by the one who coined the term.
http://userpage.fu-berlin.de/~ram/pub/pub_jf47ht81Ht/doc_kay...
> I didn't like the way Simula I or Simula 67 did inheritance (though I thought Nygaard and Dahl were just tremendous thinkers and designers). So I decided to leave out inheritance as a built-in feature until I understood it better.
> OOP to me means only messaging, local retention and protection and hiding of state-process, and extreme late-binding of all things. It can be done in Smalltalk and in LISP.
Fun that this was mentioned too:
>> it uses Haskell-style generics
> polymorphism via v-tables and composition via traits, that's enough object-orientation for me
For which Alan Kay had this to say:
> My math background made me realize that each object could have several algebras associated with it, and there could be families of these, and that these would be very very useful. The term "polymorphism" was imposed much later (I think by Peter Wegner) and it isn't quite valid, since it really comes from the nomenclature of functions, and I wanted quite a bit more than functions. I made up a term "genericity" for dealing with generic behaviors in a quasi-algebraic form.
In a way it's fascinating to see how C++ has shaped (dare I say warped) the collective vision of how to do OOP.
For example, as a Rubyist, which was heavily influenced by Smalltalk, it is fascinating how hard it can be to explain how fundamentally different message vs call are, and what's the whole point of modules, mostly because the other viewpoint is so warped as to cognitively reject that it can be any different from the C++ model.
I completely agree with his point about reimplementing C++ badly in C. GNOME does this too in their libraries. He will be much happier with Rust.
IMHO if you deliberately use C it's because you want to keep things simple. Sometimes C++ will inevitably lead to overcomplicated designs nobody needs; there's a reason why 90's C++ feels massively outdated, while 90's C is still (mostly) accessible. C is a great way to keep the urge people have to shovel in OOP even when it doesn't really make sense. When I see some bullshit C++ code at work, I often think, "if people had to write this in C, they wouldn't have overthought this this much"...
My 2 cents is that there was a Java craze back in the '90s that basically infected all code designed in that time period, and that's how we got some very weird nonsense that in hindsight was poorly thought out. Just look at stuff like GTK+ and GObject...
There are exactly none, but there was a time during the mid- to late-90s when it was hip to implement OOP frameworks on top of C (C++ only really became a realistic option once C++98 was widely supported which took a couple of years, ...also there was such an extreme OOP hype in the 90s that it is hard to believe today - EVERYTHING had to be OOP, no matter if it made sense or not).
QEMU seems to be from around the end of that era (first release apparently in 2003), and looking at the code it looks exactly as I imagined, with meta-class pointers, constructor- and destructor-functions etc... it would almost certainly be better to use a simple non-OOP C-style, but that was an 'outdated' point of view at that time (since everything had to use that new shiny OOP paradigm).
However there is a reason to have the object model, and it was added because some things were getting out of hand. You had to write three parsers for everything you added: one for command line, one for the text-based command interface and one for the JSON API. And you had to do it once for each kind of "object" (device, network backend, disk backend, character device backend etc.). The object model is designed to let you write code just once for all three and to reuse the interface across object types.
It's a common thing to do in projects like this in C. While C++ solves some of the problems for larger projects, it brings so many more, it's usually not worth it. Projects of this kind, essentially, invent their own language, with their own conventions. C is just a convenient tool (given the alternatives...) to do that, as it's treated as a kind of a transpilation target. Think about languages like Haskell which essentially work like that, but went even further.
So, what typically happens is that memory allocation, I/O and concurrency need to be done differently from how the language runtime wants to do that. So, at this point, you basically throw away most of the existing language runtime. From my recent encounters with different bugs in QEMU, I imagine that they really, really need a lot of customization in how memory is allocated (my particular case was that QEMU was segfaulting when running ldconfig in aarch64 emulation, which turned out to be due to ldconfig having some weird custom-made memory allocation pattern which QEMU cannot account for.) It's easier to fight this battle with C than it is with C++.
In other words, if you don't want the language runtime, if you don't want C++ templates, smart pointers, classes, lambdas, exceptions etc. But, you want to be able to manage memory for eg. realtime kind of system... simply removing all the unwanted stuff from C++ is harder than it is with C.
And, if you did want some / all of those features of C++, there are saner languages to have that (and you'd pay for it by having to drag in their runtime and conventions). Before Rust was a thing, I've seen people choose D instead, for example, as a kind of middle ground between the asceticism of C and unnecessary excess of C++.
Sure, Rust is better than C++ in some ways. You can have a legitimate debate about C++ versus Rust. There can be no debate about C versus C++: the latter is better in every single way.
> unnecessary excess of C++.
Is the unnecessary excess of C++ in the room with us right now?
Again, you don't have to use any part of C++ you don't like. A few minor spelling differences aside, it remains a superset of C. Using C++ not C will not hurt you in any project whatsoever.
C is simpler because you don't need to remove any of that stuff, because it's not there to begin with.
My argument about Lisp doesn't say that anyone should use Lisp instead, it just points out that people making the choice between C and C++ aren't concerned by how easy it is to build new languages using the language building tools of C or C++ because both are very bad at it. Had the objective of building a new language been the primary objective of such projects, they'd be using a language with superior language building tools (not necessarily Lisp, there are many alternatives, it's just that neither C nor C++ are good at it).
But yeah broadly speaking it's a terrible idea.
It's a full time job on its own to fix all of the compilation errors. Could probably fix it with coccinelle scripts to make some parts easier, but still, validating the codebase with different compilers to make sure there's no resulting subtle breakage either still requires a lot of effort.
If you read function specs in the OCaml docs, they're understandable and the syntax is clean; the same concepts bolted onto C++ look like line noise, both in actual syntax and the (allegedy) English-language description on en.cppreference.com.
Reading the C++ standard itself is an exercise in futility, very much unlike the C standard. (Although, latest developments in C land are repulsive too, IMO.)
The C++ committee's tantrums and scandals (physical violence at meetings!) put the worst that has been seen in open-source communities to shame. <https://izzys.casa/2024/11/on-safe-cxx/> has been posted to reddit and HN recently.
C++ compilation still takes absolutely forever, and C++ template error messages are as incomprehensible as ever.
One important problem with C++ (repeated ad nauseam, so this is nothing new) is the unforeseen and unintended harmful interactions between such language features that were supposed to be orthogonal. The only remedy for that is to stop adding stuff to the language; but nooo, it just keeps growing.
Another important problem is that, the more the compiler does for you implicitly, the less you see (and the less you can debug) what happens in the code. This is not a problem in OCaml, which is a managed language with a safe runtime, but it's a huge problem in C++, which remains a fundamentally unsafe language. (And yes, once you start extending OCaml with C, i.e., mixing implicit and explicit, OCaml too becomes unsafe, and a lot of head-scratching can occur.) A home-grown object system in C is at least explicit, so it's all there for the developer to read, instrument, step through, and so on.
When your "core guidelines" <https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines> could fill a book, there's a problem with your language.
(I'm not here to defend QEMU's object model indiscriminately; I'm here to bash C++.)
So, if you are developing something you want to see packaged in distros, it needs to be buildable with the tool versions in the distro's repositories.
(Not just rustup- Debian requires repackaging Cargo dependencies so that the build can be conducted offline entirely from source packages.)
This feels more like a CI thing for the QEMU project and I’m sure solvable by using rustup or a trusted deb repo that makes the latest tool chain available on older Debian platforms.
As for Debian itself, for toolchains it really should do a better job back porting more recent versions of toolchains (not just Rust) or at least making them available to be installed. The current policy Debian is holding is really difficult to work with and causes downstream projects to do all sorts of workarounds to make Debian builds work (not just for Rust by the way - this applies to C++ as well). And it’s not like this is something it’s unfamiliar with - you can install multiple JVM versions in parallel and choose a different default.
Most toolchains don't have as much churn as Rust.
Java in Bookworm installs JDK 17 which is three years old at this point. Java itself is on 23 with 21 being an LTS release.
This means that upstream users intentionally maintain an old toolchain just to support Debian packaging or maintain their own deb repo with newer tools.
You’re confusing cause and effect. People aren’t migrating because Debian packaging lags so badly, not because there aren’t improvements projects would love to otherwise use.
What churn? A release every 6 months? Unlike many others, toolchains (i count nodejs and co here) rust only need one toolchain because latest rustc always able to compile older rust code.
Now compare rust releases to this: https://gcc.gnu.org/releases.html
That is not true. Adding any public method to any impl can cause existing working code not to compile, and Rust adds new methods to the stdlib all the time.
trait OptionExt {
fn get_or_insert_default();
}
impl<T> OptionExt for Option<T> {
fn get_or_insert_default() {}
}
fn main() {
Option::<()>::get_or_insert_default();
}
Lots of QEMU users use it through their downstream distros. We even recommend that if you're using QEMU in a way that you care about its security then you should use a distro QEMU, because the distros will provide you timely security fix updates. Sure, we could throw all that cooperation away and say "tough, you need to use up-to-the-minute rust, if that's a problem for distro packagers we don't care". But we want to be a good citizen in the traditional distro packaging world, as we have been up til now. Not every open source project will want or need to cater to that, but I think for us it matters.
That doesn't mean that we always do the thing that is simplest for distros (that would probably be "don't use Rust at all"); but it does mean that we take distro pain into account as a factor when we're weighing up tradeoffs about what we do.
Out of curiosity though, have you explored having your own deb repo instead? I would trust QEMU-delivered security fixes on mainline far more than the Debian maintainers to backport patches.
As an upstream project, we really don't want to be in the business of making, providing and supporting binary releases. We just don't have the volunteer effort available and willing to do that work. It's much easier for us to stick to making source releases, and delegate the job of providing binaries to our downstreams.
> QEMU has historically not made particularly timely security fixes either on mainline or on branches
> It's much easier for us to stick to making source releases, and delegate the job of providing binaries to our downstreams
Am I correct that this is essentially saying "we're going to do a snapshot of the software periodically but end users are responsible for applying patches that are maintained by other users as part of building"? Where do these security patches come from and how do non-Debian distros pick them up? Are Arch maintainers in constant contact with Debian maintainers for security issues to know to apply those patches & rebuild?
[1] and also to stable branches, but not day-of-cve-announcement level of urgency.
Is it a precautionary concern that backporting patches gets more complicated if the vuln is in Rust code?
But then again Rust code isn’t even compiled by default so I guess I’m not sure why you’re bothering to support for old versions of the toolchain in mainline, at least this early in the development process. Certainly not a two year old toolchain.
That said we will probably switch to Debian rustc-web soon, and bump the lower limit to 1.75 or so.
The current approach basically guarantees that you're always targeting a ~2-4 year old version of the toolchain and that feels like a particularly weird maintenance burden given how many workarounds you're putting in to do so.
[1] https://launchpad.net/~jonathonf/+archive/ubuntu/rustlang
Anyhow starting April (August release) we will be able to target 1.75.0 while being consistent with QEMU's (C-targeted) distro support policies, which is not that bad. Maybe newer than that depending on what Ubuntu 22.04 does between now and August.
What Debian has is not a "stranglehold" but an ideology, and Debian continues to matter to (some) upstream projects because lots of users identify with Debian's hyperconservative, noncommercial ideology.
Your complaint is basically, "it's too bad that the userbase not sharing my values is large enough to matter".
> Your complaint is basically, "it's too bad that the userbase not sharing my values is large enough to matter".
Arch and rolling releases have about the same market share as Debian. Indeed, ironically, Debian's widespread adoption is seen primarily in the enterprise space where it's free as in beer nature and peer adoption is a signal it's a suitable free (as in beer) alternative to RedHat. Without Ubuntu's popularity a while back making Debian not so crazy an idea, I think "Debian" philosophy would not have anywhere near the adoption we see in commercial environments.
Does Debian have a stranglehold? AFAIK every other distro does the same thing, and all of them for good reasons.
I don't know if others have run into the same thing, but it's why I'd trust Gentoo's packaging more.
If Rust decides that it no longer supports compiling on x86, or if it starts depending on a version of LLM that no longer runs on a supported architecture, Debian must fix that for their users. That leaves the curl2bash installers that are popular with fast moving tools and languages useless for long term stability. The same goes for crates, which can be pulled at any time for almost any reason and break compiles.
Then there are other setups distros can choose to support, like having update servers/package repositories for updating servers that aren't allowed to connect to the internet or are even air gapped. You can't save rustup to a DVD, but you can take a copy of the Debian repository and update air gapped servers in a way that leaves a permanent trace (as long as you archive the disks).
Not all distros have this problem. In theory the Rust people could set up an apt/repository Debian users can add to get the best features of a package manager and the latest version, and distros like Arch/Gentoo don't bother with the stability guarantees most distros have.
Qemu can ignore these requirements and opt into not being packaged into distros like Debian or Ubuntu or Fedora or RHEL or Oracle Linux. That'd cost them a lot of contributions from those platforms, though, and may cause a risk of the project being forked (or even worse, end up in the ffmpeg/avconv situation).
Then you stop publishing new versions of the toolchain for x86 releases of the distro? I fail to see the problem. Nothing prevents you from not making a newer version of the toolchain available. Indeed, that's the default.
> The same goes for crates, which can be pulled at any time for almost any reason and break compiles.
I never said you have to extend this by making all versions of crates available. You can freeze the crates using the same mechanism to redirect Cargo as they do today.
I'm going to ignore the rustup stuff as I wasn't proposing rustup be used for building the base Debian image itself - that was a comment more about the environment QEMU itself can advocate for for people building it & leaving it to the distros to solve their own packaging problems.