Writing an NVMe Driver in Rust [pdf]
db.in.tum.de
db.in.tum.de
Basically: this is good work (great work for an undergraduate thesis). But it's very much "solving the easy part" and not really showing off Rust in any particularly impressive way. You can write very similar userspace "drivers" (and I have!) in Python.
[1] Though on modern hardware designed to manage arbitrary scatter/gather command queues on its own, that's not really such a big deal. In performance situations the hardware will always have something to do anyway, and idle hardware can be sent a command synchronously. Fixing this amounts to a power optimization only.
I wrote a small NVMe user-space driver using VFIO more than a decade ago this way. Coming from having virtualized ATAPI (SATA) and SCSI, NVMe was such a refreshingly excellent design.
Does Rust fundamentally guarantee that if you make a struct, its fields will lay out in memory in the order that you defined them? Can it be used to interact with APIs (really ABIs) who expect a C struct (or pointer to one)?
I think my main frustration with stuff like Go and Swift in this case is that their structs are not binary-compatible with C structs in this way because they rearrange things to be better aligned/packed/whatever.
You have to specify this behavior with #[repr(C)]. Otherwise, the compiler will rearrange fields to try to optimize packing and alignment.
Does not currently do what ?
Rust certainly will re-arrange the layout of a default repr(Rust) struct to make it smaller, for example: https://rust.godbolt.org/z/7KsqvnE9o
[Edited to provide a nicer Godbolt example which compares the two layout strategies]
https://doc.rust-lang.org/reference/type-layout.html#the-c-r...
https://pkg.go.dev/golang.org/x/tools/go/analysis/passes/fie...
If you need binary-compatibility with C structs in Swift, you can define them in a bridging header.
[0] https://github.com/bootreer/vroom/blob/37bd8a22f5e0550b2cbc9... [1] https://doc.rust-lang.org/reference/type-layout.html#the-ali...
Does anyone have nice resources to share like this one that focuses on a specific port/connection and implement a driver/reader/parser? I'd very like to learn more of this.
Bitluni has a couple of easily consumable videos about VGA.
I was like -> oh, it really just is a huge bus and there's A LOT happening.
The only bits you need to worry about that looks like hardware are getting the correct physical addresses filled in, and on some platforms worring about memory ordering and/or cache management to be sure the device sees the same memory state you think it does.
Clear writing and ideas
However, I was waiting for the touted memory safety to be mentioned beyond the introduction, but it never really came up again. I was hoping for the paper to make a stronger argument for memory-safe languages like Rust, something like "our driver did not have bugs X, Y, and Z, which were found in other drivers, because the compiler caught them".
Additionally, in a userspace device driver that is given control of a piece of hardware that can do DMA, like an NVMe controller, the most critical memory safety feature is an IOMMU, which the driver covered by the paper does not enable; no amount of memory safety in the driver code itself matters when the hardware can be programmed to read or write anywhere in the physical address space, including memory belonging to other processes or even the kernel, from totally "safe" (in Rust semantics) code.
While the driver from the paper may certainly have a "simplified API and less code", I don't expect much of this to be related to the implementation language; it's comparing a clean-sheet minimal design to a project that has been around for a while and has had additional features incrementally added to it over time, making the older codebase inevitably larger and more complex. This doesn't seem like a particularly surprising result or an endorsement of a particular language, though it perhaps does indicate that it would be useful to start from scratch now and again just to see what the minimum viable system can look like. I certainly would have liked to rewrite it in Rust, but that wasn't really feasible. :)
In any case, it's great to see proof that a Rust driver can have comparable performance to one written in C, since it will hopefully encourage new code to be written in a nicer language than C. I definitely don't miss having to deal with manual memory management and chasing down use-after-frees now that I write Rust instead of C.
(As a side note, I'd encourage anyone thinking of using a userspace storage driver on Linux to check out io_uring first before going all in; if io_uring had existed before SPDK, I don't know that SPDK would have been written, given that io_uring gets you most of the way there performance-wise and integrates nicely with the rest of the kernel. A userspace driver has its uses, but I would consider it to be a last resort after exhausting all other options, since you have to reinvent all of the other functionality normally provided by the kernel like I/O scheduling, filesystems, encryption, etc., not just the NVMe driver itself. That is, assuming the io_uring security issues get resolved over time, and I expect they will.)
The more general point is that if you need any of the many features of a general-purpose OS kernel, a full userspace driver may not be a very good fit, since you will end up reinventing a lot of wheels. Cases where it could be a good fit would be things like database backends or dedicated block storage appliances, situations where the OS would just get in the way and where it's viable to dedicate a whole storage device (or several) and a whole CPU (or several) to one task.
But for those who want to get a handle on how rust might be used for something other than yet another crypto project or a toy webasm app, TFA is exactly what the doctor ordered.
Normal block I/O use cases don't really need NVMe io_uring passthrough, but it addresses the more exotic cases and is available in mainline Linux. And NVMe passthrough might eke out a little more performance.
A lot of memory and thread safety guarantees in Rust are only applicable to user space.
Removing that, there are still some great things in Rust (e.g. enums ergonomics) but also a few questionable things.
Rust programming style is building safe (zero-cost) abstractions on top of unsafe primitives, turning all other code into safe "glue" code. If you design for it, you can have a lot of "boring" code even in a kernel.
A kernel will have more unsafe primitives to implement, but the safe/unsafe division still helps testing the unsafe parts, and still prevents bugs caused by misuse of these APIs.
The type system that gives safety in userspace still exists in the kernel space. So even if the allocators and threads are different, you still have the type system tools to write safer APIs for them.
Rust's culture says you must mark abstractions that leak safety as unsafe. If this Rust function named "make_doodad" is labelled safe, it is not OK that I could run it without a doodad_manager, and yet in this case it blows up. Either somehow require me to prove I have a doodad_manager, or, mark it unsafe and document the requirement, or re-design the function so that it checks for a doodad_manager and fails cleanly when one is not present. In some cases you might decide all three are needed: make_doodad_with_manager(&manager) -> Doodad, unsafe make_doodad_unchecked() -> Doodad and make_doodad() -> Result<Doodad,Problem>
I'm not sure what you are thinking of here. Rust memory safety doesn't care about the environment at all.
There are some ideas around Rust code patterns and structure for bare metal, see for example the RTFM work (now renamed). But they all do have some drawbacks such as redeuced readability and IMO too much abstraction.
Anyway, my point was that since of the guarantees Rust provides build upon certain assumptions about the environment that generally don't exist in kernel space.
Also, `unsafe` doesn't disable all language features or the type system, it just provides an escape hatch to use raw pointers. Which, yes is quite a big step away from "normal Rust", but that's why we abstract around them.
It sure is extra and boring work, but it's entirely possible to create ergonomic APIs around unsafe low level primitives. I mean, that's how a lot of stuff gets implemented in stdlib or even in some crates. We just don't interact with it frequently, though.
Not kernel developer myself, but I've done some embedded Rust and written drivers
For the more technical points:
The kernel needs to have it available in a somewhat complex way to be able to mount the drive. I.e. init becomes a lot more involved with an initram that first needs to load the driver (potentially loopback it?) and the mount actual root.
To some degree there can also be issues around syscall boundaries. I.e. the usual monolith vs. microkernel. I haven't checked the API they hook into to provide the device to other components. But it likely requires the kernel to jump back into userspace in various "hot-ish" paths for IO.
I.e. it can only be used by a single consumer, that consumer doesn't get nice things like ther kernel's file systems or device mapper (raid/crypt/verity...) features. This can be fine, e.g. when the consumer is a database that just needs a block device, or something like ceph/minio etc. that provides the storage api to its consumers.
It'd have to use something like NBD (~fuse for block devices) to actually integrate and then my previous post describes some of the downsides to that setup.
When queue depth is low(like qd=1), random 4k read iops is far more less(14.5 kiops vs 128 kiops) than 4k random write iops. When queue depth is high, like qd=32, the read iops and write iops becomes similar. But read iops is still less than write iops.(436 kiops vs 608 kiops)
I wonder why read is slower than write? Is it because ssd has a fast write cache, and it will finish the write request once the data is written into cache? Or it simply report that the data is written and actually write them in batch in background?