Hacker News Reader

> As a basic open source citizen, this would also help in "giving credit where credit is due". Bravo for stitching all of this together, but the project pages/repo certainly make it seem as if LibVF.IO/GVM did the work, and that's not really honest

GVM/Mdev-GPU depends on the Nvidia driver in this release. I think that's pretty clear but if you're not aware the page this thread is about does reference OpenRM driver interactions.

Not that this thread is about LibVF.IO which you've devoted most of your attention to throughout this discussion but we do give credit to Looking Glass, vGPU_Unlock, AMD GPU-IOV Module, and Intel i915 in the LibVF.IO install guide - they are discussed as part of the install process where they're mentioned explicitly by name. We also link them at the bottom of the article: https://arccompute.com/blog/libvfio-commodity-gpu-multiplexi...

> That said, a link to how mdev internals work isn't useful to end-users, who just want to know "how do I partition my card"? As-in "which driver/utilities do I need to install"?

OpenMdev.io is meant for developers, not for users.

> Is there some way in which LibVF.IO differs from just being a wrapper around KVM/qemu?

No, it's a Libvirt alternative with convenience functions for VFIO users. Here's the documentation:

https://openmdev.io/index.php/LibVF.IO

>"defines available mdev devices in the mdevctl list" only as a side effect of the fact that it's doing a whole bunch of other stuff to the system.

GVM/Mdev-GPU is unrelated to LibVF.IO which I think is where you're getting confused. LibVF.IO does not actually have any integration with GVM/Mdev-GPU so if you're reading that code you're not going to learn how GVM/Mdev-GPU works. We're planning to integrate the two but it's not done yet.

GVM/Mdev-GPU creates the mediated devices that are exposed in the mdevctl list. Read this code instead: https://github.com/Arc-Compute/Mdev-GPU/

>Similarly, it strongly appears that arcd could be more or less replaced by any given binding to the libvirt API, which would also allow the VMs to be easily migrated (assuming identical hardware), the libvirt XML shared with others, snapshotting, storage pooling, listed in virt-manager/virsh, and so on.

Sure, arcd is a reference Virtual Machine Monitor as it says at the top of this page: https://openmdev.io/index.php/LibVF.IO

It's actually unrelated to GVM. You can use GVM with whatever you want, including Libvirt/Virsh/Virt-Manager because we wanted to support users of those things with GVM rather than requiring that they use LibVF.IO.

> GVM, rather than "a GPU Virtual Machine ..." is Haskell bindings for the RMAPI.

Well, we do create mediated devices exposed in mdevctl defined by a user config file, so I would say it goes a fair amount beyond Haskell bindings for the RMAPI.

I think it's reasonable to describe a GPU mediated device as a virtual GPU given you get a virtual function that represents a scheduling share and virtual BAR space with a share of the device VRAM (partition of the GPU) which you can pass to one or several guests to allow them to run an unmodified guest GPU driver. I can't really think of a better definition for a vGPU. The Mediated Device Internals article pretty much explains the APIs GVM is dealing with - I believe we even link some sample code: https://openmdev.io/index.php/Mediated_Device_Internals

Your comment seems kind of trollish so I'm not really sure what benefit continuing this thread has. I think most of the stuff you're asking about is more or less documented and spelled out as openly as we're able to. What we're trying to do here is to make this stuff more open and available to people rather than locked away behind binary blobs. More or less everything we do is put into our wiki with very few exceptions. OpenMdev.io is made to be open to our community of folks working on Mediated Device/IO Virtualization functions on various projects so if you're a developer on this stuff and think anything is lacking you're welcome to contribute or or make suggestions in our IRC or Discord. I'm sure there's always room to improve and we put a ton of effort into trying to listen to feedback and improve upon things ourselves as well as accept contributions from others.

> OpenMdev.io is meant for developers, not for users.

Frankly, it isn't meant for developers, either. Almost every page on that site is either woefully incomplete, or crib notes from docs/talks, which is fine for a high-level overview, but it's not an API reference developers can use either. The sample code is mostly just lifted from other places (such as https://github.com/torvalds/linux/blob/master/samples/vfio-m...), so useless you're better off reading the source (https://openmdev.io/index.php/OpenRM), or just links to other people's APIs which interested devs can find.

It's fine to collate this, but it's far more like someone's personal aggregator than any kind of reference site.

> No, it's a Libvirt alternative with convenience functions for VFIO users. Here's the documentation:

> https://openmdev.io/index.php/LibVF.IO

I read this before I ever wrote a reply, which you should have guessed because there was no other way to get any information. None of it tells anyone WHY this should use this instead of the bindings which have 100 developers on them, which have been battle tested for years, and for which the original author of VFIO wrote exhaustive, excellent manuals on the blog I linked earlier 7.5 years ago.

What advantages does your system offer?

> GVM/Mdev-GPU is unrelated to LibVF.IO which I think is where you're getting confused. LibVF.IO does not actually have any integration with GVM/Mdev-GPU so if you're reading that code you're not going to learn how GVM/Mdev-GPU works. We're planning to integrate the two but it's not done yet.

I'm not confused either about how GVM/mdev-gpu works or about its relation to libvf.io. It's not hard to read between the missing lines of your project roadmap.

> GVM/Mdev-GPU creates the mediated devices that are exposed in the mdevctl list. Read this code instead: https://github.com/Arc-Compute/Mdev-GPU/

I did read that. There's nowhere else I would have said "Haskell bindings to RMAPI". I didn't call it anything else because it doesn't manage any other kind of mediated device, it's a pretty thin shim, and there's no real way to suss out what it's doing other than reading the code or the autogenerated module docs, which don't actually tell any developers where to get the values they need to populate it, which they can only get by reading other API docs (not yours), and if they're going to do that, they may as well just write their own in a language they like better.

It's not clear from the outset what the advantage is over just submitting a PR to mdevctl to echo into /sys/devices/..../[create|remove], and overall, the README doesn't give any information about it whatsoever, even `--help` output to show the args and defaults.

> Sure, arcd is a reference Virtual Machine Monitor as it says at the top of this page: https://openmdev.io/index.php/LibVF.IO

No, it is not. Point blank, it is not. libvirt also isn't. Even qemu isn't for hardware virt, and you're not doing IOMMU operations on emulated CPU calls. kvm is. It's a toolkit to manage virtual machines, maybe.

This does not answer the question at all of "why not libvirt?"

> It's actually unrelated to GVM. You can use GVM with whatever you want, including Libvirt/Virsh/Virt-Manager because we wanted to support users of those things with GVM rather than requiring that they use LibVF.IO.

It's unrelated... for now. And there is zero reason to use this instead of libvirt hooks which were written by and are tested by teams which already do this for libvirt (which virsh and virt-manager are just interfaces to anyway).

Again, I'm not saying this to be critical. There are plenty of libvirt-based projects out there which would welcome a standardized tool which they could all use as an entrypoint to this, because libvirt strictly does not (and will not, even with modularity) cover this use case rather than everyone re-inventing their own tooling to handle creation. It is unlikely in the extreme that the current state of GVM will work for anyone else's use case, primarily because "give me the UUIDs of existing devices" is already handled by walking /sys, and creating a new one of a given type (or removing it on VM shutdown) in more or less the same way.

GVM isn't built in a way which is usable by any other project. That's ok, but it does nothing to explain the design decision.

> Well, we do create mediated devices exposed in mdevctl defined by a user config file, so I would say it goes a fair amount beyond Haskell bindings for the RMAPI. I think it's reasonable to describe a GPU mediated device as a virtual GPU given you get a virtual function that represents a scheduling share and virtual BAR space with a share of the device VRAM (partition of the GPU) which you can pass to one or several guests to allow them to run an unmodified guest GPU driver. I can't really think of a better definition for a vGPU. The Mediated Device Internals article pretty much explains the APIs GVM is dealing with - I believe we even link some sample code: https://openmdev.io/index.php/Mediated_Device_Internals

You create mediated devices for nVidia devices by sending ioctls exposed via RMAPI. This is potato/potato. The explanation you just gave STILL makes it sound as if this is a novel thing done by GVM/mdev-gpu rather than something common, and talking down to someone who is asking informed questions about why you did it this way by linking to internals (when I was physically there for most of those talks and helped write some of the docs) doesn't paint a pretty picture.

> Your comment seems kind of trollish so I'm not really sure what benefit continuing this thread has. I think most of the stuff you're asking about is more or less documented and spelled out as openly as we're able to. What we're trying to do here is to make this stuff more open and available to people rather than locked away behind binary blobs. More or less everything we do is put into our wiki with very few exceptions. OpenMdev.io is made to be open to our community of folks working on Mediated Device/IO Virtualization functions on various projects so if you're a developer on this stuff and think anything is lacking you're welcome to contribute or suggest it to us in our IRC or Discord. I'm sure there's always room to improve and we put a ton of effort into trying to listen to feedback and improve upon things ourselves as well as accept contributions from others

NONE OF THE STUFF I'M ASKING ABOUT IS DOCUMENTED OR SPELLED OUT. That's the point. From someone who was a maintainer, engineering leader, etc on a major open source virtualization platform who literally wrote code which does this kind of scheduling/creation across a cluster, I am telling you that your documentation is opaque, misleading, takes credit for things you did not invent, doesn't explain your use case, doesn't explain why you re-invented the wheel, doesn't explain why there's a gaping "missing middle" between "here are kernel sources/function signatures in drivers and here's a tool" (where that "missing middle" is /sys/devices/.../mdev_supported_types[/...] and "echo|uuidgen"), etc.

This is, or could be, a great start to a unified ecosystem. You are going to have a very hard time getting a developer/user ecosystem if you do not provide better documentation, "what these tools do", find a way to talk with other virt developers without condescending to them, present usable interfaces other projects can call which are not "here is YAML/JSON to operate on with exec()", etc, and most of all, to acknowledge the work other have done/the knowledge they have rather than presenting any of this like it's brand new or novel. It could be a great utility. Or it could be something no other project ever uses. That's up to you.

My comments are not intended to be trollish. They are intended to tell you "as someone who has written very similar code and done very similar things for a long time, the only way to figure out what the hell any of this was supposed to do was to literally read the source and make educated guesses". The average developer/user is not going to have the knowledge base to make those guesses at all, but they may see references to "arcd ..." like it's "developer documentation", go find it, and ask "why the hell is this managing qemu directly instead of libvirt", or "why is no libvirt XML/qemu hook provided"?

These are real problems for the project. Docs, always, for every project. I know yours is new, but these are of unusually low quality for a submission to HN, and doubling down with links to the same inadequate docs like everyone you're talking to is a moron doesn't help your reputation. Additionally, examples. And reach out to others -- proxmox, ovirt, xcp, openstack (nova). See if you can collaborate. This will mean using (or at least providing) libvirt bindings/XML snippets like everyone else. It will be worth it.

> GVM isn't built in a way which is usable by any other project. That's ok, but it does nothing to explain the design decision.

It is. You configure a YAML/JSON file to define the mdev types that are presented by the vendor driver in the standard sysfsdev mdev_bus path. Those appear precisely the same way any mdev type appears to any program which uses the VFIO-Mdev API. In other words you can install GVM/Mdev-GPU underneath Libvirt or talk to it using mdevctl.

> You create mediated devices for nVidia devices by sending ioctls exposed via RMAPI. This is potato/potato. The explanation you just gave STILL makes it sound as if this is a novel thing done by GVM/mdev-gpu rather than something common, and talking down to someone who is asking informed questions about why you did it this way by linking to internals (when I was physically there for most of those talks and helped write some of the docs) doesn't paint a pretty picture.

This isn't potato/potato. All of these functions are artificially locked out on closed source programs that gatekeep the ability to use your own GPU. No, we're not making it sound like something special. All of our documentation that you seem to think linking to amounts to "talking down to you" because of your position in industry references common virtualization APIs in the kernel and drivers. We link to sample code on how to talk to those APIs. If anything we're showing people this stuff isn't magic! That's the whole point of doing this as open source and making it work on everything we can rather than making it closed source and locking it out to certain devices then charging you for the use of your own hardware.

> NONE OF THE STUFF I'M ASKING ABOUT IS DOCUMENTED OR SPELLED OUT. That's the point. From someone who was a maintainer, engineering leader, etc on a major open source virtualization platform who literally wrote code which does this kind of scheduling/creation across a cluster

I'm glad you have no problem gatekeeping in your comments or propping yourself up based on your job history. Many folks have offered constructive criticism as well as guidance on where to place our attention next many of whom I strongly suspect are at least as experienced as you (those folks who notably don't feel the need to often repeat how senior they are) but thanks again for the grandstanding so plainly.

>I am telling you that your documentation is opaque, misleading, takes credit for things you did not invent

No it doesn't. I constantly link to everything we're talking about on OpenMdev and to all the original sources, talks, mailing list posts, sample code. Please find any example you like of inappropriately taking credit for things and I'll happily correct it.

> where that "missing middle" is /sys/devices/.../mdev_supported_types[/...] and "echo|uuidgen"), etc.

This comment makes no sense. /sys/devices/.../mdev_supported_types[/...] and "echo|uuidgen" has nothing to do with GVM/Mdev-GPU.

> This is, or could be, a great start to a unified ecosystem. You are going to have a very hard time getting a developer/user ecosystem if you do not provide better documentation,

I agree we need to improve our documentation a lot still yet. It's open for community contribution. You obviously have some idea of what you're talking about enough that we can have this argument so as much as this has dragged on for an uncomfortable amount of time and I still think you have several fundamental misconceptions about what GVM/Mdev-GPU actually does it would be nice if I could structure this interaction with you into something I could use to condense what you're saying here into documentation improvements rather than a flame war in a web forum comment section.

> and most of all, to acknowledge the work other have done/the knowledge they have rather than presenting any of this like it's brand new or novel. It could be a great utility. Or it could be something no other project ever uses. That's up to you.

We're CONSTANTLY citing sources on OpenMdev.io and in our other project (LibVF.IO which is again unrelated to this thread) as I said we mention all the other open sources we deal with by name multiple times in the install article and also link them at the bottom of the article again. I even dedicate a section of all of our open source talks (FOSDEM/X.org) just to thanking the authors of those projects! I mention them by name in those talks as well multiple times!

https://www.youtube.com/watch?v=xbVKMQ1Rz2Y

https://www.youtube.com/watch?v=8pVrTyLqV_I

The whole point here is to make it clear to folks this stuff is NOT MAGIC. It's easy to understand and we show exactly how it works to the best of our ability citing sources for all the documents as well as literally linking to code samples on how to write your own programs against these APIs and making our own programs available as free open source software that makes this work on most hardware.

> My comments are not intended to be trollish. They are intended to tell you "as someone who has written very similar code and done very similar things for a long time, the only way to figure out what the hell any of this was supposed to do was to literally read the source and make educated guesses". The average developer/user is not going to have the knowledge base to make those guesses at all

First off you don't have to read the source and make an educated guess because we've gone and done the work of building an implementation for you that does the things you're not allowed to do by design (anti-user limitations). We do have work to do on improving our documentation.

> but they may see references to "arcd ..." like it's "developer documentation", go find it, and ask "why the hell is this managing qemu directly instead of libvirt", or "why is no libvirt XML/qemu hook provided"?

I really am getting tired of explaining that GVM/Mdev-GPU IS NOT LibVF.IO and it has zero integration with it. GVM/Mdev-GPU works with anything that works with the VFIO-Mdev API including Libvirt/Virsh/Virt-Manager. I really hope by the next round of comments this point is very clear so I don't need to retread this point yet again.

> I know yours is new, but these are of unusually low quality for a submission to HN, and doubling down with links to the same inadequate docs like everyone you're talking to is a moron doesn't help your reputation. Additionally, examples.

I fail to see specifically how the links are inadequate in the context in which I've linked them other than taking your word for it. I know we're missing things - they don't cover every category of knowledge yet but I'm trying to improve upon them and do them as openly as possible so other folks can benefit from them/help as they like. I don't think everyone is a moron and I'm literally trying to share everything I learn because I believe that's the best way to lift people up, but my guess is you think I am a moron because you seem to assert the same falsehoods over and over despite that they've been corrected more than once (GVM/Mdev-GPU is not operable with Libvirt / LibVF.IO is related to GVM/Mdev-GPU / GVM/Mdev-GPU could be a PR to mdevctl - a bash script for echoing values into sysfsdev mdev_bus).

> Additionally, examples. And reach out to others -- proxmox, ovirt, xcp, openstack (nova). See if you can collaborate. This will mean using (or at least providing) libvirt bindings/XML snippets like everyone else. It will be worth it.

I'm trying to talk with everyone I can talk to. There have been some folks from some larger projects (QubesOS, Xen Project, OpenXT) offering their mentorship and who have given countless hours to offering advice - To them I've very grateful and I always hope to expand that circle. Besides the public community discussion threads in IRC/Discord we have community calls on the first Wednesday of every month.

> Frankly, it isn't meant for developers, either. Almost every page on that site is either woefully incomplete, or crib notes from docs/talks, which is fine for a high-level overview, but it's not an API reference developers can use either. The sample code is mostly just lifted from other places (such as https://github.com/torvalds/linux/blob/master/samples/vfio-m...), so useless you're better off reading the source (https://openmdev.io/index.php/OpenRM), or just links to other people's APIs which interested devs can find.

Well, the Mediated Device Internals article does cite every piece of information used but it's still incomplete. I made that page because it's largely a collection of everything I've learned while developing software. The "lifted" samples literally have the authors at the top of every header and the original link is the first line beneath the title in the README.md: https://github.com/OpenMdev/VFIO-Mdev_Samples

> I'm not confused either about how GVM/mdev-gpu works or about its relation to libvf.io. It's not hard to read between the missing lines of your project roadmap.

You define types that are presented in mdevctl via a YAML config file the user can write themselves or via a JSON file. So LibVF.IO might as well be entirely ignorant to the fact that GVM is on the system. All it sees are different arbitrary mdev types that the user decides to create. Not sure where you're going with this comment.

>I read this before I ever wrote a reply, which you should have guessed because there was no other way to get any information. None of it tells anyone WHY this should use this instead of the bindings which have 100 developers on them, which have been battle tested for years, and for which the original author of VFIO wrote exhaustive, excellent manuals on the blog I linked earlier 7.5 years ago.

>What advantages does your system offer?

Well for starters I've never argued LibVF.IO is better than libvirt. In-fact when folks bring it up I generally say there's a lot we can learn from libvirt. GVM/Mdev-GPU isn't LibVF.IO so I'm kind of blown away that you keep bringing that up as if they're the same. GVM/Mdev-GPU controls which types the vendor driver presents to mdevctl - GVM/Mdev-GPU is neither an mdevctl replacement nor does it share code with or even integrate with LibVF.IO. I don't think you really grasp that. GVM/Mdev-GPU enables the creation of arbitrary mdev types against the Nvidia driver as defined by the user. Those types generally have "capabilities" restricted by arbitrary policy and "available mdev types" are stored in an XML file alongside signatures which attempt to stop the user from defining their own types. GVM/Mdev-GPU gives control back to the user. Also our codebase is GPLv2 and we don't enforce any kind of "device-specific" restrictions. Nvidia's equivalent is closed source and intentionally locks out consumer cards.

>I did read that. There's nowhere else I would have said "Haskell bindings to RMAPI". I didn't call it anything else because it doesn't manage any other kind of mediated device, it's a pretty thin shim, and there's no real way to suss out what it's doing other than reading the code or the autogenerated module docs, which don't actually tell any developers where to get the values they need to populate it, which they can only get by reading other API docs (not yours), and if they're going to do that, they may as well just write their own in a language they like better.

It's unclear what you mean by a "thin shim". Yes, I agree we do need to document how to use it better.

>It's not clear from the outset what the advantage is over just submitting a PR to mdevctl to echo into /sys/devices/..../[create|remove], and overall, the README doesn't give any information about it whatsoever, even `--help` output to show the args and defaults.

mdevctl is a bash script for echoing values. GVM/Mdev-GPU is creating the available virtual functions that mdevctl sees underneath it (a layer lower). I hope that clears that up.

> This does not answer the question at all of "why not libvirt?"

You can. I think I've covered that LibVF.IO isn't the only thing that works with GVM/Mdev-GPU. You can use Libvirt or whatever you like. It's completely agnostic.

> It's unrelated... for now. And there is zero reason to use this instead of libvirt hooks which were written by and are tested by teams which already do this for libvirt (which virsh and virt-manager are just interfaces to anyway).

Okay, how about you can't create your own mediated device types? You're confined to those that Nvidia generously offers to you with arbitrary restrictions on their "capabilities" - "capabilities" which you have to differently pay for to virtualize your own GPU - A GPU I will remind you which is entirely restricted to their enterprise product lineup and artificially locks out consumer GPUs. When the user can define their own mdev types it puts them in control of how they can virtualize their own GPUs. If you want to use mdevctl on top of GVM/Mdev-GPU or Libvirt or anything else there's absolutely nothing stopping you. We also aren't stopping you from using your normal person consumer GPU (the kind most people own).

> It is unlikely in the extreme that the current state of GVM will work for anyone else's use case, primarily because "give me the UUIDs of existing devices" is already handled by walking /sys, and creating a new one of a given type (or removing it on VM shutdown) in more or less the same way.

"Give me the UUIDs of existing devices" is not what GVM/Mdev-GPU does. It allows the user to create their own arbitrary mdev types. That has no connection at all to "give me the UUIDs of existing devices" or creating a new one of a given type or removing it on VM shutdown. This is literally defining the mdev types that are presented by the vendor driver and seen by mdevctl. It has nothing to do with what you just said.

>> https://github.com/intel/linux-intel-lts/commit/41ef979f0894

> This is pretty unhelpful. Legitimately

Indeed, "98 changed files with 11,276 additions and 46 deletions" and no idea if it will work on a vanilla kernel.

I would like to try running linux baremetal to virtualize Windows 11 running in fullscreen mode with control over the mouse and keyboard, but I may wait until that's mainlined.

I am also looking forward to when it's mainlined. Having said that until a short time ago everything going on with i915 SR-IOV was a secret (nobody on the i915 mailing list would talk about it, there was no open source) so I'd say we're moving in the right direction.