ArcVRArthur

Hey YC, I helped Cohere scale GPUs to a 10k+ total GPU count for hybrid training and inference - if you’re curious about anything I wrote in the article I think there’s a good opportunity to read / reply to your comment. This article is made for fun.

Hey YC, I helped Cohere scale GPUs to a 10k+ total GPU count for hybrid training/inference computer - if you’re curious about anything I wrote in the article I think there’s a good opportunity to read / reply to your comment here. Please leave a comment!

I had the opportunity to help Cohere work on scaling transformers in the last year. If you have any questions about the article leave them below and I’ll do my best to answer openly. :)

I had the opportunity to help Cohere work on scaling transformers in the last year. If you have any questions about the article leave them below and I’ll do my best to answer openly. :)

I had the opportunity to help Cohere work on scaling transformers in the last year. If you have any questions about the article leave them below and I’ll do my best to answer openly. :)

Hey all,

I'm the the co-author of the GPU Virtual Machine (GVM project), and LibVF.IO. We just announced our enterprise product based on GVM called GVM Server. I'd love to hear what you all think of the work we've done and give suggestions on where we can improve in the future!

I was thinking this over in the past couple days and I think the words 'that they are aware' is really key here.

Ideally if GPU virtualization were sufficiently widespread as is support today for Intel VT-d, and AMD-v (IOMMU APIs for hardware assisted CPU virtualization) then software could make use of these functions without the user being aware of it. We're in a situation similar to that of CPU virtualization without hardware assistance with the early Xenoservers project from Cambridge (what would later become the Xen hypervisor and XenSource company). At that time there was not widespread support for virtualization assistance on most CPUs, and as a result Xen used methods like ring de-privileging to place the entire guest in ring 3 (userspace and kernel) while the hypervisor ran in ring 0 in order to virtualize any ordinary CPU model - my understanding is these were known as PV-guests (paravirtual guests). Over time however CPU companies began to introduce widespread support for features like VT-d and AMD-v to all of their models of CPU which enabled VM-exits/context save-restore with the use of shadow registers rather than ring de-privileging while Intel added new 'virtualization enhancements' through feature suites like vPro (SGX2 for example) which were only available on certain models of CPU (for example Xeon devices). Xen would adopt VT-d and AMD-v as HVM-guests (Hardware assisted virtualization) as they became more common on ubiquitous hardware and at the same time commercial forks of Xen would take advantage of these vPro features (like SGX2) for enterprise and high security government use-cases:

https://wiki.xenproject.org/wiki/Xen_Project_Software_Overvi...

Like before (around the time of the Xenoservers project) today we can effectively virtualize the GPU without hardware assistance mechanisms:

https://openmdev.io/index.php/GPU_Support

https://openmdev.io/index.php/Virtual_I/O_Internals#Mdev_Mod...

Since it's now practical to virtualize any GPU device (as was the case in the past with early Xen on CPUs supporting virtualization for various use-cases regardless of whether or not the hardware provided assistance mechanisms) it might then be time to start moving to a new paradigm of 'enterprise' vs. 'consumer' - in other words new 'virtualization enhancements' (similar to vPro on Intel's Xeons, ect..) are developed for enterprise GPUs (for example shadow page deduplication in VRAM, import/export of redundant objects between IO Virtual Address buffers, IOMMU protected balloon/deballoon, ect..) and basic hardware assistance mechanisms like SR-IOV & SIOV are enabled by default, across the board:

https://openmdev.io/index.php/Virtual_I/O_Internals#SR-IOV_M...

https://openmdev.io/index.php/Virtual_I/O_Internals#SIOV_Mod...

Thanks!! We do our best to keep the code as clean/readable as possible. The first version was a bit of a mess but we rewrote it again clean slate to improve over the first implementation. :)

That depends on your use-case. In general I would recommend you consider purchasing Nvidia's GPUs for the best price/performance and GVM support. Intel's Xe architecture is currently improving but the performance isn't quite there for a number of use-cases however some appear to work quite well and I expect that will improve with time. The 2080Ti works well with current software. If you are a developer and would like to help us improve support for devices you can purchase a 3090Ti (support in GVM for this device is under active development).

We have Ampere support working on some devices (and it is in development on other Ampere devices) as well as 11th & 12th generation Intel Xe. :)

Here's the GPU Support page if you'd like to take a look:

https://openmdev.io/index.php/GPU_Support

Ya, that's accurate. The precise driver implementation matters a lot. Having said that there are some good 'best practices' that seem to make a difference. In my opinion 'IOMMU Aware Mediated Device' could also make some much needed improvements here as it would allow for more granular IOMMU allocations - perhaps this mode could help further support the 'App VMs' use-case using shared work queues without breaking IO virtual address translation:

https://lwn.net/ml/linux-kernel/20190222021927.13132-1-baolu...

GVM uses IOMMU for compartmentalization:

This page has a comparison of the various IO assistance modes GVM can make use of (see comparison of assistance modes, the Mdev Mode section, and the SR-IOV Mode section):

https://openmdev.io/index.php/Virtual_IO_Internals

This will probably also play a role in future developments like SIOV (Scalable IO Virtualization):

https://lwn.net/ml/linux-kernel/20190222021927.13132-1-baolu...

Ya! You can use VMs that use X11 in the guest without issue. X11 also works on the host. Wayland is also working on the host - I haven't tested yet with Wayland guests yet so that's something to try.

Ya, LibVF.IO & GVM are built for things like this! For example I have a friend who uses it for various Adobe programs which also don't work well on Linux.

For sure! They can reach me at arthur@arccompute.io

I'll also be attending KVM Forum this year so I'd love to chat with folks there as well! :)

Thanks!! We'll do our best to keep improving things for everyone. Hopefully security by compartmentalization folks benefit from our work as well. I'll be going to QubesOS Summit so hopefully there will be more good conversations there. :)

If they would consider helping with the things I mention here we'd do our best to support them!

https://news.ycombinator.com/item?id=32585950

Unfortunately there is limited support on AMD. I would like to try to make it better though!

https://openmdev.io/index.php/GPU_Support

I hope I can work with them some time to improve support! I outlined a few things they would need to do to help support GVM using the amdgpu driver on this page:

https://openmdev.io/index.php/AMDGPU

Ideally some folks who know about amdgpu might consider helping our open source community by adding similar information to that page to the information we added on the Nvidia Open Kernel Modules page:

https://openmdev.io/index.php/OpenRM

If that could be done then we would do our best to add in AMD support to GVM.

There are also some benefits in load balancing GPUs. For instance processes can be controlled with controlled groups (cgroups) and niceness/thread affinity to make sure scheduling / resource allocation is done equitably between processes on the system. Using GVM and 'app VMs' it is possible to do similar things with the GPU (restrict GPU processes to a user configurable slice of the GPU so it cannot deny service to other processes on the system which are also using the GPU):

https://openmdev.io/index.php/Mdev-GPU#fbLen

Since this type of load balancing was originally used in the datacenter where virtual machine multi-tenancy was the use case the Quality of Service (QoS) functions here are fairly robust.

For my use I tend to run a high performing Windows VM (very good with most games and other GPU accelerated programs - even chrome/adobe these days uses the GPU) inside Linux.

This can also be used for various high assurance use cases such as OpenXT / QubesOS (security by compartmentalization). For example a laptop computer being used by an automotive company for CAD to keep their information safe from other programs on the system (like a browser in another VM) the prying eyes of competitors. I made a video of that here:

https://news.ycombinator.com/item?id=32585333

We have also been working on something called "LIME Is Mediated Emulation" which is a Win64 binary compatibility layer similar to WINE but using vGPUs. You can read about that here:

https://openmdev.io/index.php/LIME_Is_Mediated_Emulation

It is slightly different - this runs an unmodified GPU driver in the guest (no para-virtualization) and in our observations our performance is close to native which I believe is a fair improvement over the overhead of Virgl. Also there's availability of more APIs - for example DirectX, Vulkan, and OpenGL work with GVM guests. I don't think all of those work with Virgl. We make use of various different assistance modes (either hardware or software I/O virtualization/resource sharing) via VFIO-Mdev (VFIO Mediated Device), or via SR-IOV (Single Root IO Virtualization), or SIOV (Scalable IO Virtualization).

I made this web page to try consolidate some information from various folks who have contributed a lot in this area of open source to show how it works (there are some novel additions we've made as well based on our own work with GVM):

https://openmdev.io/index.php/Virtual_IO_Internals

Ya, you can use multiple VMs. Here is a video I took of a laptop with LibVF.IO and an early pre-release version of GVM installed - at the start of the video you can see a fullscreen Windows VM and then I back out and open another compartmentalized 'app VM' (browser):

https://streamable.com/gd45bh

Both VMs are actually based on Windows (one simply doesn't have explorer.exe running) and each has it's own virtual GPU attached.

This laptop has a Nvidia graphics processor inside but it also works with laptops that have an Xe Intel graphics processor. I'm hoping at some point it will work well on AMD too.

That depends on the driver implementation and what exactly the features are that you are looking for. In general if you're using virtualization tools like KVM/Xen you can configure a computer to make virtual machines that can be accessed remotely (for example you would likely want to configure the network to provide your machine with an IP address). You can use GVM for GPU accelerating various programs like those that use DirectX, Vulkan, and OpenGL so if that meets your needs from a remote machine and you have the necessary resources (networking resources such as IP addresses, time and know-how to configure such a setup) then I would say yes.

I would love to work with folks like the people at RedHat! We're working on some new tools in C (GPLv2) which we think will make it easier to work with existing kernel/driver maintainers. We'll announce some of those things early next month and right after we'll be at QubesOS Summit & KVM Forum. I hope we get the opportunity to talk to some of the RedHat folks! :)

According to https://docs.linux-gvm.org/

"GVM ... may be used in combination with KVM or other platform hypervisors such as Xen* to provide a complete virtualization solution via both central processing (CPU) and graphics processing (GPU) hardware acceleration."

Thanks for your kind words! We're all virtualization users ourselves so I definitely get what you mean - we started this project because we wanted to help people to get access to the magic at home using the hardware they already have! I will say though that Nvidia has done a lot towards improving their support for open source. While it might be a bit of an uncommon take in my view many of the folks who probably are most passionate about bringing exciting GPU technology to the world work at Nvidia. :)

Ya, that's exactly correct. :)

Hey, I'm the co-creator of LibVF.IO and the GVM project. I'd love to hear what you think about our latest release where we added support for GVM components in LibVF.IO.

Side note: We also added in support for Libvirt by allowing users to install standalone GVM components via ./scripts/install-standalone-gvm-components.sh.

Ya! It's in the install guide.

Most package updates don't disrupt anything. Depending on the vendor driver you use (Intel i915, AMD GPU-IOV Module, Nvidia) kernel updates should be okay if we use DKMS. GPU-IOV Module works okay with DKMS if you know what you're doing. I can't speak too much to i915. On Nvidia DKMS is hit or miss but that depends on which driver version is used. Some folks report DKMS doesn't work at all, others say they've never had a problem with it. I think it also depends on the kernel version DKMS is compiling against as in some cases the driver hasn't been updated yet to support the latest kernel version.

We try to help with that by making patch files that run ahead of official driver/kernel version support (they basically just make the driver work on a newer kernel - that's all):

https://github.com/Arc-Compute/LibVF.IO/tree/master/patches/