Making Xen's dom0 I/O path NUMA aware
edera.dev
edera.dev
I like it because it is not a bag of parts. Everything is included. It is a VM toolkit that JFW, and it is not what keeps me awake at night
The only reason I suggest switching to XCP-ng is if you are fleeing VMware or the warts of your current system exceed your tolerance level. Personally, starting from a greenfield, I would go to XCP-ng without hesitation because XCP-ng is that level of production quality.
> XenSource got acquired by Citrix, who closely aligned with Microsoft in order to leverage the Xen commercialization against VMware ~ and the opensource community is hyper-aware of the long term damaging outcomes of these types of alliances with Redmond's powers for FUD.
> KVM's parent org. got acquired by Red Hat and that makes it more palatable for those same reasons.
> Hyperscalers who use KVM to manage multiple client VM instances on the same hardware via over-provisioning get to have kernel level insight into the KVM instances behavior via native, low-latency access to the hardware and direct integration with the Linux kernel's performance framework (perf) - this makes the noisy neighbor problem easier to mitigate than the same task on Xen which requires another layer of abstraction and has less out of the box native support choice of tools.
> KVM is/became a kernel module, and its development is now 'in-tree' for mainline kernel which also meant the kernel can perform the type-1 hypervisor activities using the standard Linux memory and hardware management toolkit. i.e. less maintenance
> there are more reasons, having to do with Xen pre-existing from the days when hardware didn't natively support virtualization and thus the need to modify the VM kernels to be PV (paravirtualized) - but putting KVM into mainline made the vanilla kernel hypervisor-aware as both host and guest
but mostly it's the first reason, and the rest sort of cascade from that one - the 'lack of support' from the standard tools developing opensource community is probably the largest reason.
The original question was vague, but we are getting a bit all over the place. Hyperscalers and most other uses don't really overlap that much.
TLDR; Its complicated. Open source communities have learned they need Institution/Foundation grade protection to keep IP from becoming commercial property in some cases. KVM won by having that realization earlier than most. There are very functional Enterprise tools for Xen which are AGPL but are behind a paid pre-compiled binary which discourages free-as-in-beer AND free-as-in-speech adoption. For anyone is willing to modify source and compile for themselves the tools are both kinds of free.
For the more 'all over the place version' here goes nothing:
I was in the thick of it over two decades ago. Avi (KVM's original coder) had very good insights and organizational wisdom. He started development when the virtualization hardware for intel/amd was already invented and that made virtualization possible as a normal linux kernel process. This is distinct from the Xen point-of-view which is about using the microkernel's abstraction as a separation of concerns, for several reasons including a basic security posture.
The 'sentiment' in regards to the risk of corporate vs true opensource was perhaps within the smaller community at the time and not highly visible outside of that community. It did however exist and later the changes in licensing for XenServer ultimately led to the fork of XCP-ng.
Other acquisitions about the same time which led to similar sentiment include:
Android's acquisition by Google just prior to the Citrix and Xen point in time, shortly after that MySQL was aquired by Sun which then got acquired by Oracle, leading to MariaDB.
The Qt framework acquired by Nokia, and mismanagement thereafter leading to the KDE Free Qt Foundation's origin.
Java/OpenJDK also went through the Sun to Oracle path which resulted in IP lawsuits against Google by Oracle for Java use on/in Android.
Another software suite which had been sponsored in a very cooperative way by Sun was OpenOffice.
Following the Oracle acquisition and on the heels of the aggressive licensing stance by Oracle in other software stacks motivated OpenOffice original core of developers to create TheDocumentFoundation and the libre-office fork to keep the development out from under Oracle's / Larry's corporate thumb.
Sun all but admitted that they had lost the necessary developer participation and market inertia by 'donating' (dumping) OpenOffice on the Apache Foundation.
We can perhaps agree that the donation didn't really get the community back together and reunite libre-office and OpenOffice in any meaningful way as the move put OO under the commercialize as you please Apache License vs LGPL/MPL,
So; this all establishes a theme of concern and developer communities tendency to a reaction to those concerns in the open source world in the 'several decades ago' time. Citrix as corporate owner of a significant part of the Xen ecosystem versus Red Hat's stewardship of KVM was part of the motive for decisions way back then, by my best recollection of the time.
The current AWS (arguably useful as a surrogate for the broader hyperscaler use case) approach is Nitro, which is KVM adjacent so for the sake of KVM vs Xen, and although AWS did stay in the Xen camp for quite a while (still offers Xen-on-Nitro for those who need it) I expect that the same reason applies - standard tooling and mainline support. Although Nitro does require some Amazon proprietary hardware to function.
I remember thinking at the time that if Xen had better stewardship it could have been a better contender for main kernel support, not instead of KVM, but as an also option. But the virtualization hardware was a big win for KVM and it took Xen a little bit to catch up with the PVH option.
Xen has remained solidly useful for virtualization in many places, including non-hyperscaler type use.
Xen Orchestra supports both XenServer or XCP-ng backends. While being technically open source it does have a a few features including backups which are paywalled in the precompiled binaries with a feature flag which prohibits free license users from accessing those functionalities.
The reluctance of users / enterprise customers to edit the source code to disable the 'check for paid/free license' and compiling clean binary appliance, and to do that for every update of the original source is their 'moat'.
While they are fully compliant with the AGPLv3 and can say "our stuff is open source", this satisfies only the strictest sense of the license. It doesn't supporting integration with most distributions' repositories. This ends up looking like a commercial product to most users look for personal to enterprise scale management front ends.
This 'moat' might not be expected to last with the arrival of code-monkey via agent. Enterprises previously opting not to maintain in-house code repos for management tools like this and just pay the 2k to 4k USD minimum entry cost for Xen Orchestra or similar are already starting to look at multiples of that in token subscription fees. As in-house code repositories become standard practice, why pay for both if the one will get you the other for the asking.
/thank_you_for_reading_my_TED-of-the-day
KDE Free Qt Foundation predates the whole Nokia debacle and smartphones by more than a decade.
You are right. It was Nokia buying Trolltech and getting the seats on the board of the KDE Free Qt Foundation in 2008 that I was not recalling properly. It's been a minute.
Thanks for catching my miss.
We can all continue to rely on the freedom aspect of Qt and KDE, the Foundation was ahead of the pack in protecting the openness through the risks of privatization.
I agree, but XCP-ng is still the most attractive system on the market for home / small business use in my opinion. Xen Orchestra itself isn't needed either, I do most stuff on the cli and code-monkeying arguably reinforces that even more. Patching up something that allows you to VNC a VM for the initial setup/debug doesn't require much investment / knowhow.
Say what you will about—or try to prompt out—their rhetorical flourishes, they can link together a series of background, observations, implications, …
But man, they are hopeless at crafting a high-level narrative. It’s one big hedge maze of a garden path:
This part makes good on that - and along the way it covers two things we did not plan to fix. One we hit before the real work could start: dom0's own memory placement was so skewed that synthesising a topology for dom0 without fixing it first would have been an exercise in lying to a kernel that had no memory to back the lie with. That one is a prerequisite, not a detour - everything else in this part is built on top of it. The other we hit after the work was nominally done: a memory-placement bug in our own toolstack that had been silently broken for as long as multi-vnode vNUMA had existed in our tree, and only ever surfaced under a memory-bandwidth benchmark. That one is the genuine detour, and it waits until the end.
Let's start with the prerequisite.
Where are we going here and why?!