plam503711

Vates is the company doing both XCP-ng and Xen Orchestra, and now selling support for both in a single "bundle" called Vates VMS.

I'm not sure you are aware about the cost of migrating from one virtualization platform to another, especially when you have 4000 VMs. I can tell you it's not exactly easy, and that's even our business now (migrating from VMware to our stack).

It's not like changing a light bulb.

It is not, but yeah, we also have NASA as customers. However, we do not chase specifically aerospace companies. We are simply an open source alternative to VMware. So doing an ad explaining how to literally git pull the product without even talking to anyone or giving your email to our sales would be a weird strategy :D

"forgettable infrastructure component": this is what runs their entire IT. We build both the hypervisor and the backup/orchestration for it. Our stack could kill their entire operations if it's down because $whatever. 4000 virtual machines running isn't just the print server or the coffee machine.

I'm open to discuss, please add me on LinkedIn :) (you can find it in the author icon at the end of the blog post)

Well, now I’ve seen it — and yes, lesson learned. But here’s the good news about humanity: they’re the only ones abusing it at this scale. So far, it seems most people still choose sanity over spreadsheets of throwaway emails.

We operate globally, and this company isn’t even on our continent. On top of that, it’s a semi state-operated entity — so you can probably imagine where any legal effort would end up: somewhere between bureaucratic limbo and /dev/null.

I first write my entire text and then after that I use a LLM to fix the grammar and have a better flow. I'm doing my best but I'm not a native US speaker. Before LLMs, people complained about the weird sentences or mistakes I made. Pick your poison ;)

Anyway, I'm doing my best to keep my own "signature" in writing, but it's really hard when you see a better phrasing generated on your original more limited vocabulary. But anyway, I'll do better next time, thanks for the feedback!

I’m actually considering reaching out directly to the CEO and telling the full story. But honestly? There’s a good chance he’s fully aware — and totally fine with it. That’s part of what makes it so disappointing.

We’re not rushing into legal action — it’s not worth the energy for now — but publicly calling out the behavior felt necessary. It also sends a message to others in the ecosystem about the kind of nonsense OSS maintainers sometimes face.

And yes, while I’m still holding off on naming the company directly… I haven’t ruled it out.

You're absolutely right that businesses act within whatever constraints exist — and yes, we were a bit naive. We assumed that if someone had a fully functional, free, open source version available (well-documented and easy to install), nobody sane would go out of their way to abuse the trial system instead.

To be clear, it’s not just trial abuse — it’s actively ignoring the better, freer option in favor of repeatedly faking evaluations just to get the “easy mode.”

We’ll definitely tighten things up going forward. But in nearly a decade of doing this, they're the only ones to push it to this scale. So yeah, they've earned a spot in our open source hall of shame

Someone got the ref, finally ;)

We do not have any unpaid internship. So yes, I will hire them, until we reach a limit of mentors for them, because we cannot leave them alone.

That's entirely true (yeah I'm biased), frankly that's one of the best place in France for IT. And a lot cheaper/less stressful than Paris :p

100% agree and that's exactly what we do. Almost all of our juniors are near the office. Because they need to see what's working at a company means at least once in their life.

There's few exception when we don't have the choice, but really, I cannot imagine myself to be full remote without having seen a company "in the flesh" before.

I agree. If it's already hard for simpler stacks, you can imagine how hard it is for more critical or complex ones. And it's even worse if you inherited some of it (ie collecting technical debt that's not yours, which is the case here as some part of the stack are the result of a fork).

Sometimes it's even a catch-22 situation, where the technical/generic knowledge is already hard to find, but you absolutely need it to train more junior people. Luckily we found such experimented people, but then you also need to use their expertise to actually fix stuff and not just mentor juniors. A very very delicate balance to find, especially in a timed market.

Hi,

I respectfully disagree with much of your comment.

First, this wasn't intended as a promotional piece. It's a personal blog post where I share some of the challenges involved in building a full virtualization stack — a stack that happens to be fully open source. It's unfortunate that sharing real-world experience is sometimes immediately perceived as promotional.

Second, I think there's some confusion between using a hypervisor and mastering one — or building and maintaining an entire stack around it. KVM/QEMU is widely used, but it has significant issues, especially regarding security, performance, and latency consistency. Very few groups in the world are actively trying to tackle these challenges holistically (even major players like VMware have made some questionable shortcuts).

When it comes to low-latency, real-time use cases with a strong security model, Xen remains unique among open-source hypervisors. It's definitely not boring — in fact, it's one of the few that enable certain classes of critical applications at all.

We also work closely with academic research labs, and I can tell you: there’s still a lot of exciting work happening around Xen — even if it's less visible than buzz around newer projects like Firecracker or crosvm.

Because we cannot afford that, we aren't Google, and still a relatively small company vs the task of building a full-stack virtualization solution. Luckily, we have other strong points helping a lot (remote first, no micro-management, a great culture promoting human values and so on etc.)

Sure, let me explain it a bit better. It's more like in the sense of the "stack" is very deep now. Clearly, we have/hire Xen/hypervisors specialist, and we do not ask them to be CSS experts. However, deeper in the stack (at lower levels) harder it is to find them, because of the lack of expertise in universities and/or appeal of doing such job.

And if you find or train those low-level/system-oriented people, they also need to understand how a feature they build will be exposed functionally to a user (and why they need it in the first place). Because things are not make into thin-air but required to work in a bigger picture (ie: the product).

It's true but until some extent. When you are talking about a hypervisor (like Xen), and many many subtle things depending on your CPU brand/model, it's really really *hard*, even with an LLM (and even more with an LLM hallucinating some CPU features or forgetting basic things like Meltdown and Spectre).

However I agree: to learn a topic, LLMs are providing a great speedup. As a CEO/co-founder, I have no issue to hire people without a degree if they are good at what they do. However, our biggest chances are to scout directly in universities to find motivated students (motivation >>> everything else)

I agree but there's also the extra difficulty to do mentoring remotely (we are a remote first company). I *really* like being remote first and provide the choice if you want to work on site or wherever you want. But it does come with some challenges.

Hi!

It's not a promotional piece of something, it's my personal experience as a CEO and co-founder of a company using Xen as the core of our stack. I like to share my views in a transparent fashion on how it's hard to do very technical stuff, not just for technical reasons, but due to the lack of people trained for.

Nice catch! Fixing it now.

Hi, I’m the author of the blog post.

The goal wasn’t to dive deep into technical internals this time, but rather to share some perspective on what it actually takes to master an entire virtualization stack — especially after years of working only on the top layer (orchestration, backup, etc.). We’ve seen firsthand how much more complex things get the deeper you go.

That said, I totally get the desire for more technical content — and we’ve published more in-depth pieces before. For example, here’s a detailed post on Xen’s grant table mechanism and memory sharing, if that’s more your kind of read: https://xcp-ng.org/blog/2022/07/27/grant-table-in-xen/

Appreciate the feedback — and happy to go deeper on specific areas if there’s interest!

That's because of the confusing names. Xen is not XenServer:

* Xen is the hypervisor, the "upstream" used by both XenServer & XCP-ng

* XenServer or XCP-ng is the platform, using Xen and various other things, to deliver a "turnkey" experience for server virtualization

Xen Project is part of the Linux foundation, and have various contributors, including AMD and Arm (for the embedded world) and others (including Vates, Citrix etc.)

Xen provides a great security design and a protocol to do hypercall that makes sense (unlike kvm+virtio which is DMA all the way, with all the plus in terms of simplicity but the bad on the isolation aspects).

If I wanted to caricature the situation: KVM is more simple to work with in terms of dev (you have results fast), but kind of "fuck security".

Xen is hard from the dev perspective, because it's a more micro kernel by itself, and you can't cheat to have access to the memory, you have to use grant tables (see https://xcp-ng.org/blog/2022/07/27/grant-table-in-xen/ ).

So if a part of the industry took a shortcut, doesn't mean Xen isn't still relevant :)

I don't think it will be "instant change". I agree on the gradual result, but I think it might be faster than we think. Yes, it also depends on the ecosystem, but I think the world is more ready for ISA diversity than ever.

(XCP-ng founder here)

I'm more optimistic for RISC-V. For all the devs who worked on it here (at https://vates.tech), they told me it's very easy to work with since it's close to many Arm design principles.

That's why I believe it's important to prepare the platform today for those future machines. I think it's a great opportunity to not only get an alternative both x86 and Arm, but also really opening the choice of the design, letting new players mastering both hardware and software (I have to admit that's something I'm considering for my business at some point).

Thank you, both for being kind and also providing a constructive feedback. "So" is very common trap for French speakers :D

It's not a custom Xen: it's *is* Xen (AFAIK, with possibly their own patch queue for some specific needs on top of it). What's custom is the toolstack around it :)

Xen Project is far being dead (there's a lot of activity in the mailing list, and now, thanks to new contributors like Vates/XCP-ng, there's also more initiatives to have a decent project tracking, see https://gitlab.com/groups/xen-project/-/epics?state=opened&p... for example).

Regarding nested virt, you are mostly right: it's only "working-ish" for basic things, but indeed, it's broken when you start to use anything heavy in your nested VM. The main reason nobody fixed it is because it's not really used: as any other open source project, you find what you need if you contribute. Obviously, as soon someone will need this and willing to contribute, it will change :)