Qubes-Lite with KVM and Wayland | Hacker News Reader

Qubes-Lite with KVM and Wayland

roscidus.com

141 pointstodsacerdoti48 comments

This is an amazing effort, very cool!

I'm not sure why the author is having so much pain with Qubes. Indeed the lack of GPU in guest VMs is annoying but it is possible now to assign a GPU to a HVM fairly reliably thanks to all of the VFIO/gaming-on-linux enthusiasm in the past years. Otherwise, I also find that running browsers in multiple VMs on laptop is a problem if you don't disable JS by default because modern websites have become so bloated, it's a tragedy. The LVM remark is also strange. It's very reliable for plenty of people, but there is the risk of running out of space for metadata [0]. Thin-pools for VM storage allows for some great Time-Machine-esque incremental backups also [1]. But for managing multiple development environments, Qubes is a blessing, not even including all the security benefits.

Another option for Xen fans is XCP-ng + a thin client machine for accessing the VMs. One can also use firejail+Xephyr to achieve graphical isolation [2] (not sure about Wayland).

It looks like architecture changes in Qubes future [3] may make KVM a reality.

This is still a very cool effort, I'll have to give the Wayland bits a close read.

[0]: https://github.com/QubesOS/qubes-issues/issues/3243, https://listman.redhat.com/archives/linux-lvm/2018-July/msg0...

[1]: https://github.com/tasket/wyng-backup

[2]: https://wiki.gentoo.org/wiki/User:Sakaki/Sakaki's_EFI_Instal...

[3]: https://www.qubes-os.org/news/2020/03/18/gui-domain/

The Qubes people don't recommend doing GPU passthrough because of the security implications.

As for the OP, I feel like if somebody cares about security, they shouldn't be doing any of this. Trying to come up with some self-designed hodgepodge of things isn't really enough security-wise, even if you do use VMs, and I'd find it hard to trust something like this as a platform to do anything important on.

> The Qubes people don't recommend doing GPU passthrough because of the security implications.

Why? DMA?

I don't know details. I was thinking about doing GPU passthrough myself, but whenever developers chimed in on any posts about the topic, this is essentially what they said.

Some links I remember going through:

https://www.qubes-os.org/doc/device-handling-security/#pci-s...

> Additionally, Qubes restricts the config-space a VM may use to communicate with a PCI device. Only whitelisted registers are accessible. However, some devices or applications require full PCI access. In these cases, the whole config-space may be allowed. You’re potentially weakening the device isolation, especially if your system is not equipped with a VT-d Interrupt Remapping unit. This increases the VM’s ability to run a side channel attack and vulnerability to the same. See Xen PCI Passthrough: PV guests and PCI quirks and Software Attacks on Intel VT-d (page 7) for more details.

https://security.stackexchange.com/questions/162122/gpu-pass...

"Clarifications on GPU security": https://groups.google.com/g/qubes-devel/c/MeLYpHyLRHQ

But the fact is, even if you are doing GPI passthrough in Qubes, it's much more secure than running any other system.

Basically, yes.

Theoretically, all modern systems have IOMMUs, which are supposed to be able to allow an operating system (or hypervisor in this case) to restrict what a device can do.

In practice, IOMMUs can't be trusted, for two reasons.

First, the implementations (i.e., the actual hardware) are frequently full of unpatchable security holes. It's not just the CPUs themselves that need to be correct; every chip in the PCI chain has to DTRT from a security perspective or risk opening up a vulnerability.

Secondly, particularly with GPUs, many systems have 'magic backdoors' that side-step the IOMMU systems completely. Frequently this is to make certain operations "faster" or "easier".

Basically, unless you have done your own audit and testing of the hardware you own (or someone you trust has done the same thing), you have to more or less assume that a malicious guest with control of a physical device could break out of the system.

In Xen's SUPPORT.md document, we very carefully tried to balance this:

Because of hardware limitations (affecting any operating system or hypervisor), it is generally not safe to use [PCI passthrough] to expose a physical device to completely untrusted guests. However, this feature can still confer significant security benefit when used to remove drivers and backends from domain 0 (i.e., Driver Domains).

"The Qubes people" have a product to develop and maintain. They aren't the single highest authority on secure desktop setups.

Security isn't a black or white issue. There are levels of security. Many tech people want something better than the (very insecure) standard setup on Linux/Windows, but they don't want the Qubes straight-jacket. This means they search or develop alternatives and that is overall a good thing.

I have been using Qubes for a couple of years. I was surprised at his description of his experience. I run it on an ASUS 4-core skylake laptop which has Intel integrated graphics on a 4k screen, and an NVidia thing I keep powered off. The only compatibility problems I find are that (1) ASUS BIOSes and UEFI only just barely tolerate one another, and (2) palm suppression on the touchpad doesn't work. I mapped a key to toggle the touchpad on and off. (It probably would work right with a newer dom0.)

But I haven't seen anything slow. I even play 4k full-screen video in a VM, which does burn 3 cores. I have it maxed out at 16GB of RAM, which is only just barely enough, with ZRAM swap enabled. It allows a VM, when I ask, access to the built-in microphone and camera, as needed, and takes them back. Wifi works, audio works. I keep an external 4k monitor plugged in, and that works. The touch screen can be made to work with trivial fiddling.

I was surprised to find him presenting Wayland code that looked like bad C, but is described as C++. The only C++ish thing about it was using static_cast<T>(e) instead of C-like (T)e. It is not encouraging to find such bad code in a new-ish project (you can lead a horse to water...). Good C++ code would look a lot like his OCaml code.

The only thing I have never got to work, at all, is clipboard operations between VMs. It announces it's doing something with various clipboards, but nothing apparent ever happens.

I would rather see the dom0 running NixOS than ancient Fedora.

> I would rather see the dom0 running NixOS than ancient Fedora.

https://github.com/QubesOS/qubes-issues/issues/1919

> The only thing I have never got to work, at all, is clipboard operations between VMs. It announces it's doing something with various clipboards, but nothing apparent ever happens.

This sounds very strange, has been working flawlessly for me since forever. Perhaps you could fill a bug or ask at Qubes forums: https://qubes-os.discourse.group.

virtwl.ko really is the killer app for Wayland

[Edit: virtwl.ko lets a wayland client inside a VM connect to a wayland server which is on the same machine but outside the VM].

I mean without it, all the noise about Wayland being more secure than X11 is just noise, because Wayland doesn't do remote display, so you have to allow that evil app you're scared of to run on the same machine as the compositor and all your other apps, and you can't even put it in a VM. That's less secure than being able to put it on another machine and use the network. In theory you might be able to run the evil app under another userid, but in practice this often breaks in all sorts of ways because people just don't do that very often so it exposes untested codepaths in UI/widget/GUI libraries. You can't even put the evil app in a VM -- until virtwl.ko came along.

If the Wayland folks want their security claims to no longer be laughed at they need to get virtwl.ko into mainline Linux. And get it supported by more hypervisors than just the one that comes with ChromeOS (crosvm).

I've been using sway on both my desktop and laptop for over a year now, but that doesn't mean I don't get to complain about Wayland. Since leaving X11 I did finally get everything working again (except @#*&^@%@ torbrowser), but I've been underwhelmed with the benefits gained by all the effort I had to go through in order to switch to Wayland. And I still miss remote display always working all the time without special effort from each individual app developer. Every bolt-on remote display option offered for Wayland is a flaming dumpster fire of slowness compared to X11 on the LAN or xorg-xrdp over the public Internet. And the least-flaming dumpster-fire of slowness, waypipe, has turned into abandonware.

> like many Wayland specifications, it seems to be in the process of being replaced by something else.

Funny because true.

Can you explain what virtwl.ko is? I tried looking it up but couldn't find anything even remotely relevant when searching for that term.

Here's the link from the article: https://alyssa.is/using-virtio-wl/

It doesn't really have a better name. It was written by Google to allow Android apps to run inside a VM on ChromeOS. Although it does work with mainline Linux, Google is understandably not motivated to invest effort in branding virtwl.ko with a non-cryptic name, or marketing it as its own project independent of ChromeOS. AFIAK nobody outside of Google was using it until Alyssa (same person responsible for the panfrost awesomeness) got it working on Linux.

As for what it does?

Wayland clients and servers must (a) run on the same physical machine (b) as processes on the same kernel. Virtwl.ko removes requirement (b), allowing the client to be inside of a VM.

Good explanation in the linked article!

This way of using Wayland and VM's seems very interesting and useful. What I'm wondering though, is whether it would be realistically possible to also expose some kind of GPU acceleration to clients in VM's, for example by means of a virtual OpenGL adapter such as VirGL, which itself uses virtio on top of the host GPU driver. This would not get you anything near the performance required for games, but it should be way better than software rendering inside the clients. Do you think something like this would be possible already?

> I mean without it, all the noise about Wayland being more secure than X11 is just noise, because Wayland doesn't do remote display, so you have to allow that evil app you're scared of to run on the same machine as the compositor and all your other apps, and you can't even put it in a VM.

But you can run it with different UID.

Like I said:

> In theory you might be able to run the evil app under another userid, but in practice this often breaks in all sorts of ways because people just don't do that very often so it exposes untested codepaths in UI/widget/GUI libraries.

Regardless, privilege escalation bugs are a dime a dozen these days. Running evil things as another userid is no substitute for moving them into a VM or off of the machine completely.

I used qubes for a while, but found it kind of cumbersome to do most things, and I never got over the hump.

Since then I've been using proxmox, and I'm at the point where I don't use the gui anymore, I just do everything from the command line.

You can do VM things (like run macos in a vm), but I do most things in lxc containers.

It would be kind of nice if proxmox had something like a Dockerfile, but with local containers that didn't depend on going out to dockerhub to pull in and run code.

I have been considering grabbing a Librem 14 and switching to Qubes as a daily. Could you elaborate on what you found cumbersome?

Qubes is very "opinionated" about how things work and provides a functional but very tightly-knit product that cannot be easily modified to suit your own needs. You have to accept their choices like a good nontechnical user does - you can't modify/replace components or use a modified security model easily.

For example, with version 3.2 which I used for a while, one has to use the default disk setup (no ZFS or other cool storage tech) with slow 2layer filesystems, no GPU acceleration for applications in VMs, mandatory encrypted backup scheme. Regarding security, Qubes makes some strange choices such giving the regular VM unix user root privileges accessible via simple passwordless sudo.

Qubes runs only on a subset of available hardware (motherboard has to be good enough) so watch out for that. Also interaction with VM manager and graphically intensive applications in VMs was sluggish and internet/firewall/audio would randomly stop working when restarting VMs and require system reboot to fix. Some of these may be better now with newest version 4.x, but I am doubtful.

> Qubes is very "opinionated" about how things work and provides a functional but very tightly-knit product that cannot be easily modified to suit your own needs. You have to accept their choices like a good nontechnical user does - you can't modify/replace components or use a modified security model easily.

It's worth mentioning that Qubes has good reason for this.

A big part of the purpose for their existence is to help protect journalists and others who would greatly benefit from enhanced security, but don't know how to get there themselves.

Qubes provides a very specific product. Whether it helps to "protect journalists and others who would greatly benefit from enhanced security" depends on user level of knowledge and the ways he uses the computer. If the user does not know much about security, Qubes may help to isolate different tasks in VMs, but that is not a panacea. He may still use it wrong in single VM, for example leave the passwordless sudo on.

Serious security-seeking user needs to educate himself about how the computer and internet works, about operational security, pervasive tracking, typical attacks etc, not just use a product and call it a day. Only so educated user can decide if the Qubes with its benefits/drawbacks is worth it.

Of course. But there is a difference between expecting someone to learn what it does, and why it does it, and expecting them to learn how to implement/modify it themselves.

People have work to do, work that has computers as a tool, not a goal. Lowering the bar from "Learn the exploits" to "Be careful and stay in the lines" is a Good Thing, imho.

Qubes provides an effective framework to say "This is how to use it, this is what you should be aware of, don't do this".

> graphically intensive applications in VMs was sluggish

Just as a datapoint youtube or vlc seem to work well enough. The sluggishness is definitely noticeable though. As a developer it encourages me to optimize the display performance of my web applications. If it performs ok on Qubes it's liquid smooth on my 3+ year old android phone. Take that as you will...

> internet/firewall/audio would randomly stop working when restarting VMs

I started with 4.X and I haven't noticed this. I never actually used 3.X so I can't say if it's something that was fixed or never present on my hw.

> provides a functional but very tightly-knit product that cannot be easily modified to suit your own needs.

This is a fair assessment. But I've come to realize I need the isolation Qubes provides more than I thought I would. No advanced threats or anything. Just the ability to put email, chat, password manager, and work in separate reproducible environments is really cool. Clicking a url on a chat won't open in the browser I'm signed into google or in my work browser. If you have the need for gpg or tor they offer solutions with a high degree of isolation, though I haven't looked into them much.

> Just as a datapoint youtube or vlc seem to work well enough.

As another datapoint, when I installed Qubes about six months ago on a i6600k with 16GB RAM, in the default VM and browser, YouTube was basically unwatchable.

> Regarding security, Qubes makes some strange choices such giving the regular VM unix user root privileges accessible via simple passwordless sudo.

See here for details: https://www.qubes-os.org/doc/vm-sudo/

You can replace passwordless sudo with a secure user prompt: https://www.qubes-os.org/doc/vm-sudo/#replacing-passwordless...

Notice how that argument starts with the conclusion "In Qubes VMs there is no point in isolating the root account" and proceeds to rationalize it, very poorly, by using some cherry picked examples where that may be true.

OK, an attacker capable to attack the hypervisor when running as root may be often "game over" if it runs as regular user. But that kind of attack is so rare, why is it the base of the argument at all?

Meanwhile your browser/other big SV application/script from Github can become root whenever it wants, or by accident. It can send contents of the whole VM over the Internet to anybody. Or an honest bug in it can destroy your whole VM. This is why the unix privilege separation or even more capable MAC systems exist. Seriously, running untrusted applications with root access is a braindead idea.

In Qubes, you typically don't care if your untrusted application compromises the (untrusted) virtual machine. This is the whole point of the virtualization and separating your work into security domains. See also: https://xkcd.com/1200/.

Anyway, Qubes team is not against such kinds of isolation: https://qubes-os.discourse.group/t/isolation-within-the-same....

> In Qubes, you typically don't care if your untrusted application compromises the (untrusted) virtual machine.

For that to be true, one would have to run each application in a dedicated VM. A great idea, but Qubes does not recommend that for efficiency reasons. It does pose performance and usability challenges which is why most people do not use it that way. So yes, you do care about applications in single VM not snooping/manipulating each other. Luckily there are standard unix and MAC mechanisms to isolate them.

You don't run each application in a dedicated VM, but you have security domains, each of which have equally (un)trusted applications.

Also, there are disposable VMs for truly untrusted things.

> I have been considering grabbing a Librem 14 and switching to Qubes as a daily

FWIW, I did the same thing and was unhappy with the result. The Librem hardware wasn't great quality and Qubes really requires a massive desktop to run well. And if you are paranoid enough to need Qubes, then you want ECC memory to defend against side channel attacks.

Not sure what you are talking about. Any details? Qubes runs smoothly on my Librem 15 with 32 GB RAM.

So like, you are using Proxmox as your main OS/desktop? And then doing application (eg firefox for browsing, etc) things in in proxmox LXC containers?

If so, that is pretty interesting...

Although, using LXD with Ubuntu is totally painless and easy as well..

Actually, it went sort of like this:

I used qubes as a desktop os, but it was a little clunky. Like the copy/paste or moving stuff between the vms.

But the main problem was I couldn't ssh INTO the qubes machine, and I couldn't really run server kinds of things (though maybe it was possible to dig into the networking and the firewall vm).

When I tried proxmox, it was more set up for server kinds of things. For example, all the VMs showed up as machines on my local network. I started to make it a desktop, but then backed off. I decided to leave the debian main os vanilla. Instead I made it a server. And I got away from VMs and use containers.

I do have a macos vm that I remote into though. It's a copy of my laptop which I haven't used as much the last year.

I tried running proxmox on my server for a while but I found it was more complex to do everything and I wasn't gaining any benefits from it since it looked very enterprise and multi machine focused. These days I use fedora coreos with podman/docker. I'm trying out virtual machines using libvirt and then virt manager on my desktop for control but I have not been able to find a way to do bridged networking.

Bridged networking in libvirt can be a bit of a pain but I've found that routed works pretty well. Also - if I'm not mistaken - libvirt's netfilter firewalling isn't available on bridged networks.

Is it possible to ssh in to a vm in a non bridged network? I thought you had to give the vm its own local ip on your network.

Sorry, I didn't see that comment earlier!

Routed turns your virtualization host into a router for a separate network on which your VMs live (afaik). So in order to reach that network you have to define static routes that ultimately lead to the virtual interface set up by libvirt.

When you start your routed network (with net-start), libvirt should set up a static route on the host machine automatically. You can see this route with "ip route". To make it available to all hosts on your network, you can configure a static route on your (physical) network's router that covers the IP address range of the virtual routed network and has the (physical) IP address of your virtualization host set as the gateway.

For example: Assume your virtualization host has 10.0.0.10 as its address in your "real" network and your virtual routed network covers 10.0.1.0/24. You would have to set up a static route on your "real" router that says "route 10.0.1.0/24 to 10.0.0.10".

Libvirt automatically sets up a route in the local routing table of your virtualization host that says "route 10.0.1.0/24 to virbr0" (or whatever your network's virtual interface is called).

Then, if everything worked, you should be able to connect to your VMs in 10.0.1.0/24 directly.

proxmox security is pretty thin last time i checked. everything in dom0 runs as root...

This is awesome! I've been using Qubes for 5 years. I really want to use Qubes as my daily driver. I continue to find that it's not reasonably usable. My issues are the same as the author:

1. Poor hardware compatibility - due to clinging to Xen (dead/dying)

2. Poor performance (youtube @480p or xfce desktop @4k run at 5fps on a high-end system, don't even try 3D accleration) - due to not treating performance needs seriously[1]. GUI domain[2] and hiring a community manager are signs that they're starting to listen, but it looks like it will take years to land in full.

I pray that Qube's devs are taking this project (and others[3]) as a wake-up call: there is a huge unfulfilled need that people are willing to do a lot to satisfy. Leveraging more modern projects (like crosvm & Wayland) can hopefully help reduce the cost ("evolve or die").

[1]: dozens of mailing list threads reflecting a stance of "it's the users who are wrong"

[2]: https://www.qubes-os.org/news/2020/03/18/gui-domain/

[3]: https://spectrum-os.org/ "a step towards usable secure computing" (an obvious wink to qubes), https://github.com/nrgaway/qubes-kvm-dev, etc.

The part about the qubes dom0 being 'outdated' is worrying, because one of the important things to think about for good Xen linux-on-linux (PV) performance is to use a recent kernel on the dom0, and recent version of xen.

Even ultra conservative debian stable (buster, right now) uses something fairly up to date, and xen 4.11. I'd be concerned about using anything older than that.

Qubes OS provides its own kernel for dom0, which is more recent than where the Fedora distro in dom0 stopped.

https://qubes-os.discourse.group/t/outdated-dom0-is-it-a-pro...

> it would be nice to be able to do video conferencing these days

Works fine for me, except bluetooth (see below). What's the problem with it?

> bluetooth would be handy

A dedicated Audio VM is coming in 4.1: https://www.qubes-os.org/news/2020/03/18/gui-domain/#audio-d.... Related issue: https://github.com/QubesOS/qubes-issues/issues/1590.

Is this really a good idea considering the security issues with Wayland?[1]

[1]https://github.com/Aishou/wayland-keylogger

That "security issue with Wayland" is really loading an untrusted .so file from your home directory, which is exactly what running apps in KVM VMs instead (as the article proposes) would fix. So yes, it's a good idea!

In addition, the author states that similar techniques would also work on Windows and Mac, and any platform without sandboxing... which would include most installations of X, unless there's something I'm missing.

All installations of X. X has no sandboxing. What most of these clickbait articles or concepts fail to mention is that at the very worst, you can get the same level of access that was possible before the sandboxing was added.

If an app has write access to your home folder, it has root. But with flatpak and portals, its realistic that direct home dir access will no longer be a thing for most apps.

Good point. I was imprecise in my wording.

I meant that most operating systems using X as the display server are not going to have application sandboxing.

I'm pretty sure the top level comment here was just FUD, although I assume it was well intentioned, just misunderstood.

Aye, there's like 3-6 ways to inject code into applications on Windows, and then LD_PRELOAD (and equivalents) and attaching as a debugger on *NIX platforms. There's no way a display manager, audio server, etc. can protect themselves from clients from code injected into them - outside of completely disabling these functions that allow it (and certainly have valid use cases, if much less often than illegitimate ones).

This is not a wayland security issue. This is the equivalent of calling a car key lock insecure just because someone can drive your car when you give them your key.

Reply on news.ycombinator.com