Virtio-fs: shared file system for virtual machines
lore.kernel.org
lore.kernel.org
Block storage (virtio-block) is way simpler than file systems, both in the virtio implementation and in the host filesystem implementation. Simple is good, both for security reasons and because it makes it easier to reason about performance isolation between guests. The cost is obviously in features, and in making it hard to share data (and very hard to multi-master data) between multiple VMs. To see how simple virtio-block is, take a look at Firecracker's implementation (https://github.com/firecracker-microvm/firecracker/blob/mast...).
The existing alternative, as the post talks about, is 9p which has a lot of shortcomings, so this is interesting. I suspect it's use is mostly going to be in development environments and client-side uses of containers, and server-side users of virtualization will likely stick with block (or filesystem implemented in guest userland).
Containers don't need this. It's only useful for virtual machines.
The advantage compared to running normal VMs is that it's interoperable with things that expect a container runtime like Docker or rkt - e.g., Kubernetes can run Kata Containers - and the overhead / density / startup speed is much closer to that of actual containers (in the sense of namespaces+cgroups) than traditional VMs.
Whether that’s something you care about or think is necessary is left as an exercise to the reader (i.e. it depends).
Containers are not virtual machines.
I imagine a given pool of blocks is reserved for one of VM at a time. I wouldn't want to share the storage live between two VMs unless it was being accessed by some application specifically designed to hit a shared block store.
The point of something like virtio-fs would be to allow multiple VMs to share a file tree. It is a compromise between bind mounds (which are simpler, but only work for containers) and network filesystems (which work remotely, but are less efficient and well behaved).
I even looked at writing my own 'vagrant' (to fit exactly my use case of Linux on Linux via KVM and deployed via ansible) because the NFS thing and VirtualBox issues mar what is otherwise a great idea.
vagrant-libvirt (3rd party): https://github.com/vagrant-libvirt/vagrant-libvirt
ansible provisioning (built in): https://www.vagrantup.com/docs/provisioning/ansible.html
This isn't to say these will make the guest filesystem more performant than NFS; you'll probably hit the same bottlenecks with mounting host filesystems into the guest. The solution many users arrived at with Vagrant is to use a user-space synchronization program like Unison that watches for filesystem events (eg, inotify) on both the guest and the host. This is its own can of worms: O(size of files monitored) resource use on both hosts, file ignore lists, async races, etc. There's even a Unison plugin for vagrant, but I found it quite finicky and thus unsuitable for large-scale deployment
unison: https://www.cis.upenn.edu/~bcpierce/unison/
unison-vagrant2: https://github.com/dcosson/vagrant-unison2
At Airbnb, we eventually wrote our own wrapper around Unison to make it easier to use, and saw 90x better perf for filesystem-heavy loads like Webpack once we switched off of NFS in our local VMs.
Funnily enough webpack is where I had the most problems (it certainly surfaces "what happens when we smack the FS in the face repeatedly" issues better than most) which is what got me thinking down the KVM route in the first place.
In terms of hacking around on vagrant, it's ruby which isn't on the list of languages I'm fluent in and its flexibility makes it more complex than what I need for my use case.
There is something to be said sometimes for rolling your own for your own itch but I'll certainly have a look at unison as part of vagrant first.
Is there any chance you guys open sourced this wrapper, or have any plans to? :)
1. I no longer work at Airbnb, so I can't help you there.
2. The wrapper is somewhat deeply embedded in the dev tools swiss army knife, and is written in Ruby, so (a) it's difficult to extract since its tied with other Airbnb specific concerns and support libraries, and (b) distributing perf-sensitive Ruby code to end-users is challenging so you might not want it anyways.
You might want to take a look at Mutagen, a unison replacement written in Go which seems to avoid most of what makes Unison annoying to work with. I haven't tested it, though, so I can't vouch for it.
(And while I'm being curious of those folks, what their expectations are around virtio-wl going forward, upstream, etc?)
I never found a solution other then to upload it somewhere on the internet and then download it on the host.
Yes, I know you can probably configure something before starting the VM to make VM and host share files.
But I usually want to have them completely sepereated. Only when the need arises to transfer a file, I would like to do so ad hoc.
Is there a solution?
You can expose GlusterFS directory trees to virtual machines using virtio-fs but virtio-fs itself doesn't do the network communication or policy decisions about where and how data is stored on the network - that's the job of GlusterFS.
The reason you might want to use both together is to hide the details of the GlusterFS configuration from the virtual machine. That way the virtual machine only sees a virtio-fs device and doesn't have network access to your GlusterFS cluster.
Can't we modify syncthing so that it becomes like a low latency shared filesystem and uses bitttorent as an underlying protocol.
Or is the bitorrent not fit for making a shared/distributed file system?
Well... at that point you're writing new software, so you might as well start from the ground up with something actually designed for this use case, which provides support for things like live coherency, writeback and caching.
It's not easy to just make a program "become" low-latency. Syncthing is extremely high-latency to begin with, and is specifically not meant to be used as a real-time sync application which multiple applications can depend on for live coherency. This is the design of Syncthing. Same for bittorent. Tons of overhead due to the design of the system. For something like this which requires true low latency, there are just too many design decisions which would get in the way.
One major pain point about Syncthing in particular is that there isn't a very good way to maintain complete consistency between different OS environments. Syncing files from Linux which are not valid in Windows (name, metadata, etc) either causes filesystem corruption, reduces coherency between guests and host, or straight just drops incompatible files.
Try copying a file from Linux to Windows which has a trailing space. Windows will go bonkers and cannot delete or modify this file. You have to move it to a partition which you then delete.
I was so frustrated with using Syncthing that I gave up trying to make it useful.