Hacker News Reader

Comment by gonzaloalvarez | Hacker News Reader

> One of the significant benefits of vDPA is its strong abstraction, enabling the implementation of virtio devices in both hardware and software—whether in the kernel or user space. This unification under a single framework, where devices appear identical for QEMU facilitates the seamless integration of hardware and software components.

I honestly don’t know what this means. Is it faster? Is it more secure? Why would I use this vDPA thing instead of the good’ol virtio blk driver? Looking at the examples it certainly looks more cumbersome to setup…

View on HN

I don't get it either, and I'm a maintainer of SPDK which provides multiple implementations of virtualized devices and is frequently used inside DPUs to present storage devices.

If I'm implementing a hardware device anyway, why would I not just use NVMe as the interface? NVMe is superior to virtio-blk in every way that I can think of.

Even for a software device in userspace, why not use a technology like vfio-user to present an NVMe device, or just use vhost-user to present the virtio-blk device?

I've never really been able to get a clear value proposition for vDPA for storage laid out for me. Maybe I'm missing something critical - it's certainly possible.

Another top-level comment has covered this at least with regards to networking, hasn't discussed the storage side as much: https://news.ycombinator.com/item?id=39355644

Yes, I've seen some clearer cases made for networking. In networking there is no standard for the hardware interface. Every vendor does their own thing. Except many can at least handle virtqueues carrying virtio-net messages for the data path, so some framework like vDPA may make sense (I'd prefer to see a full NIC interface standard emerge instead).

In storage, however, the industry has agreed on NVMe. This is a full standard for control and data plane. All storage products on the market, including DPUs and SmartNICs, just present NVMe devices. So there's no case to be made for vDPA at all. It just isn't necessary.

Yes, I see your point and agree that NVMe can be used for the same purpose. But several HW vendors have implemented virtio-net devices in their SmartNic and may find it convenient to support virtio-blk to reuse most of the building blocks.

As for vhost-user, it's perfect for VM use cases, but with containers or applications in the host, it's not easy to use. Whereas, a vDPA device (HW or SW) can easily be attached to the host kernel (using the virtio-vdpa bus) and be managed with the standard virtio-blk driver.

Software block devices let you do all implement all sorts of disk-like things in userspace. nbdkit is a more flexible version of this (also compatible with qemu), so to get a flavour of the types of things you can do just peruse the list of nbdkit plugins: https://libguestfs.org/nbdkit.1.html#Plugins

Ceph Block Device 3rd Party Integrations: https://docs.ceph.com/en/latest/rbd/rbd-integrations/ : Kernel Modules, QEMU, libvirt, Kubernetes, Nomad, OpenStack, CloudStack, LIO iSCSI Gateway, Windows, qemu: https://docs.ceph.com/en/latest/rbd/qemu-rbd/#qemu-and-block...

`cephadm bootstrap` requires docker or podman and ssh: https://docs.ceph.com/en/latest/cephadm/install/#bootstrap-a...

Ceph Object Gateway: radosgw: https://docs.ceph.com/en/latest/radosgw/ :

> The Ceph Object Gateway provides interfaces that are compatible with both Amazon S3 and OpenStack Swift, and it has its own user management. Ceph Object Gateway can use a single Ceph Storage cluster to store data from Ceph File System and from Ceph Block device clients. The S3 API and the Swift API share a common namespace, which means that it is possible to write data to a Ceph Storage Cluster with one API and then retrieve that data with the other API.

virtio-blk is probably faster, but then do HA redundancy with physically separate nodes and network io anyway; or LocalPersistentVolumes

S3 or radosgw are not very relevant to a discussion about block storage devices.

Ceph's RBD is "special" in the sense that the client understands the clustering, and talks to multiple servers. If you wanted that in the mix, you'd have to run a local Ceph client -- like the Ceph software stack exposing block devices from kernel does. The only way I can see vDPA being relevant to that is to avoid middlemen layers, VM -> host kernel block device abstraction -> Ceph kernelspace RBD client. But the block device abstraction is pretty thin.

The real use case for vDPA, when talking about storage, seems to be standardizing an interface the hardware can provide. And then we're back to "why not NVMe?".

(Disclaimer: ex-Ceph-employee)

Reply on news.ycombinator.com