Vulkan video shenanigans – FFmpeg and RADV integration experiments

Given how long it took for VA-API to be implemented, Vulkan might be outdated by then.

mpv uses Vulkan for some filters, but access to hardware accelerated video decoding in it happens through VA-API currently.

See: https://www.khronos.org/blog/an-introduction-to-vulkan-video

An (overengineered?) API to access HW to decode and encode video.

I have looked right now into the mpv Github repository. Not even the latest mpv version appears to support any kind of Vulkan video decoding.

Your mpv probably uses Vulkan as a graphics backend (--vo=gpu --gpu-api=vulkan), but VA-API for video decoding (--hwdec=vaapi).

The Vulkan video decoding/encoding API has just been standardized a few weeks ago, but without the AV1 functions, which will be added in a revision of the standard, later this year.

Yet another way for Khronos to try to make a living out of C based API standards, from the members that pay into the party.

So when you decode using hardware acceleration the library you're using usually handles writing the output to the backing store which lets you get performant hardware accelerate video inside a compositor. The problem when you're trying to use it as a texture inside Vulkan is that, best case scenario, the library then has to copy the data out to system memory, then provides a pointer to that buffer to your Vk app, then you have to copy it back into texture memory. Worst case scenario the library falls back to CPU decoding because of the final destination of the frame being system memory.

Vulkan video is going to allow that video decoding to render straight to a Vulkan texture in texture memory bypassing the copy out and copy back in. It'll be far more performant than having to copy it twice across the PCIe bus taking up valuable system memory bandwidth to the CPU doing literally nothing.

TBF, integrating video playback with 3D rendering isn't such a great deal if you can get decoded video frames somehow for uploading into a texture (trivial but expensive with open source video codecs running on the CPU for instance).

For integrated platforms like the browser, access to decoded video for 3D rendering has been working for a long time with WebGL (you can simply hook up a HTML video element to a WebGL texture, e.g.: https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API/T...

...or in WebGPU: https://austin-eng.com/webgpu-samples/samples/videoUploading

I guess the point of the Vulkan video extension is that it connects the Vulkan API to proprietary closed-source video decoders running on the GPU though.