Vulkan video extensions for accelerated H.264 and H.265 encode

Do CPUs have fixed video decoders? I thought those were only included in GPUs, including iGPUs.

VAAPI is also using the GPU. The fixed-function video decoders are on the GPUs, not the CPU. That just happens to be using the iGPU on Intel so you don't need a discreet GPU necessarily, but this is also missing if you have a CPU-only sku like the -f suffix parts

The only difference is the vulkan spec is _in theory_ cross-platform. However, since MacOS doesn't support Vulkan and MoltenVK doesn't implement this extension (and probably won't), and this extension also isn't implemented on Android, then your scope of actual availability is quite pathetic. So it probably doesn't make sense to move off of VAAPI regardless unless you really want Linux & Windows to share a codepath, even though you have something OS-specific anyways for MacOS/iOS & Android.

Vulkan Video Extensions use the same physical hardware as VAAPI, but with a little bit less of forcing you to do codec-specific nonsense. It should be better in that it's much easier to get a zero-copy Vulkan texture from the decoded frame.

The GPU has also the fixed function video encode like integrated CPUs. Both VAAPI and Vulkan Video address the same hardware. Differences are in the API itself (with Vulkan being more explicit) and potentially in which vendors (or OSS driver developers) support which API in what degree.

Could anyone elaborate on the difference between software decoding and CPU decoding? Software decoding uses the CPU right?

I think not being tied to VAAPI reduces the chain of development (like you don't need to wait for Intel to add something first since they are the primary developers of VAAPI in general).

> nvidia drivers that have DirectX 12, but not Vulkan

not a thing. at worst you get a situation where the ICD loader isn't installed

I don't really care much about Windows and macOS. But I'd like to see it using Vulkan video on Linux in addition or instead of VAAPI.

As pointed out elsewhere, it's not possible on macOS either way. Apple are too stuck up not supporting Vulkan and it won't work with MoltenVK.

How can I confirm whether Firefox is actually using hardware decode for videos on my machine for a given codec? Is there a magic keyword I should search for within the about:support page, or is it recorded somewhere else? Mine shows for instance "VP9_HW_DECODE default available", does this means it's using hardware decode for VP9, or merely that it might be possible for my Firefox version?

Whilst Firefox may support hardware video decoding, Mesa since March 2022 disables patent encumbered codecs by default[1], and distributions such as Fedora and OpenSuse do not explicitly enable these patent encumbered codecs to avoid possible legal problems. Even Gentoo (built from source code by the user) requires the user to explicitly enable a USE flag (proprietary-codes) to use patent encumbered codecs.[2]

The thought process is that AMD, NVIDIA, Intel and the likes are not providing a patent license with their hardware.[3] They are instead just supplying part of an overall system that together with operating system kernel, display manager software, video player software, etc allows the decoding and encoding of patent encumbered video files. Open source software projects and distributions are concerned they'd be found to be infringing patents by enabling a complete solution out-of-the-box. Hence they put some hurdles in place so that a user has to go out of their way to separately piece together the various parts to form a complete system capable of encoding and decoding patent encumbered codecs.

edit: To clarify, if your Intel or AMD GPU/APU supports a patent-free codec such as AV1 (most GPUs/APUs available for sale?), Firefox on a standard Linux distribution will use hardware video decoding out of the box by default for the patent-free codec. The issue is really one of whether you're sourcing content from a provider that uses a good choice of codec like AV1. The good news is that patent trolls are doing a good job of pushing laggard content providers down this path.[4]

[1] https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15...

[2] https://github.com/gentoo/gentoo/commit/1265a159743d7f07185a...

[3] https://lists.fedoraproject.org/archives/list/devel@lists.fe...

[4] https://news.ycombinator.com/item?id=38249527

My bad, that's what I get for commenting before bed. I meant to tack on an "on Nvidia" in there. AMD and Intel are well supported with VAAPI, but I've never managed to get any of the NV-to-VAAPI wrappers and shims working.

You can make the same argument for VDPAU. AMD officially supports it, and there is an unofficial translation layer with limited capabilities for some Intel GPUs. Is VDPAU not a vendor specific API anymore then?

Intel, AMD and NVIDIA have their own vendor-specific video APIs, and even when they provide official support for the API of another vendor, it tends to expose a limited subset of the full functionality (like the list of available codecs and encoding features).

You are free to call these vendor specific APIs for what they are or something else, but the reality has been that there is no single video API officially supported by Intel, AMD and NVIDIA. This changed with Vulkan Video.

But Vulkan Video isn't just about desktop: mobile devices, Raspberry Pi, etc. are expected to get on board with it eventually, just like they did with Vulkan.

> It's supported by the open source drivers for GPUs from all 3 vendors.

Which 3 vendors are you referring to? Intel, AMD, and who?

> It's just that the open source drivers for Nvidia cards are not very practical and the proprietary drivers only support vdpau and nvdec/nvenc

Why are you bringing up open source drivers, and what is not practical? Both official open source drivers (open-gpu-kernel-module) and unofficial open source drivers (nouveau, through binary firmware) support VDPAU. However, NVIDIA's drivers (open source or binary) does not support VA-API.

Right, this is yet another instance of Nvidia not wanting to play nice with the other kids.

I'm always annoyed how any Linux media player or encoder needs to bring its own entire media operating system, down to each individual nut and bolt.

On Windows there's Windows Media Foundation and DirectShow that centrally manage everything and also support the "individual nut and bolt" approach. Android has its own central thing (MediaCodec?) that must be used. MacOS and iOS presumably have their own central manager (Quicktime?) too.

But Linux? It doesn't serve as an operating system for media. It's tremendously inconvenient as an admin/user rather than an evangelist.

What is "incorrect"? The full sentence that you conveniently chose to cut in the middle before quoting (apparently to fit into some pessimistic forecast about the significance of Linux desktop) reads

> Firefox currently supports hardware video decoding with Intel's vendor specific VA-API only on Linux, which is not supported by NVIDIA.

(emphasis added)

You further wrote:

> Firefox uses Windows Media Foundation, which is cross-vendor, on Windows. It uses MediaCodec on Android which is again cross-vendor.

And? None of those APIs are cross-platform. Vulkan Video will eventually allow developers (including Firefox developers) to write a single code path for video to cover a wide range of platforms and vendors (likely with the exception of walled gardens like Apple-land, although someone might find a way to support like via a wrapper like MoltenVk for Vulkan).

Also, to get best compression you don't want to do key frames at even intervals, but rather when it pays off (ie large changes between two frames, like a scene change).

Convolutions and Discrete Cosine Transforms and many matrix operations (rotations, shifts) depends on operations around and ARE parallelizable.

It's the non-linear-algebra stuff that is usually problematic.

It means that e.g. in H265 you can split the frame into tiles that are much larger than macroblocks: it's like the video is a set of n videos that are then composed to build a complete frame. This allows both encoding and decoding to work in parallel even inside a single frame. In addition, in VR 360 context, it allows decoding only frames the viewer can see.

It reduces the compression efficiency, but only a little, because the tiles will be quite large anyway, such as 6 tiles per frame (but can be more for 360 video applications). It can be limited by the number of decoding instances a piece of hardware can have concurrently.

AV1 also has "tiles". This is basically slicing each frame into N separate frames and encoding each mostly separately. You lose a little efficiency (as you can't exploit cross-tile redundancy very well or at all) but unlock a lot of parallelism on both encode and decode.