Khronos Releases AV1 Decode in Vulkan Video with SDK Support for H.264/H.265 | Hacker News Reader

Khronos Releases AV1 Decode in Vulkan Video with SDK Support for H.264/H.265

khronos.org

164 pointsdoener66 comments

Does this mean they decode AV1 using a compute shader as a fallback for GPUs that don't include an AV1 ASIC?

That's up to each driver to decide. Support for the extension isn't mandatory, it's just meant to expose the hardware on GPUs.

There's no shader fallback included, it's just a extension spec for the accelerator interface.

How useful are hardware decoders? My understanding is that they are not necessarily faster than software decoders (in my experience it is the opposite), so the only benefit I can see is less CPU consumption. I see the benefit when you run on a battery (laptop / smartphone) but on a desktop or server, unless you critically need the CPU at that time, I am not sure how useful it is.

That's exactly what makes them useful. Energy consumption is vastly reduced.

>I see the benefit when you run on a battery (laptop / smartphone) but on a desktop or server, unless you critically need the CPU at that time, I am not sure how useful it is.

Yes it is. Very useful. The vast majority of people use laptops now, not desktop computers with 120mm tower coolers anymore, and if you want to switch to Linux then not having video decode nuke your battery life and turn your tiny fans into a Pratt & Whitney, then this is another welcome change to closing the gap to Windows.

People, especially on the Linux side, need to stop viewing the world as if everyone still has desktop towers PCs tucked under their desks while disregarding the software issues and deficiencies that only plague laptops, which are the majority of the marketshare and have vastly different constrains than desktops.

Software video decoding isn't some monstrous workload, though. We're not in the era of Pentium 4 and h.264 decoding drowning every CPU out there to a screeching halt anymore, the CPUs of today hardly acknowledge the load even in the low-end tiers.

The benefits of hardware decoding are lower power consumption (read: electricity bills) and offloading the work to dedicated co-processors freeing up resources on the CPU proper, which are indeed far less relevant on desktops and even laptops which can just pack big batteries.

The biggest beneficiaries are by far mobile devices and really low power hardware. Computers which either cannot practically carry more batteries or which use hardware that simply aren't performant for more important reasons.

Desktops and laptops meanwhile are better geared to benefit from the perks of software decoding: Better quality and versatility, and fine-tuning of filters and rendering paths.

With all due respect, I think you must be out of touch if you don't acknowledge the negative effects of lacking HW decode on video playback and content consumption on regular notebooks. People do notice this even if they don't know where the fault comes from.

Even on a modern Ryzen laptop APU, if I run H265 or AV1 4k decode in software the CPU usage is vastly noticeable. On my older Intel i5 6300HQ they're pegging it at 100% .

I watch software decoded video all the time because I like watching anime. I have never, ever had any problems for quite literally over a decade thanks to how powerful CPUs have gotten.

Way back 20 years ago, you would have had a point. But today? CPUs are several orders of magnitudes of overpowered for the task.

>. I have never, ever had any problems for quite literally over a decade thanks to how powerful CPUs have gotten.

I wasn't talking about having problems decoding, I was talking about software decode screwing your battery life due to high CPU usage on certain streams. Which is a problem, but not something most people who don't have laptops care about.

Do you watch your anime on a desktop or a laptop on battery?

I watch on battery if I'm out traveling.

The very slight increase in CPU processing and thus power use doesn't bother me at all. The screen takes more power than the CPU.

I habitually have Task Manager minimized in the tray area because I like monitoring my CPU usage at a glance, and decoding video in software simply does not even register as a noticable blip anymore.

So yes for 720p low bitrate stuff it's not so much of a difference, but as others said watch some real content in 1080p or better 4k and your CPU will definitely break a sweat and the fans spin up.

Now that I think of it with your claim of it not even registering as a blip in taskmanager I'd actually argue you're probably using hardware decoding without noticing, because I wouldn't know which player (esp if on Windows) wouldn't make use of it.

mpv (definitely the best player for advanced users) does not use it by default. I'll simply quote `man mpv`:

> Hardware decoding is not enabled by default, to keep the out-of-the-box configuration as reliable as possible. However, when using modern hardware, hardware video decoding should work correctly, offering reduced CPU usage, and possibly lower power consumption. On older systems, it may be necessary to use hardware decoding due to insufficient CPU re‐sources; and even on modern systems, sufficiently complex content (eg: 4K60 AV1) may require it.

I am definitely not using hardware decoding.

My setup is Media Player Classic Home Cinema, splitting and decoding via LAVFilters, then fed through ffdshow for some filters, before finally rendering on Enhanced Video Renderer.

Nowhere in the pipeline is the GPU involved as far as decoding is concerned, it's all deliberately software on the CPU.

Let's talk specifics. What resolution, what codec, what CPU, what usage.

You're pretty vague so far, so it could be that you're watching a software decoded 360p h.264 video on a Threadripper, in that case I agree.

It will be a very different situation when you try to watch 4K h.265 / AV1 video on a couple of years old laptop dual-core CPU.

Generally 8-bit or 10-bit h.264, occasionally h.265, at either 1280x720 or 1920x1080 progressive with frame rate usually 23.976. Split and decoded in software via LAVFilters then ran through ffdshow before rendering on Enhanced Video Renderer.

CPU anything ranging from an i7-14700K to an i3-2100 (yes, Sandy Bridge). Seriously, decoding video has never been a significant workload for over a decade.

>4K h.265 / AV1 video on a couple of years old laptop dual-core CPU.

Kindly, why the hell would I even watch 4K video on a laptop? Y'all keep throwing out contrived situations like that, meanwhile I'll be a sane man living in reality and re-encode it (in software, finer tuning of parameters) in my spare time down to 1080p or 720p so I save myself precious disk space and CPU usage while travelling.

> Kindly, why the hell would I even watch 4K video on a laptop?

Because that's the file I have on hand. Why on earth would I re-encode it if I want to watch it once?

I also connect my laptop to my 32" 4K screen. There the battery life is not a consideration, but the spinning fan is.

> in my spare time down to 1080p or 720p so I save myself precious disk space and CPU usage while travelling.

Your use case might work for you, and that's fine, but you claim that software decoding is universally not a problem. But it is, if you don't limit yourself to 720p h.264 pre-encoded at home. Most people are not fine with having to do this and having to limit themselves to low resolutions / bad IQ.

> Kindly, why the hell would I even watch 4K video on a laptop

How about if you have a 4K monitor plugged in? Or if your notebook display is itself 4K (which is a completely valid configuration nowadays)?

I have a pretty beefy laptop with an RTX 3080. I regularly watch BDRips that exceed 50+ Mbps, and software decoding even on my 8-core Intel Xeon will cause some stutters. Hardware decoding is just so much faster.

Decoding horsepower required increases sharply with increase in resolution/framerate. 720p30 doesn't cause my fan to even run but 1080p60 does and 4K30/60 drops frames.

Just to be clear, 4K is 9 times the load of 720p. 4K60 is 18 times.

Anime is like 480p, low fps video isn't it?

Otherwise even my desktop struggles when it comes to 4K60 video decoding on the CPU.

Definitely not. Many anime release groups are early adopters of new video coding standards. It was the first popular media that saw heavy use of AVC back in probably 2006 or so, then HEVC (roughly in 2014 or thereabouts), then AV1 (since ~2021), and sometimes even VVC, although that's currently being held back by a lack of support in mainline ffmpeg.

It was also the first to start using 10 bit color profiles.

Some release groups are particularly insistent on producing high quality releases and use FLAC for audio (even if the source tracks were in lossy format), and very high bitrate video.

Why would a type of animation dictate resolution and framerate?

Decoding has a component that is proportional to the bitrate. Anime has a much lower bitrate than say a 4k bluray movie. The truth though is somewhere in the middle, as recent CPUs are getting to a point where they can do all the heavy decoding on their own, but older models (that are still widespread) still struggle.

But also I understand the GPU is using a video encoding/decoding chip which will have a fixed performance whatever the model of GPU within a generation, whereas software can fully utilise your CPU, and scale up if you run multiple streams in parallel. All of that is irrelevant for real time decoding, but for converting files, my experience if you are using hardware encoding is that hardware decoding can become the bottleneck and you may get faster performance with software decoding + hardware decoding.

Av1 consumes most modern CPUs at 4k+. Sure if its 264 with 1080p res you haven't had issues.

And its made even worse nowadays to properly support video conferencing, as often you need to decode multiple streams (video from multiple participants) as well as encode one or more (camera + slides/screen) streams from your end, at the same time, in real time.

Haha, a normal 1440p / 4K YouTube video is a pretty monstrous workload for my Intel Core i5-8259U CPU at least! I get by completely thanks to the Intel Iris Plus 655 HW decoder. Otherwise my Intel NUC would indeed be a screeching, hot mess, and then I'm only talking of H.264 and H.265.

You have to consider hardware limitations, you want maximum battery (thus minimize power burn, by reducing CPU and GPU load), and that can only be achieved by optimizing for the highest possible speed.

Battery life is everything, no one cares about electricity bills on a laptop or phone, but they do care if the phone or laptop dies sooner than it has to.

> People, especially on the Linux side, need to stop viewing the world as if everyone still has desktop towers PCs tucked under their desks while disregarding the software issues and deficiencies that only plague laptops, which are the majority of the marketshare and have vastly different constrains than desktops.

Linux (especially the kernel) isn’t even tuned for towers. It’s tuned for servers.

The defaults make horrible optimizations all in the name of throughput > latency.

I can tell you from my Chromebook and Macbook experience, HW decoders and encoders are the difference between a productive remote video meetup with good collaboration, and a frustrating lagged glitchy suttering meet with no work done, if CPU only.

They're usually substantially faster. Many low end devices can't do decode on cpu at all well enough to say play YT. Add hw decoder and magic.

Re desktop use - even there the energy efficiency is a win, especially for continuous use. e.g. I've always got YT open

Software decoding 4K x265 requires insane resources. You could live transcode it to 1080p quite reasonably on 1 average core, but simply decoding it requires ~3 average cores

how come decoding is more work intensive vs transcoding ?

With JPEG you can scale down in power-of-two factors by just not processing the higher-frequency components.

You're still left with the non-lossy compression step, so one could do something like "unzip", repackage data throwing away the high-frequency stuff, "zip".

Wouldn't surprise me if you can do something similar with h264/h265.

I'd be surprised if that worked for interframe compression like video but even if it still did, and extremely well, encoding to the resulting format still has to fit in and that's where the majority of the work should be even when you just decoded the source normally.

I think they mean transcoding as an additional step upon already decoded frames.

For encoding, it's like riding a bus vs pushing a bus.

AWS sells a $100k 1u encoder box to broadcast companies. They buy that because there's hardly a 2nd best to speak of.

Encoding 15min of 8k source material on i9 9900k (that was 2019 and THE fastest CPU on the market) took me 19h.

Same for decoding, any mobile device will drink battery like it's water on Sahara.

Video is unusable without HW acceleration.

Encoding is very different. Most codecs are designed to be slow to encode, fast to decode. HW encoding is day and night vs software. But I see ffmpeg mentioned in the post. Switching to hardware decoding is likely to slow down your pipeline.

> Switching to hardware decoding is likely to slow down your pipeline.

If thats your experience, then something is off with your setup.

Hardware decoding on a modern computer might seem equally fast as software until you look at power usage where hw decode is a hard win. Even so, the resources used to decode modern high end codecs like x265 or av1 will be major and hard to miss.

but anyway discussed are encoding &decoding.

And video encoding goes from like 3 fps in software to 100s of fps in hardware, making it possible at all with any resolution and quality to speak of.

You need encoding to stream your video, like in a video meeting.

Maybe I am doing it wrong. But I tried the following commands to test it on h264/h265 and 1080p/4k sources:

  Soft: ffmpeg -i {src}  -map 0:v -f null - 
  Hard: ffmpeg -hwaccel cuda -i {src}  -map 0:v -f null -

the result I get is:

  Source       Hard  Soft
  1080p-h264  25.1x 37.1x
  1080p-h265  26.4x 36.5x
  4k-h264      5.5x  7.1x
  4k-h265      5.5x  6.5x

this is on an old CPU (i7-6700K) and a somewhat recent GPU (RTX 2060) and with ffmpeg 6

I am pretty sure that does not activate the hardware encoder. https://trac.ffmpeg.org/wiki/HWAccelIntro

We are talking about decoding here. You meant decoder? I can certainly see the hardware decoder being busy in task manager.

I think you should use nvdec instead of cuda. GPU itself is not a hardware decoder, the special decoder unit inside of it is.

I barely know what I'm talking about, but in my experience to use software I had to explicitly enable it with "-allow_sw" but this may have only been necessary due to videotoolbox

Cuda is not a normal hardware decoding API. Look for VA-API and VDPAU.

You are using cuda for some reason? cuda is for general gpu computing, not the same as using the dedicated hardware enc/dec engines.

See ffmpeg docs for the hw decoding name, like nvdec iirc for nvidia cards.

What type of "setup" are you talking about? Servers or home use?

I've been working with video transcoding/broadcasts a lot and software decoding was still worth it in large amount of cases - mostly because the CPUs these days (threadrippers & co) can handle significantly more concurrent encodes than the HW decoders.

HW decoders are built to play video on your PC so you can watch a movie and usually don't supoprt all that many concurrent streams and aren't all that fast (they "just" need to be realtime, after all). That's amazing on playback devices (pretty much mandatory for H.265/AV1), but for "2U racks at Amazon" that's not very useful and large cored CPUs are still kings. Especially since software encdoders are still massively winning on visual quality per second per MB of video.

(Why am I talking about servers? Because this thread has started with AWS 2U video racks, not Apple TV boxes.)

Asahi linux is a good example. There is no hw acceleration support but video decoding is working pretty flawlessly. Of course at the expense of battery life

Sorry if I'm misunderstanding your comment's implication. But this is not hardware decoding. This is software decoding on the GPU.

NVIDIA for example has specialized (en/de)-coding hardware on their GPUs, which is different. And even that works together with their GPGPU Cuda cores.

No, the entire point of those vulkan extensions is to use the hardware-accelerated video processing units. You can use computer shaders for software decoding on the GPU, there is no need for special extensions to do that. AMD and Intel have specialized encoding/decoding hw too.

All three big vendors (Nvidia, AMD, Intel) and also bunch of ARM SoC vendors do have specialized blocks in their graphic solution whose only purpose is to decode and encode specific codec profiles. It is hardware decoding, not decoding on GPU.

Yes, real-time decoding on a desktop is just nice-to-have, with a couple of exceptions

(1) if your decoder software runs too slow for real-time decoding. I think it historically has happened every once in a while until CPUs and decoder optimizations catch up again. Eg 4k H.265 was too much for many desktops for a while.

(2) you do a lot of it, in which low power decoding->saving electricity is still good

eg Linux web browsers for ages had sw-only decoding at it worked mostly fine, just laptops having battery eaten in browser apps & electron based apps (videoconferencing).

As codecs are historically a top source of remote code execution vulnerabilities that get our device pwned, the poor sandboxability of hw decoders makes them a security problem as well. eg as a pathway to exploit the flaky and high-privileged GPU driver code paths.

Too bad there is no mentioning of MacOS support.

AFAIK Apple does not support Vulkan

Yeah it doesn't. But there is MoltenVK[0], which translates to Apple's Metal.

[0]: https://moltengl.com/moltenvk

AMD not even ready for the Linux support yet? that's disappointing.

Not sure why this release is so slanted, but av1 encode on AMD Linux has been available since October. https://www.phoronix.com/news/Mesa-23.3-HQ-AV1-Radeon

How Nvidia managed to emerge looking good on this PR is a colossal surprise & fucked up-ed-ly inappropriate.

I suppose the above comment was about amdvlk. radv is indeed often ahead of it these days.

If anything, I think AMD should drop amdvlk on Linux (to collaborate on radv there) and instead try to develop radv for Windows if that's doable to replace amdvlk altogether.

Yeah. What even is the of keeping things split?

RADV even has big commercial customers now (Valve).

Many of the people working on RADV are already employed by AMD (or paid by them indirectly), so arguably it's already collaborating.

I see the amdvlk/radv drivers a bit like the windows "Radeon Pro"/"Adrenalin" driver split, one is for people who stare at solidworks all day, the other for everyone else.

I think AMD wanted to phase out that distinction for OpenGL for example, so why wouldn't they want the same for Vulkan There is no need for them to be different really in that sense.

amdvlk and radv exist for historic reasons since they started independently and amdvlk was designed to support Windows too. But today I don't really see a big point in admvlk except where radv doesn't exist.

So AMD can do what Intel does. Support radv on Linux and amdvlk on Windows for example, or as above, try to make radv work on Windows too.

And meanwhile Chrome outright refuses to support vaapi outside of intel (and thats only because of chromebooks).

With the right Mesa patch (https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26...) and chromium flags to force enable vaapi, I've got working HW decode with amdgpu here.

Seems like with av1, Google had to stop self-sabotaging (turning on va-api has worked for most users for more than half a decade) & kind of had to move (because av1 is too beastly to do on cpu). Sounds like va-api has been turned on for almost a month now, but I haven't confirmed. https://www.phoronix.com/news/Google-Chrome-Wayland-VA-API

Maybe I'm missing something, but from what I could dig up it sounds like Firefox only enables va-api on Intel. https://www.phoronix.com/news/Mozilla-Firefox-115

AMD AV1 encoder is really bad, you arent missing much.

https://goughlui.com/2023/12/31/video-codec-round-up-2023-pa...

https://goughlui.com/2024/01/07/video-codec-round-up-2023-pa...

https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27...

Reply on news.ycombinator.com