Firefox 80 and my confusion over its hardware accelerated video on Linux
utcc.utoronto.ca
utcc.utoronto.ca
And I know technical users would appreciate it. I still don't know what is and isn't getting hardware video decode, or whether I actually need a particular about:config flag or system package, and I have tried hard to figure out.
Even Crostini relies on containers and virtualization to expose a second kernel, WSL2 style.
In the meantime, to save you rummaging around bugzilla, follow https://wiki.archlinux.org/index.php/Firefox#Hardware_video_... , it's the most constantly {up-to-date, concise, reliable} source you'll find. You don't have to be using Arch, the information contained in this section applies to any Linux/Firefox distro.
Note that Firefox isn't the only browser to struggle with it; see https://wiki.archlinux.org/index.php/Chromium#Hardware_video... for the similarly non-obvious instructions in Chromium (and by the way, currently it's plain impossible in Chrome).
Props to Martin Stransky from RedHat who implemented this for both Wayland and Xorg during the last months.
See my before-last paragraph above: no, regarding hardware video acceleration, even Chrome doesn't do what you describe. What you describe stands for non-video content crashes, which isn't what's being discussed here. Could the technique you mention be applied to video gpu crashes? I personally don't know and I'd lean towards "nope" (else Mozilla & Google engineers wouldn't be so exceedingly cautious about it). If anyone is knowledgeable enough to precise / contradict this, please chime in.
At any rate, even Chrome is built with zero hardware video acceleration support on Linux (hardware acceleration yes, hardware video acceleration no). Chromium does offer the option, just as hidden behind flags as Firefox, and until last month you had to build it yourself with a VAAPI patch left unmerged for years by Chromium devs.
ffmpeg has supported this for forever and yet firefox doesn't. It's bizarre.
Under Windows and MacOS, it has been implemented years ago. Under Linux, it wasn't. Linux has API for video acceleration and players like VLC, mpv or Kodi have been supporting it for years, as well as ffmpeg, just browsers ignored it.
I'm not going to belittle the effort, quite opposite, but it is telling when this feature has been implemented by Redhat employee and not Mozilla.
Since Flash plugins were killed, Linux has been a mess for hardware accelerated video on the Web.
Many apps ranging from vlc to Totem seem to hardware accelerate just fine, all without seeming to know or care which specific hardware I'm using.
What makes it so much more difficult for FF?
Hardware accelerated decoding of DRM'ed video is a completely different story, and the only browser I have seen do that on my machine even on Windows 10 is MS Edge. But there's a lot of non-DRM'ed video on the web, and getting it decoded in hardware on Linux isn't a given either.
Most video players have a defined pipeline from video decode to display; they may have some things they can do to the video before displaying it, but they can handle accelerated video.
Among other things, HTML5 video allows JavaScript/WASM code to get at individual frames and do further processing prior to display. It's not as simple as "decode video and render it".
As for HTML5 video allowing JS to get individual pixels: no one made any performance guarantees. The expectation is if you use these APIs (don't!) that you will incur at minimum a full framebuffer copy into a CPU shadow buffer, and likely hog your AVX-esque units that have to decode from some GPU-specific format into raster or do colorspace stuff.
Note that in this thread we equate hardware video decoding with GPU, but on mobile or mobile-derived hardware this is not at all the case. Your hardware video decoder is likely separate silicon that prefers to decode into its own YUV-happy format, and that format is in turn more or less only guaranteed to be understood by the hardware compositor - the GPU might not be able to share it. So if you start trying to do anything with video beyond displaying it in it's own plane, you can very easily force the hardware into a path that will be slow and energy consuming. This is well understood by the people that build Android and iOS, not so much web developers.
It is the same for desktop hardware. The encode/decode block is extra silicon on the graphics card or extra block on the iGPU, not GPU itself. There have been some experiments with hybrid codecs - i.e. using shaders to for decoding, for example VP9 on pre-Raven Ridge AMD hardware (windows only) - but it was only stop gap solution and the next generation had proper decoder and encoder.
> So if you start trying to do anything with video beyond displaying it in it's own plane, you can very easily force the hardware into a path that will be slow and energy consuming
Mobile chips have their own bunch of problems; if you have too many planes (where too many is defined by the specific chip), you are getting into not enough memory bandwidth territory.
Anyways, I think it is a case where they were burned before trying to compensate for different levels of driver quality for hardware acceleration and so they are more skittish now and want to proceed slowly.
Of course, I have no sources to cite and this is based on half-remembered blogposts from a decade ago, so take this with a grain of salt.
I think a broader answer to your question is that most apps accept that if a driver causes a crash or security problem, it's not their problem and it's up to the user (and driver developer) to remediate it. Firefox has chosen to attempt software fallback to avoid the issues. There are obviously many trade-offs for good and ill with this approach.
As does gstreamer (used by Totem): https://blogs.igalia.com/vjaquez/2018/03/28/gstreamer-va-api...
VLC on Linux systematically hardware decodes H.264 on my aging integrated GPU, but I haven't got it to work on Firefox ever.
(Most web video nowadays is probably VP9 or something, which my GPU doesn't decode, unless you coerce YouTube et al. to serve H.264 using a browser extension, but that's another story; even if I do that, it still doesn't get decoded in hardware.)
I'm sure the browser developers have reasons for being conservative about enabling VA-API acceleration, ranging from possible issue with their codebase to buggy drivers, but it does seem like there are way more problems than e.g. in VLC.
On modern Linux, it’s possible, and not even terribly hard, to implement hardware-accelerated video playback without any libraries or frameworks, by directly consuming V4L2 API exposed by the kernel. Here’s a proof, I did it alone spending a month or so part time, and resource use is lower than VLC’s despite my code is in .NET: https://github.com/Const-me/Vrmac/tree/master/VrmacVideo
However, that wasn’t always the case. Until recently software had to fallback to software decoders, and implement other workarounds. Software developers are reluctant to throw away old code and do something completely different.
1. They spent much time on that old code (sunk cost fallacy).
2. Old code usually works more often than it doesn’t (don’t fix things which aren’t broken).
3. Massive refactor of large software is very expensive in general. Moving stuff from CPU to GPU is going to cause large changes in many parts. You can’t usually afford to copy uncompressed video frames from VRAM back to system RAM and call it a day, that would be too much bandwidth. Instead, you have to change a lot of other code to handle GPU textures instead of system RAM images.
Do you mean V4L2 M2M decoding? That's only supported on a very small subset of devices, mostly ARM devices. Other devices might use VA-API (Intel), VDPAU (NVIDIA proprietary), or something from the OpenMAX standard (some AMD drivers).
M2M is far from easy to handle in a lot of circumstances, though it depends on the kinds of devices you use. I don't really know why you're going into a screed about "sunk cost fallacy".
Nothing prevents Firefox from consuming these things. The majority of time I’ve spent on that library was not on V4L2, it was containers parsing (they are incredibly complicated these days, as I found out in the process), bitstream and parameter sets parsing, also audio codecs shenanigans: https://github.com/Const-me/DtsDecoder
> M2M is far from easy to handle in a lot of circumstances
I know, and I never said it was trivially simple. But it’s not terribly complicated either.
I have implemented that alone, for fun, spending a month part time. My implementation plays all h264 video files on my hardrives (more than 1TB of them). Because of that, I would expect better from a company with 9-digit annual revenue. BTW I had no prior experience with Linux multimedia, albeit I did a lot of related things on Windows, mostly with Media Foundation.
> why you're going into a screed about "sunk cost fallacy"
I have no idea how Firefox is developed. However, I’ve been developing other software for living for couple decades now, worked with many codebases including very old ones, and I think I know a few things about software development in general. That was just my educated guess.
There were other considerations as well. For example, if I would have forked or re-built ffmpeg, I would then have to support my version. That’s something I try to avoid when I can. I think OS maintainers are doing much better job updating these third-party packages than I can possibly do myself, at least for an unpaid hobby project.
Very cool to see a combination of low-level Linux kernel APIs and .NET.
True hacker spirit right there!
ALSA from libasound.so.2, there: https://github.com/Const-me/Vrmac/blob/master/VrmacVideo/IO/... and the rest of libasound.*.cs files in that folder (parts of the same static class). Audio output is harder than necessary on all OSes I developer for, including Windows. From the internets I got an impression that at least on RPi, ALSA PCM is more reliable than the rest of them. Also doesn’t depend on any desktop components, important because the library can also be used without any desktop environment, rendering on a physical display.
Message queues from librt.so https://github.com/Const-me/Vrmac/blob/master/VrmacVideo/IO/... .NET has equivalent stuff already. I needed poll() to get notified about multiple things not just queued messages, as both V4L2 and ALSA expose poll()-compatible file handles in their respective APIs.
Martin Stransky (the implementer of the feature) explains this in one of the related bugzilla bugs, if you want details.
Maybe a controversial opinion, but I think a browser should not be in the business of delivering video codecs. The operating system (distribution) should have one set of codecs that is tested, optimized, and accelerated, and all apps should be able to use them.
Naturally Linux desktop being as it is, this solution will never work.
I can understand the desire to avoid using drivers which cause a poor user experience. I don't envy the task of trying to implement and maintain a system to achieve that. I do wish it wasn't so painful and confusing for users when the behavior doesn't match our expectations.
It's not possible to get VA-API to work in FF 80 on X11 or Wayland because VA-API has been broken since FF 79. The fix will be in FF 81.
Just guessing, I have not looked into the topic except in this thread. What would be the bugzilla id for that?
VA-API hardware video acceleration was broken in Firefox 79 and won't be fixed until Firefox 81. Therefore, it's not possible to make VA-API work, under either Wayland or X11, in Firefox 80.
In FF 81 it will be confusing to the non-experienced again: You need to fiddle with config and environment variables. Well, unless a new bug comes up...
This is complex stuff. I spent a day last week trying to figure out why hevc vaapi patches for OBS were failing with mkv containers but not mp4 ones. Its just really ugly C apis all the way down.
It does not support MPEG-4, or it's disabled by default. I don't know how widespread that format is, but I believe it was developed with streaming in mind.
(My pet theory: I think often it's just the attitude that is missing. So much can be accompished with the right attitude.)