Hacker News Reader

Comment by JoshTriplett | Hacker News Reader

Accelerated video decoding really wants to decode into a hardware texture and then render directly from that texture, and copy/move the data around as little as possible; ideally, any further processing happens entirely on the GPU.

Most video players have a defined pipeline from video decode to display; they may have some things they can do to the video before displaying it, but they can handle accelerated video.

Among other things, HTML5 video allows JavaScript/WASM code to get at individual frames and do further processing prior to display. It's not as simple as "decode video and render it".

View on HN

All modern browsers have done GPU compositing for a while now, so if anything GPU decode to a texture is the nicer path anyway.

As for HTML5 video allowing JS to get individual pixels: no one made any performance guarantees. The expectation is if you use these APIs (don't!) that you will incur at minimum a full framebuffer copy into a CPU shadow buffer, and likely hog your AVX-esque units that have to decode from some GPU-specific format into raster or do colorspace stuff.

Note that in this thread we equate hardware video decoding with GPU, but on mobile or mobile-derived hardware this is not at all the case. Your hardware video decoder is likely separate silicon that prefers to decode into its own YUV-happy format, and that format is in turn more or less only guaranteed to be understood by the hardware compositor - the GPU might not be able to share it. So if you start trying to do anything with video beyond displaying it in it's own plane, you can very easily force the hardware into a path that will be slow and energy consuming. This is well understood by the people that build Android and iOS, not so much web developers.

> Note that in this thread we equate hardware video decoding with GPU, but on mobile or mobile-derived hardware this is not at all the case.

It is the same for desktop hardware. The encode/decode block is extra silicon on the graphics card or extra block on the iGPU, not GPU itself. There have been some experiments with hybrid codecs - i.e. using shaders to for decoding, for example VP9 on pre-Raven Ridge AMD hardware (windows only) - but it was only stop gap solution and the next generation had proper decoder and encoder.

> So if you start trying to do anything with video beyond displaying it in it's own plane, you can very easily force the hardware into a path that will be slow and energy consuming

Mobile chips have their own bunch of problems; if you have too many planes (where too many is defined by the specific chip), you are getting into not enough memory bandwidth territory.

I think the expectation is that WebGL is used to manipulate video frames.

I see two parallel processes here: "decode video and render it" and interpret JavaScript/WASM at every frame.

The problem is that code runs between decode and render, and wherever possible the video data needs to stay on the GPU.

Reply on news.ycombinator.com