Video Decode Acceleration framework available on Mac OS X 10.6.3 | Hacker News Reader

Video Decode Acceleration framework available on Mac OS X 10.6.3

developer.apple.com

22 pointsblasdel15 comments

Is this what Adobe was complaining wasn't available & their excuse for why flash video on the mac sucks?

According to http://en.wikipedia.org/wiki/DirectX_Video_Acceleration, Flash did not use DXVA (hardware-accelerated video playback) until 10.1. Video being the complicated beast that it is, there are probably many reasons why Mac flash performance was inferior to Windows flash.

And Flash 10.1 won't be shipped for at least another six months. The beta release with DXVA was only released in November.

Yep, this is it. I can't help but feel like this is a direct response to Adobe's criticisms.

The question now is how long it will take Adobe to get a Flash 10.2 beta out that will take advantage of this. As far as we know, this API is exactly what they've been asking for, so presumably they won't have to re-architect anything to be able to start using this API. I can't wait to see benchmarks of how much of a difference this makes.

On the other hand, if Adobe spends all summer making more excuses and not delivering acceptable performance on high-end hardware, we'll know that they have screwed themselves so thoroughly that Flash deserves to die.

> I can't wait to see benchmarks of how much of a difference this makes.

Some difference, but it's not the best thing ever, and not even the most important problem Flash has.

For one thing, the H.264 decoder in Flash is already faster than QuickTime's software decoder and competitive with FFmpeg, so video decoding is absolutely not the reason people think it takes too much CPU. For the size of video you usually watch in a browser, drawing other elements on top of the video (and the necessary conversion to RGB) takes more CPU than actually decoding it (see http://blogs.adobe.com/penguin.swf/2010/01/solving_different...). I don't know if they've improved that, maybe not - I guess the best solution would involve doing the whole thing in OpenGL which is tricky.

The other problem is that the browser plugin API is really old and doesn't match up with the double-buffered window system that OS X always used. This is supposedly better in Flash 10.1 and would improve it more than anything done here if so. Of course 10.1 isn't out yet, so who knows.

I've measured the CPU usage of playing an H.264 clip at 18% for QuickTime Player, and 90% for Flash 10.1 beta. This was on a machine that does not have any hardware to accelerate H.264, and the number for Flash was measured while there was no interaction with the Flash object, and nothing was being overlaid above the video itself. If we assume that Flash's software decoder actually is at least as efficient as the one built in to the OS, then that means that the non-decoding work (color space conversions, compositing, scaling, and any overhead due to the plugin API) adds up to at least four times more work than the decoding itself. This is while the compositing and scaling are trivial! (The video was being played at native size, not fullscreen, and with nothing overlaid.)

If the plugin system really causes that much trouble, then why was Adobe complaining about the lack of a low-level API for hardware-accelerated H.264 decoding? It would seem that they are in more dire need of a much-improved plugin architecture, so that they can outsource more of the compositing and color space conversion.

Since Adobe has tried to excuse a lot of the performance problems as due to the lack of an appropriate H.264 API, it's time for them to put their money where their mouth is. If they release a new plugin that uses this new API, and the performance is hardly improved, then it means the complaints about the lack of this API were merely a red herring to shift as much blame as possible to Apple and to distract from the real causes. If performance does increase significantly, it will mean that Flash's software decoder was shit all along, and we can move on to bashing Apple for only supporting 3 of the many hardware H.264 decoders that they ship.

The blog you link to makes it clear that there need to be major changes to how in-browser video is handled. It also makes it clear that Adobe's stance is that it is the duty of the operating system to provide APIs that fit Flash's rendering pipeline, rather than have Adobe redesign the Flash player so that it can take advantage of existing acceleration APIs. In the case of the Linux plugin, there are several different APIs that are potentially useful, but Adobe is waiting for the dust to settle and for a clear winner to emerge. Given what we've seen on OS X about how the exact nature of the APIs matters so much to Adobe, it is odd that they are avoiding participating in the open-source process by supporting a particular API and helping ensure that it will meet their needs.

If the plugin system really causes that much trouble, then why was Adobe complaining about the lack of a low-level API for hardware-accelerated H.264 decoding?

As you pointed out in your next paragraph, it was because that way Adobe can point a finger at Apple and say "it's them to blame". This blog post [ http://www.kaourantin.net/2010/02/core-animation.html ], quite similarly informative as the one astrange linked to, describes very well part of the reason why the Flash plugin is a dog on Mac OS X. I can only add to it a detail that is not explicitly mentioned in there - the drawing contexts mess and lack of support (e.g. OpenGL) is only applicable to NPAPI plugins. WebKit has an alternative, modern, clean, and fairly flexible plugin API. So if Adobe really wanted to make Flash highly performant on Mac OS X, they could have done what Apple do with their Quicktime plugin - have an NPAPI one for compatibility and have a WebKit one for performance for most users.

The Quicktime Player Plug-in actually has the same limitations as any other Safari plug-in/Flash Player itself. The hardware accelerated Quicktime in Safari uses Core Animation for rendering into the browser. There's only on the other hand only DirectDraw in Windows XP / Firefox and they have made video acceleration happen there. Before 10.6 QT Player plugin used QuickDraw :) I don't think Adobe wants to use Core Video any how. Though.

I find it interesting that they don't support any of the video cards in the Mac Pro (GT 120) or high end iMacs (ATI Radeon). I guess it will make the biggest difference with battery life in the laptop line so they're not concerned with the desktops. Still somewhat disappointing.

Some honest questions:

Does the caller of this API need a patent license to do so? Does the patent license of H.264 cover decoding of the format through OS APIs, or does each caller need its own license?

Of course you don't. That would be like having to get your own patent license to use iTunes.

Although you do actually need a license to use the QuickTime AAC encoder in your own program, but only on Windows. Or so I remember from mailing list traffic, haven't tried it myself.

Yes you will need it, not for using quicktime, but this is a low level API where you will implement a lot of patent stuff yourself.

Oh, I forgot this reply existed.

It isn't. An example of an API that does would be XvMC, which requires parsing and doing large parts of the video decode yourself.

The most you'd have to to use this framework is parse a .mp4 file yourself. Such things are not patented.

I believe you pay per "branded decoder" in software, and you pay more for a "branded decoder" intended to be included in an operating system.

You would hope that the extra fees would cover the downstream, but the "branded" language makes it sound like you'd have to be explicit that you're calling out to Quicktime or whatever.

Obviously with hardware, as with OS decoders, you're expecting it to be called by others but again the "branded" language may confuse matters.

Plus, you're almost certainly only covered for "personal and non-commercial" decoding in any of these cases.

Reply on news.ycombinator.com