A JavaScript H.264 decoder
github.com
github.com
Still not sure how this is meant to actually be useful though. The problem with H.264 isn't availability of implementations, it's being non-free and heavily patented.
I can kind of see a use for this if you are a big content provider with a bunch of cash and want to distribute video to Firefox users without transcoding but it still seems pretty derpy.
The patches linked from the github README aren't necessary to run the transpiled version that was in the demo -- it's a memory optimization that's being used in the tuned-for-JS version.
As regards derpiness: it lets content distributors decide to pay for H.264, if they want, exactly, and paves the way for other codecs to be deployed by such distributors as well. It also runs the codec in a managed environment -- format decoders are often very fertile territory for exploitable bugs, since they are pretty much by definition all about pointer math and byte-poking. But the initial intent, when they decided a week ago to try it, was to push the envelope of JS performance such that we find new ways to extend said envelope.
That's a 2.4GHz C2D. Need to try it on my i52400.
Is that intentional, or a problem with Nightly?
We are using standard Firefox, no special patches. However, we used the Firefox nightly, not current stable. The decoder runs much faster in nightly, due to JS engine improvements that landed over the last few months, and are not yet in stable.
> Still not sure how this is meant to actually be useful though. The problem with H.264 isn't availability of implementations, it's being non-free and heavily patented.
First thing, this at least gives you another option. That is, if we get this codec to run as fast as a native one, then we now have the choice of either the browser or the video website providing the decoder (and properly licensing the decoder, if they are in a country that has pure software patents). More options are never a bad thing.
But I think the real potential in this approach is something entirely different. The opportunity is that you can download arbitrary decoders. So instead of the current world we live in, where you have a few decoders installed, you can have custom ones for different websites. Imagine a website that has cartoon videos or anime etc. - in principle, they could use a custom codec that is heavily optimized for that kind of content, as opposed to being forced to use stock decoders.
Also, it prevents being frozen in time: If you can download decoders from the web, you can improve them constantly while making sure your users have the proper decoder (since you ship it to them yourself), which you can't do if you rely on stock preinstalled decoders.
Native ones have chunks written in hand-tuned assembly language, offload parts to specialized hardware, and other such tricks not available to ECMAScript. I'm not even sure why "as fast" is being considered a possibility.
1. JS can also use hardware acceleration through WebGL. We have not done this yet, but will. 2. JS has some proposed extensions, WebCL and Intel's RiverTrail, which let it utilize SIMD and other computing hardware. We will investigate using those too.
With those two things, we believe JS performance will be very good. How close it will be to native code, though, is hard to say at this point in time.
However, there is one big advantage a JS decoder would have, that native code does not: A JS decoder can be downloaded and run securely. As a consequence, you can continually improve your decoder in JS and your users will run the latest, most optimized version, while standard native decoders are typically upgraded much, much less frequently. Also worth noting is the potential to ship specialized decoders, as I mentioned in another comment, imagine an anime website that ships a video decoder heavily optimized for that specific type of content. That could be much more efficient than a stock native decoder.
Finally, it's worth noting that the decoder we compiled from C, Android's H.264 decoder, does not have any substantial amount of handwritten assembly. I had assumed like you said, that real-world decoders would have such things, and am curious why it doesn't. If anyone reading knows the answer I'd be very interested in that.
Perhaps by then it'll be completely routine for video to be encoded and decoded using Javascript on a mobile device.
note: I realize it's not production code. I'm asking more about the concept.
If x264 does, so does VLC, mplayer, ffmpeg, gstreamer, and dozens of other applications that use video and audio decoders in Linux. Fortunately quite a lot of the world is not the United States, and today VLC is the world's second most popular media player and has never paid one cent in patent licensing fees.
But of course, yes, being open source does not magically exempt you from patent laws in countries with insane, broken patent laws. In practice, if you want to make a large-scale commercial application that will be distributed in the US that uses x264, you will probably need to pay for an MPEG-LA license. They're quite cheap, though: 0 cents per unit up to 100k units, 20 cents per unit after that until 5 million, and 10 cents per unit after that.
Largely, the question of whether a distro contains any particular piece of software is whether the people who run the repositories are willing to host it. This applies both to possibly-patented software, but also to libraries like DeCSS (necessary to play DVDs) that violate the laws in some countries, but not others.
I apologize.
"Sale (Sell) (Sold) (Seller) – shall mean any sale, rental, lease, license, copying, transfer, reproduction, Transmission, or other form of distribution of an AVC Product or the Transmission by any means of AVC Video either directly or through a chain of distribution."
This is consistent with the discussions I've been party to with the MPEG-LA directly.
Edited to add: I would very much appreciate a reference to MPEG-LA saying the things you indicate in your comment. It would be very interesting to be able to point to those statements, in a number of ways.
1.7 fps
It's cool, but I don't see any real world applications. Anyone got any ideas?
What browser, OS and what Pentium 4 processor? And for how long did you run the video?
Mine drops to 1.7 after a few seconds, the first few frames are dark so in the beginning I get 6 fps.
Says a lot about the importance of software over hardware :)
As far as applications, it will let content providers who want to ship H.264 and pay the license fees do so without requiring all browser developers to pay such fees. (For Firefox it would be a meaningful portion of Mozilla's engineering budget.)
And it's a pretty compelling demo, IMO, of the fact that we're not done with JS performance yet, and that people don't need to be running to Dart or NaCl or other rip-and-replace technologies in order to get great perf. There's a lot of opportunity for even better performance on the engine side as well as the library side: they started a week ago with emscripten, Closure and the libstagefright C sources...
2. This does not use any hardware acceleration yet. This is simply compiling the Android C decoder into JS, nothing else - just a few days of work. We will now start to look at actually optimizing the code specifically for JS, and also to use GPU shaders for the relevant part of the code. Both of those can potentially make this even faster.
Regarding real-world applications, the interesting possibility is for websites to ship their own codecs. As I mentioned in a comment above, imagine an anime video website that ships a video decoder optimized for that kind of content - it could be much more efficient than stock H.264. Also, if websites can ship their own decoders, they can continually improve them (unlike now where websites rely on client decoders which are not constantly being improved).