A Future Without MPEG
blog.chiariglione.org
blog.chiariglione.org
So we can finally have open standards for video that anyone can just use and implement. AV1 in particular seems to be on its way to replacing them: https://research.mozilla.org/av1-media-codecs/
As someone who mostly uses video in the context of online systems, MP4 (H.264) is the closest we have to a universally supported format on the user's end and it has been for years. However, it's complicated, patent encumbered and very inefficient by modern standards. And even MP4 isn't 100% universally supported.
Once encoding and decoding at reasonable speeds is possible -- we're not there yet, but there has been a lot of progress already -- I see no reason that AV1 shouldn't render H.264 and similar legacy formats obsolete quite quickly. The new format doesn't need to be perfect at first, just good enough to use with mid-range to high-end devices. Then the openness and improved compression/quality will make it an attractive alternative, with legacy formats reduced to being just a fallback for less capable devices. One generation of devices later, almost everyone can play it. Two generations later, you probably have hardware support, but at that point it's more about efficiency things like reducing battery drain on the viewer's side and hardware costs on the encoding side, because the format itself has already won.
I think H.264 hardware is pretty widespread. H.265 less-so. Anything older than H.264 has been dead in the water for years.
For next-generation video beyond H.264 (AVC) and H.265 (HEVC), that consensus has broken-down and his prediction seems to be that open-source standards like Alliance for Open Media behind the AV1 video codec will prevail. It’s worth noting that H.264 and H.265 are telecom videoconferencing standards defined by the ITU-T, not consumer electronics standards defined by ISO, so I’d say the loss of relevance of MPEG happened 20 years ago, even if MPEG (part of ISO) was involved in both standards.
As for the royalty-free nature of AV1, it will only be known once all the patent lawsuits are litigated. The AOM patent litigation fund in itself is no guarantee, given the capricious nature of common-law courts.
Mozilla could absolutely use and distribute x265 but that would also require them to have a license for the patents required for HVEC and would also require the users to have a license (which is of course unrealistic).
The only reason firefox can play h264 is because Cisco paid for an unlimited license with subdistribution and assigned it to mozilla.
So as h265/HVEC is concerned the problem for mozilla is that even if someone wanted to pay the bill like cisco did they would not be able to get a straight answer who and what they needed to pay for a license, right?
This is at least partly by design: On2 had an explicit mission to create competitive codecs that avoided conflict with the MPEG patents. I'm pretty sure this was a major factor in their appeal to Google.
(Disclaimer: I was CTO &/or CEO of On2 up until 2003)
Disclaimer: I was founder of on2 Technologies, acquired by Google and the basis for av1.
There really Is quite a bit to this story, starting in the early 90s and leading up to the situation today.
Our team in Clifton Park was the engine that built the VPx line of codecs. Many of the engineers who worked there are now at Google in Mountain View.
I think what might really be interesting was the decision process to open-source vp3, AKA Theora, the ancestor of vp8, vp9, and AV1. Even though it was only marginally competitive with mpeg-4, the existence of a free alternative, seemingly unencumbered by the sort of patent problems afflicting MPEG, put a check on MPEG becoming a virtual monopoly.
In 2010, Google took the baton with the acquisition of on2, and doubled down on the open-source approach.
The post here says "EVC is promising because it provides a quality that is comparable with or better than AV1, although less than VVC. EVC may have a chance if a licence will be published. However, this has not happened yet." I can only assume, from those numbers above, that "comparable with or better than AV1" applies only to the encumbered/enhanced variant.
It's hard to see why anyone would bother to implement the "base" standard vs. the already widely-deployed AVC/H.264, nor the "enhanced" one which seems to be roughly comparable to AV1 but with licensing costs attached (and as the post points out, no certainty at all about what that licensing would actually entail). Apart from those companies that hold the relevant patents, of course.
The beauty of EVC baseline is that it is quite power efficient. Considering AVC already require the least computation in modern codec, EVC baseline offer 30% reduction in bitrate while requiring NO increase in decoding complexity and is actually 40 to 50% less in encoding complexity.
Note: None of these has been tested outside of its members as it does not provide a reference encoder due to "new" patents arrangement with this codec. So we have no way to verify those claims.
Edit: Will add the link to document with those claimed figures later.
Anyway, if those claims bear out then it would seem like EVC has a possible future as an "AVC killer," though the computation complexity factors are complicated by the wide deployment of hardware acceleration for AVC encode/decode. Of course, hardware support could materialize for EVC as well, but it does rather complicate possible adoption for what we'll call a "sub-generational" improvement, in a world coming around right about now to AV1 hardware support.
Perhaps reduced computational demands could lead to a niche of usage in unaccelerated environments. I don't know if there's an example of significant uptake for an "in-between" step like this that one could point to as a possible model for success, though.
Oh yes. I just double checked. I stand corrected. I was under the impression they only use high profile with MutiView Videos.
https://goughlui.com/2020/05/17/c-band-adventures-looking-fo...
https://goughlui.com/2020/05/21/satellite-more-avs-avs2-head...
[1] Seemingly no source code available for AVS2 decoders, dubious claims of beating HEVC without independent tests, and China history of "independent" development by copy&paste notwithstanding
If a physicist can't patent superconductivity (you can't patent discoveries and laws of nature), should it be possible to patent compression algorithms? The case for patents in software, as additional protection on top on copyright, is not clear to me.
Frankly neither copyright nor the patent law is actually suitable for software. Software deserves a tailor-made IP law that actually recognises the difference between compiled code and source code, the rights of the consumer to modify his device and keep it updated, etc. There should be an escrow-type system, where protection is granted if code is hosted in a public repository, and is made public after 20 years or so.
All executable should be treated in a standard way, and there should be no EULA outside of enterprise systems. A consumer person does not need to read EULA when they buy a book or a car, why should they do so when they buy a PC?
Until this happens, I feel developing nations shouldn't even recognise software patents - copyright is protection enough.
I would say yes, at least for lossy compression, where the challenge is not about shuffling bits but finding semi-artistic graphical approximations of common video patterns.
I also approve of an industry-wide effort to patent-unencumbered solutions.
EVC Baseline is expected to be 20% better than AVC High Profile, and EVC Main Profile is expected to be 30% better than HEVC. The Codec is backed by Huawei, Samsung and Qualcomm. If you are unfamiliar with the Smartphone market, The three would represent roughly 80% of marketshare excluding Apple.
Now on to VVC. I am surprised by the sentence
MC-IF has 31 members, 7 of which are licensing entities (i.e. a little less than ¼ of all members). The “industry” members account for just ½ of the HEVC patent holders. It is hardly believable that VVC will fare better than HEVC. It could very well fare worse because VVC adoption in broadcasting will take years.
Well Yes. That is because 7 of those licensing entities already covered most if not all of the industry. ( Excluding a few Open Media Alliance Member of course ) The MC-IF actually includes ALL of the current HEVC patents holders. That is HEVC Advance and Velos Media, which is basically Samsung and Qualcomm, along with many others that were not in any HEVC patent pool due to disagreement in the first place.
( And If you notice the removal of the infamous Technicolor, they sold their patents to another entity that is inside MC-IF, but I cant remember which one on top of my head )
VVC is expected to be 50% more efficient than HEVC. And judging from its reference encoder, this is the first time since AVC / H.264 era a video codec that might actually live up to its claims. ( Normally marketing likes to use unrealistic claims ) It really is the state of the art Video Codec, at a decoding complexity that is quite manageable. ( Lower than AV1 )
So what does all that means? Samsung, Huawei and Qualcomm are also inside MC-IF ( represented by different groups ). My guess is that EVC is basically a backup plan or a gesture to MC-IF, if the licensing deal can be agreed upon, they will go with EVC.
As a video codec enthusiast, I am extremely excited with both VVC and EVC.
As to MPEG ( Not to be confused with MPEG-LA ), I am not quite sure why he said it is dead. I reread the article a few times and still dont quite understand it. May be I am missing some context?
I am afraid this has not been substantiated by any of the public decoder demonstrations I've seen. Please see the most recent VVC technical update presented at the MC-IF meeting on March 2nd of this year:
https://a7dce6fd-e8f0-45f7-b0b0-255c5c9a28e1.filesusr.com/ug...
On slide 10 is a graph of VVC performance showing the VTM (VVC) decoder at 2.0x the complexity of HM (HEVC). On slide 12, the Ittiam production decoder boasts 1920x1080 @ 24fps on a 4-core Cortex A75 @ 2.5GHz.
Compare that with this recent study of dav1d (AV1) decoder complexity on a broad set of mobile SOCs, where 1920x1080 @ 24fps was easily reached by a Google Pixel 1 from 2016! Using just the two LITTLE cores! Even higher frame rates were achieved with more modern devices:
https://www.reddit.com/r/AV1/comments/gncplq/av1_multithread...
Full disclosure, I contribute to the dav1d project and performed this study.
But dav1d is a truly amazing pieces of work!
Sure, but my comparison was to the Ittiam production decoder.
I think that is a very partial reading of the forces at play.
Apple may be 20% of unit sales, but an overwhelming majority of industry profits and has a lock on the best demographics (the rich and the young). AFAIK they haven’t decided, even if historically they tended to prefer ITU/MPEG standards (but boycotted Blu-Ray).
AV1 has Netflix and Google/YouTube behind it. Not sure about Amazon, but they own Twitch which is on the AV1 side. In an era where the content providers also control the app through which it is delivered, they have the control over which codec gets chosen, not the hardware manufacturer. They may tactically choose to use hardware-accelerated standards like HEVC when trading off the royalties they have to pay vs. using less battery life on their customers’ devices, but in the long term it seems to me AV1 will prevail
They want VVC. After all MC-IF was co-founded by Apple. But that doesn't mean MC-IF will succeed. Having everyone on the table does not necessarily mean positive results will come.
VVC will finalise in July. So we will know soon.
Those three have one funny thing in common, they dont produce any content.
The next device to get converted to the web is the smart tv, there is no reason it literally has be anything different then a webpage, same goes for Roku.
True enough. But at least in the USA all the satellite companies, and their set top boxes, are being forced to switch to HEVC because their half of their physical frequency spectrum was stolen by telcos. So there'll be substantial amounts of people making hardware for quite a while. The web isn't everything yet.
Having the "standard" means having the lion's share of the market, which means more money.
So the idea of patents is, if Alice invents something really fucking amazing she explains the invention, pays the government some money and they publicise her explanation but she is entitled to control who uses the invention for a period of time.
This is a trade because now Alice's invention is available to everybody, no chance Alice loses interest and it's lost for a thousand years before somebody else re-discovers it, but on the other hand Alice doesn't need to secure investors and risk the invention being a failure, she can demand terms like 10% of the profits from anyone taking those risks.
But, it soon turned out you don't need a complete working invention. Alice describes half an invention, and then nobody else can use it without the other half, but even if they figure out the other half on their own they owe Alice! Alice doesn't even need to invent the whole thing, she can just describe the easy half, the government signs off and then when somebody smarter finishes it Alice gets rich!
Today in most of the world people have found ways to argue that computer software, which is a work of literature and thus protected by Copyright (which has different problems we won't discuss here) can also count as an invention and be patented.
So the best outcome for an outfit like MPEG is that Bob invents a really clever technique that turns moving images into much less data than before, totally just as a single flash of inspiration somehow, and rather than keeping it secret, or using just in one MSDOS program in 1989 and then never again he publishes it as a patentable invention, then MPEG incorporates it in a video standard called, say, MPEG Bob. And maybe everybody in the world cheerfully pays Bob $100 each for this amazing invention. Hooray.
But very quickly the problem with MPEG is that Charlie gets a government patent for the invention of, say, dividing five into three equal integers and then hires a very good lawyer. Charlie's lawyer says it doesn't matter that this is nonsense and can't work, because $10M worth of lawyers say you owe Charlie for using this invention in MPEG Bob.
In between these extremes there are lots of problems less intrusive than Charlie. The Springfield Higher Institute of Technology gets a patent on an idea you, an expert in the field, have been telling people about for years, but you never wrote it down so you can't prove it. Did they really invent it, or just hear it second hand? Either way, they demand $1 each, but they are a university so maybe that's good? Although it is billions of dollars, and it was really your idea if it was anybody's...
Leonardo, the author of this piece and MPEG leader, is quite sure that patents are necessary, mostly because of the Bobs in this world though I suspect he'd be sympathetic to the Springfield Institute too - to him the existence of Charlie is an annoyance that we should all try to find some way past rather than a fatal flaw in the entire endeavour.
Hey, you just described S3TC patent! Funnily enough S3 ripped off Apple, copying verbatim Hoffert work at Apple Advanced Technology Group and consequent patents from 1990 (US5046119A) for Apple Video 'road pizza' codec, except their patent added "for texture compression" at the end. The patent is about dividing colorspace between two points by dividing colors by 3 :-) and is directly coped from Apple. Gotta love how wikipedia calls direct copy a "This mode operates similarly to mode 0xC0 of the original Apple Video codec".
Amazingly S3 even had the balls to try and sue Apple for S3TC royalties in 2010. Sadly Apple loves patents and didnt bother trying to invalidate, they were content winning on technicality. S3TC patents are expired now.
Which is why it's so infuriating to read Leonardo's grandstanding "Those who have created a new intellectual object have to right to exploit it". What exactly did this "creation" build on? It was not produced in isolation.
A predatory business model based primarily on large companies exploiting legal technicalities to seek rent from innovators has been rendered obsolete because the large companies couldn't get their act together and the innovators did a better job anyway.
Efforts to co-ordinate those large companies in those exploits are probably now doomed, as are the organisations behind them.
The future almost certainly belongs to open standards and the community, though someone who has built their career around the old business model isn't happy about it.
The billion dollar plus a year residual stream can pay for a lot of efforts to keep trying to refresh the stream.
The author of the post seems to hold the position that mpeg falling apart is because many companies have been too aggressive. But the lesson that some are going to walk away with it that they haven't been aggressive enough.
I think it's clear which way the water is flowing, though, and I don't think any amount of distress on the part of the fading powers of yesteryear is going to change that. There is too much incentive and too much technical expertise on the other side, and that side also has enough big players with their own legal teams to block attempts at obstruction by questionable legal challenge.
But don't underestimate what a firehose of money can do, especially one in the hands of which is effectively a dying industry. (A small one, for sure, but one that was formerly extremely profitable)
The big players are can bring significant resources to bare to be sure, but they're also the least committed fundamentally. The right extremely beneficial licensing terms for say, HEVC, in exchange for dropping out of AOM would probably be extremely uhh "business smart" for more than a few of those big names.
One of the challenges in this space has always been that patent licensing for existing codec has zero marginal cost. Vendors can respond to the threat of open alternatives with licensing concessions. I think part of why we see the system collapsing a bit now for video is because they've managed to thicket themselves to the point where its become extremely politically difficult to negotiate in that manner.