The Last MPEG-4 Visual Patent Has Expired
phoronix.com
phoronix.com
https://meta.wikimedia.org/wiki/Have_the_patents_for_H.264_M...
At which point, the continued march towards higher resolution and bandwidth video will probably limit their utility
Just as the patent system intended: maximum extraction was enabled while the product was useful.
Why are you entitled to it for gratis? It’s a taking for public benefit.
Ancient writings like Plato or Marcus Aurelius somehow feel different. But high-fidelity live recording feels like it shouldn’t go into the public domain. It’s someone’s commercial work product
https://en.wikipedia.org/wiki/Copyright_Clause
The purpose of the system as it was originally conceived was not to provide maximum economic extraction -- the purpose was to promote the progress of science and useful arts.
It is not necessarily the case that the maximization of economic extraction coincides with the progress of science and useful arts.
When I was in law school we were taught what seemed obvious then and is obvious now: that Congress has the power to promote the progress of science and useful arts by giving authors and inventors exclusive right to the economic fruits of their works. You can argue about flaws in the system but it’s pretty laughable to suggest that copyrights and patents don’t exist for the purposes of maximum economic value extraction for inventors and authors.
There is quite literally no other reason for granting a copyright or patent. It’s also why if a plaintiff is successful in a claim for an infringement of either, he’ll get awarded money in the United States that at a bare minimum is designed to compensate him for the money he would have made if the patent or copyright had been complied with. Europe has some different ideas about moral rights in this space, but since you’re quoting US material I assume you aren’t referring to our funny brothers across the pond.
It’s obvious to most people that humans act in certain ways due to incentives, and we like to incentivise invention and creativity in the West.
They exist to maximize the production of scientific and artistic works and do so by incentivizing creators with a limited exclusionary property right over the created work.
If the goal of this particular section were to maximize what authors and inventors can extract from their creations then explicitly assigning to them the property rights in perpetuity of said creations would have been the way to ensure that outcome.
Given that they specifically wrote "by securing for limited Times" it is entirely reasonable to make this interpretation and I'm not sure why you weren't taught this interpretation in law school -- it's a pretty settled thing and there have been many times that this subject has come up in US courts over the years.
And this isn't so whack if you stop and think about it. There are all kinds of limitations on intellectual property rights.
Like take trade secret protection for exampple -- it has no term limit and no disclosure requirement. If maximum extraction were the goal then patents would look like trade secrets. Instead the bargain that the patent system forces on creators requires them to publish their invention and then hands the rights to that invention to the public after a finite period of time. The system is explicitly designed to be less extractive compared to others on purpose.
And if copyright is built for maximum extraction why does it carve out an affirmative defense in the form of fair use for others to use copyrighted works without paying? Same with first sale, same with the idea/expression dichotomy -- so much of the antisocial contemporary intellectual property regime is configured around providing people free access to the use of intellectual property.
> There is quite literally no other reason for granting a copyright or patent.
"To promote the Progress of Science and useful Arts, by securing for limited Times to Authors and Inventors the exclusive Right to their respective Writings and Discoveries."
It seems pretty clear to me from that sentence that the stated purpose of these laws was to drive the creation of science and art and that a limited time profit incentive being enshrined in law was merely a carrot to create an incentive for that.
The framers understood the complex interplay between property rights, freedoms, and incentives in driving innovation and creation.
Knowing that granting someone an exclusive right to something that isn't subject to finite physical limitations effectively denies everyone else in society that same right the framers still understood that it was necessary to abridge the freedoms of some to enrich society as a whole through the development of science and art.
In choosing to use the words "for limited times" the framers encoded the economic priority of the majority of society over the economic priority of the minority of creators into this section but they also did not give a specific time which to me speaks to their understanding of the complexity the interplay between all the factors and allowed for the possibility that they could change over time.
On damages: that's a non-sequitur. How a system enforces an entitlement once created tells you nothing about why it was created. Tort damages compensate plaintiffs but nobody woudl say that tort law exists to maximize plaintiff recovery.
It isn't in dispute that people respond to incentives. That's precisely why the size of the incentives and the configuration of the incentive structure matters and why we have things like term limits and fair use.
Actually it’s the opposite - both patents and copyrights expire after a period of time, which is the compromise we’ve reached between incentivising people to make money for themselves personally (giving them a legally protected right over their creations), and allowing wider society to benefit from inventions and scientific discovery. The fact that patents and copyrights do not exist in perpetuity points towards the idea that they exist to promote invention and scientific discovery in ways that can be commercially exploited but only for a limited time, usually by their creator or inventor (or his heirs under certain circumstances).
Not sure what else to tell you. This really isn’t that complicated or controversial, in the US at least.
IDK.... its all bits and bytes. This seems to stifle innovation more than anything imo.
Also, it's just not true that products will be useless by the time their patents expire. Example: semaglutide patents have already expired in my country. Competitors immediately entered the market and it led to prices free falling to about a third of what they used to be. These past few months I had drug reps visit every week with lower and lower prices to the point it became hilarious to watch them try to outmatch each other. So many people are going to benefit from this. Looking forward to the day tirzepatide gets its turn.
What's next, I can charge people money for using the number 65234563124 because I'm the first to file a patent for it?
Absolutely.
Creative works are files in a computer, data. All data is information. All information is sequences of bits. All sequences of bits are numbers.
I can even calculate how big those numbers are given their bit count.
digits(bits) = ceil(bits * log10(2))
digits(32) = 10
digits(64) = 20
digits(128) = 39
digits(256) = 78
digits(512) = 155
digits(1024) = 309
digits(20 KiB) = 49,321
digits(2 GiB) = 5,171,655,946
That's how absurd this system is.
Patents, unless it's for something truly exceptional, just seem like a good system for getting lawyers rich.
Trademarks I believe in though. If I buy a Caterpillar bulldozer, I should be able to believe that I'm getting a genuine Caterpillar bulldozer, and no one else is allowed to use the name.
Our systems of intellectual properties have been abused, so it's fun to think about culture shifting to one of intellectual property nihilism, but I don't see that happening in my lifetime.
Tolerability is a function of how long the monopoly lasts. The social contract was we'd all pretend we couldn't trivially copy works for some years so that creators could make their well deserved profit, and then the works would enter the public domain. When's the last time you saw your culture enter the public domain? JK Rowling has made her fortunes several times over, yet you and I will be dead for around a century before those works become public.
Nearly two hundred years ago, one man warned everyone this would happen.
https://www.thepublicdomain.org/2014/07/24/macaulay-on-copyr...
> At present the holder of copyright has the public feeling on his side.
> Those who invade copyright are regarded as knaves who take the bread out of the mouths of deserving men.
> Pass this law: and that feeling is at an end.
> Men very different from the present race of piratical booksellers will soon infringe this intolerable monopoly.
> Great masses of capital will be constantly employed in the violation of the law.
> Every art will be employed to evade legal pursuit; and the whole nation will be in the plot
> once it ceases to be considered as wrong and discreditable to invade literary property, no person can say where the invasion will stop. The public seldom makes nice distinctions.
> The wholesome copyright which now exists will share in the disgrace and danger of the new copyright which you are about to create.
> in attempting to impose unreasonable restraints on the reprinting of the works of the dead, you have, to a great extent, annulled those restraints which now prevent men from pillaging and defrauding the living.
Did people listen? No. They doubled down on functionally infinite copyrights. If lasts even longer now. These are merely the results.
As it becomes cheaper to capture and process higher res content, we're bound to push it to bigger displays (VR included). I guess the human eye will be the limit when we're at retina levels of DPI on a sphere.
We'll see how much 360 content shows up, but the size gap is so big it basically needs to be handled entirely differently. A 16x9 screen, 40 degrees wide, is only 2% of a sphere.
Assuming the product companies accept the plateau on resolution (I agree it’s likely), there are many ways to exploit the other stuff. Colour space, tacling each type of compression artifact, additional channels (probly not long until we have HDR RGBW), etc etc and those will each come with their committees made of patent-hungry companies seeing non-encumbered encodings as “undesirable”
For a lot of use cases, including Basic and Main profile, H.264 is already free.
Something tells me H264 is going to be sticking around for a while just on sheer compatibility and compute requirements
Since MPEG-this Part that is hard to keep straight, this is the MPEG standard backing Xvid and DivX codecs. So let's encode video and torrent like it's 2002!
Let's have a meet up a Circuit City!!
I do believe that more modern codecs like h265 and AV1 have a lot to offer us, but having h264 be fully unencumbered by patents will be nice. By modern standards it is very well-supported in both hardware and software, computationally cheap to encode and decode, and still does a good enough job for most use cases. h264 doesn't seem like it will be displaced any time soon to the degree that h264 once displaced its own predecessors, though obviously that's not to say that we don't see or won't see more adoption of newer video codecs across the internet in general, just that I think the h264 long tail will be one of the longest long tails of a video codec.
And thanks to the availability of codecs like AV1 which are effectively not patent encumbered (seems there is no practical reason to take the threats otherwise seriously) it seems the era of patent encumbered media formats is slowly coming to a close. Good riddance.
edit: Bit of a mess but here is some source for the h264 expiration.
https://meta.wikimedia.org/wiki/Have_the_patents_for_H.264_M...
So it seems 2030-11-10 is the date where h264 (version 3) will become patent unencumbered world wide, a bit earlier for the U.S. The other profiles/newer versions like AVC/SVC will take a while longer.
Well I'm not sure about decode, but as of a few years ago weren't h.265 and AV1 able to do much better compression than h.264 at almost all levels of compute? https://people.videolan.org/~unlord/SVT-AV1_BD-rate.png And h.266 encoders can probably go faster now too.
However, at the same time, h265, h266 and AV1 are all vastly more complicated, and will have less broad hardware acceleration support than h264, which, again, even if it is not optimally efficient use of compute, just doesn't require a whole lot by modern standards.
So I'd argue there is really no reason to be rushing away from h264 unless you have a compelling reason. There are some obvious compelling reasons in some cases; nobody is going to be terribly surprised that an entity like Netflix is eager to switch to codecs that will save bandwidth, because at their scale saving bytes definitely adds up, bonus points if it can increase the quality at the same time.
On the other hand, though, unless you are absolutely sure you can switch to only new codecs, you will probably want to keep some h264 encodings around to act as a fallback baseline for legacy devices. And thus, you also have to take into account the complexity brought on by needing to store and maintain multiple encodings - in many use cases, like simple <video>s thrown into website backgrounds, I can see just eating the extra bandwidth costs and keeping it all h264 as a valid strategy.
(MP3 is in a similar place. AAC and Opus are technically superior, but everything supports MP3.)
Actually MP3 is more like the H.263 / Divx here. AAC-LC is more like H.264. The difference in support of AAC-LC and MP3 is near zero, both in terms of hardware and software. While AAC-LC is vastly more superior. AAC-LC has been declared patent free ( all patent expired ) by Redhat for 9 years already.
The only part that is not perfect is there isn't a decent open source AAC-LC encoder. While you could use QAAC using Apple's encoder in iTunes. It would still have been better if there was an open source version. Luckily we just have a new AAC encoder with FFmpeg, it is doing great for and although sill not Apple's quality. But hopefully improve in the future.
Or Apple could have just open source the god damn thing.
Still, I don't see the point. Unlike video, audio decoding is so cheap you can practically do it on software even on fairly constrained devices. So really only in places like Bluetooth headphones where codec complexity can have a direct impact on battery life does it actually factor in. I say for most purposes people should just be going Opus today: it is far ahead of basically anything else. In most cases, for old devices you can simply ship software codecs instead.
Hell, Wikipedia ships video software codecs in the form of ogv.js, mainly because a lot of Apple devices wouldn't expose VP8/VP9 support even on SoCs that had hardware support for it, and that works surprisingly well. So for something like Opus, obviously it is trivial even on older hardware, even in the confines of a web browser. (And of course, Wikipedia uses it for Opus as well, but my understanding is this is only necessary for 1-2% of Internet traffic since most devices/browsers support Opus these days)
That may leave some niches where AAC-LC is still a reasonable choice, like maybe old PMPs. However, I reckon that there are probably a lot of older PMPs that in fact, don't natively support AAC. For example, the trusty Sansa Clip+ from 2009 doesn't have AAC support. If you were to install third-party software such as Rockbox, well, then you'd have Opus support.
So for audio, I think barring any specific reason not to, the meta is to basically always choose Opus.
Yes. But subjective testing by IgorC ( of HydrogenAudio ) shows it wasn't as good as Apple. It was a lot better than previous FFMPEG though. I believe we will have a few more listening test results coming out soon.
>That may leave some niches...
iPod has over 80%+ of portable music players market share. And almost all the others shipped at the time, from Creative to other Chinese brands, car audio, TV and media player has AAC support. AAC-LC hit a sweet spot of near same MP3 hardware support while being so much better. 320Kbps MP3 still don't compete with 256Kbps AAC-LC on Apple Encoder. On the other hand, there are only a few samples where 256Kbps AAC-LC isn't transparent, barely noticeable difference but no artefacts compared to Opus in professional listening test, 95% of people wouldn't even notice. Opus do shines at 128Kbps to 160Kbps range where AAC-LC don't compete ( or dare I say wasn't even tuned ), but loses nearly all hardware compatibility.
If we were streaming, and storing music or audio tracks with 128Kbps then Opus would have had strong advantage. But we live in a world where we could even stream uncompressed audio files with plenty of bandwidth left. The file size of audio is so small it makes it negligible in consumer settings unless you are Youtube. ( I would still prefer a 256Kbps audio codec, for me it was instantly noticeable. I think Youtube at one point switched for premium but decided to switch it back )
For web then I guess there is an argument for Opus. But I keep saying the same with Video Codec as well. There is a whole world of Video and Audio Codec usage outside the web and PC / Smartphone. We could have used AAC and called it a day. And it is not just patent unencumbered, it is patent free.
If I was for a new Audio Codec, I just wish it could achieve what AAC-LC 256Kbps did with 128kbps. i.e I take quality first over potential bitrate savings. But the half bitrate marketing hasn't been true since MP3 Pro, HE-AAC, HEVC and VVC for "low bitrate". Neither HEVC or VVC could achieve what AVC did with 2Mbps in 1Mbps. VVC being very close, I would guess FVC / H.267 will finally be it, but at what complexity cost? We have hit sort of limit of compression. Both in Video and Audio. And today we have very different set of trade offs between file size, connection bandwidth, compatibility and encoding / decoding complexity then what we did 10-20 years ago. And I would choose compatibility + slightly larger file size for its simplicity.
I am not claiming that objective tests are the only tests that matter, but I am guessing Lynne tweaked the codec to win in both objective measures and Lynne's own subjective tests. I don't think tweaking it to do better on more subjective tests would be impossible though, just needs more work, probably. What I'm saying is, at least as far as CBR goes, I think FFmpeg's new NMR encoder is likely going to be the best soon if it isn't already.
There is the question of VBR still, though.
> iPod has over 80%+ of portable music players market share. And almost all the others shipped at the time, from Creative to other Chinese brands, car audio, TV and media player has AAC support. AAC-LC hit a sweet spot of near same MP3 hardware support while being so much better. 320Kbps MP3 still don't compete with 256Kbps AAC-LC on Apple Encoder. On the other hand, there are only a few samples where 256Kbps AAC-LC isn't transparent, barely noticeable difference but no artefacts compared to Opus in professional listening test, 95% of people wouldn't even notice. Opus do shines at 128Kbps to 160Kbps range where AAC-LC don't compete ( or dare I say wasn't even tuned ), but loses nearly all hardware compatibility.
Please note that I was talking about deploying Opus vs deploying AAC-LC, not whether you encode your personal collection to AAC-LC vs Opus or something like that. (My personal audio collection is mostly FLAC and gets Opus-transcoded on-the-fly.) I brought up my Sansa Clip+ not because PMPs are still relevant - let's be honest, they're not really relevant for anyone but us nerds - but just because it's an example of a device I personally owned that really doesn't support AAC-LC.
Still, a lot of old iPods can easily decode Opus if you run Rockbox on them as far as I know, so it's not like it isn't an option if you're a fan of PMPs still, which is pretty cool since they can be had for relatively cheap depending on what direction you go.
> If we were streaming, and storing music or audio tracks with 128Kbps then Opus would have had strong advantage. But we live in a world where we could even stream uncompressed audio files with plenty of bandwidth left. The file size of audio is so small it makes it negligible in consumer settings unless you are Youtube. ( I would still prefer a 256Kbps audio codec, for me it was instantly noticeable. I think Youtube at one point switched for premium but decided to switch it back )
> For web then I guess there is an argument for Opus. But I keep saying the same with Video Codec as well. There is a whole world of Video and Audio Codec usage outside the web and PC / Smartphone. We could have used AAC and called it a day. And it is not just patent unencumbered, it is patent free.
The argument for Opus in streaming is still strong on its own. If we pit 128kbps Opus against 256kbps AAC-LC, that's obviously a 50% reduction in bandwidth for similar quality.
Does it matter if we have so much bandwidth to spare? In my opinion, definitely. For people using metered data connections, which are quite common for mobile connections in the U.S., you can fit twice as much 128kbps Opus in your spare bandwidth versus 256kbps AAC-LC - around 1 megabyte every minute versus 2. The cheapest Google Fi plan has 15 GB of bandwidth per month - if you did nothing but listen to audio, I reckon that'd give you around 4 hours a day of 256kbps audio versus 8 hours a day of 128kbps audio. Since most people do more than just listen to audio, the actual difference this makes when data limit constrained is much bigger.
Does it matter if video bitrate trumps audio anyways? Yes, still. Consider the provider end of things. Obviously, YouTube isn't serving 256kbps AAC-LC, but if they were serving 256kbps AAC-LC to everyone, then assuming they have around 4 million people streaming YouTube at any given time, they would wind up saving at least over 100 petabytes of bandwidth per month. Which is probably not a huge amount relatively speaking, but it definitely would be worth doing. In actual reality, YouTube just simply won't serve 256kbps audio to everyone because it wouldn't make sense, so Opus just enables us to get quality we weren't going to get before. Similar calculus either way, though.
Why Opus and not AV1 then? Well, I think Opus and AV1 should be adopted especially where bandwidth is a question. YouTube, Netflix, really any big provider is going to want to adopt both. AV1 offers a similar magnitude of efficiency improvement over H.264 as Opus over AAC-LC. However, the great thing about Opus is that it is just super cheap to decode, so there are limited circumstances where we're limited by compute; the only cases where it would be relevant are in fact fixed function devices. AV1 and video in general is just a lot more computationally expensive to decode, so not having hardware acceleration would be a serious impediment, and even on the Internet, AV1 support is supposedly significantly less pervasive than Opus support, so you would need to fall back for a much larger percentage of users.
So the tradeoff calculus is different, IMO, for deploying AV1 and Opus. For deploying Opus, you will need to fall back for a very small percentage of users, and doing so is relatively inexpensive because you can literally just ship a software codec. For deploying AV1, you will need to fall back for more users and shipping a software codec isn't a serious option. Which leaves some potential reasoning for people to just stick to H.264 out of sheer simplicity, in spite of the efficiency cost.
For personal usage. Yes. But I was answering in terms of wider context in consumer. They don't know anything about Rockbox.
>The argument for Opus in streaming is still strong on its own. If we pit 128kbps Opus against 256kbps AAC-LC, that's obviously a 50% reduction in bandwidth for similar quality.
It is not similar quality. Not even close.
I think MPEG-1 and 2 (from which MP3 comes) might be even longer.
I don't see H.262 as becoming completely irrelevant any time soon, so it's clearly had a hell of a run, but I expect H.264 to wind up having a similarly long tail. I am not sure I'd bet on it being longer, but I wouldn't bet against it either. I'm only of the opinion that H.264 will have one of the longest of any video codec, but perhaps not the longest.
(Source: have written a H.263 decoder as a learning exercise.)
https://meta.wikimedia.org/wiki/Have_the_patents_for_H.264_M...
And, yes, I remember w32codecs until MPlayer's bundled set superseded them. And mencoder for sure ;)
There is also LCEVC ( Not connected with EVC ) that is a scaling layer on top of existing video codec. You can apply LCEVC on top of EVC, VVC, HEVC or AVC as well as AV1 and AV2. LCEVC has been selected by Brail as part of their TV 3.0 technology stack on top VVC.
While there are momentum on LCEVC, I actually wished EVC gained more momentum. I guess everything is too late now.