It’s pretty amazing people are still finding ways to make video smaller.
Is this just people being clever or is it also more processing power being thrown at the problem when decoding / encoding?
Is this just people being clever or is it also more processing power being thrown at the problem when decoding / encoding?
For example, changes from one frame to the next are encoded in rectangular areas called "superblocks" (similar to a https://en.wikipedia.org/wiki/Macroblock). You can "move" the blocks (warp them), define their change in terms of other parts of the same frame (intra-frame prediction) or by referencing previous frames (inter-frame prediction), and so on... but you have to do it within a block, as that's the basic element of the encoding.
The more tightly you can define blocks around the areas that are actually changing from frame to frame, the better. Also, it takes data to describe where these blocks are, so there are special limitations on how blocks are defined, to minimise how many bits are needed to describe them.
AV2 now lets you define blocks differently, which makes it easier to fit them around the areas of the frame that are changing. It has also doubled the size of the largest block, so if you have some really big movement on screen, it takes fewer blocks to encode that.
That's just one change, the headline improvement comes from all the different changes, but this is an important one.
There is new cleverness in the encoders, but they need to be given the tools to express that cleverness -- new agreement about what types of transforms, predictions, etc. are allowed and can be encoded in the bitstream.
Is there a reason codec's don't use the previous frame(s) as stored textures, and remap them on the screen? I can move a camera through room and a lot of the texture is just reprojectivetransformed.
I mean, that's more or less how it works already. But you still need a unit of granularity for the remapping. So the frame will store eg this block moves by this shift, this block by that shift etc.
This is exactly what I question. Why should there be block shaped units of granularity? defining a UV-textured 3D mesh that moves and carries previous decoded pixel values should have much less seams, with a textured mesh instead of blocks the only de novo pixel values would be the seams between reusable parts of the mesh, for example when an object rotates and reveals a newly visible part of its surface.
Having worked in the field of photogrammetry, I can tell you that it is a really complex task.
That's what AV1 calls global motion and warped motion. Motion deltas (translation/rotation/scaling) can be applied to the whole frame, and blocks can be sheared vertically/horizontally as well as moved.
Consider a scene with a couple of cars moving on a background, one can imagine a number of vertices around the contour of each car, and reusing the previous car, it makes no sense to force the shape of blocks. The smaller the seams between shapes (reusing previous frames as textures), the fewer pixels it needs to reconstitute de novo. The more accurate the remapping xy_old(x_prev,y_prev)-><x,y>, the lower the error signal that needs to be reconstructed.
Also the majority of new contour vertex locations can be reused as the old contour locations in the next frame decoding. Then only changes in contour vertexes over time need to be encoded, like when a new shape enters the scene, or a previously static object starts moving. So there is a lot of room for compression.
It's true you could still accidentally violate a patent but that minefield is clearing out as those patents simply have to become more esoteric in nature.
But that's not my main point. My main point is that we are going down a fitting path with codecs which makes it hard to come up with general patents that someone might stumble over. That makes patents developed by the MPEG group far less likely to apply to AOM. A lot of those more generally applicable patents, like the DCT for example, have expired.
1) it harms interoperability
2) I thought math wasn’t patentable?
At the absolute compression limit, it's no longer video, but a machine description of the scene conceptually equivalent to a textual script.
Clever matters a lot more for encoding. If you can determine good ways to figure out the motion information without trying them all, that gets you faster encoding speed. Decoding doesn't tend to have as much room for cleverness; the stream says to calculate the output from specific data, so you need to do that.
All of this requires a significant amount of extra logic gates/silicon area for hardware decoders, but the bit rate reduction is worth it.
For CPU decoders, the additional computational load is not so bad.
The real additional cost is for encoding because there’s more prediction tools to choose from for optimal compression. That’s why Google only does AV1 encoding for videos that are very popular: it doesn’t make sense to do it on videos that are seen by few.
Better codecs are an overall win for everyone involved.
They also get increased power usage, lesser battery life, higher energy bills, and potentially earlier device failures.
> Better codecs are an overall win for everyone involved.
Right.
Mobile/power constrained devices don't use software decoding, that just a path to miserable experience. Hardware decoding is basically required.
Meanwhile my desktop can SW decode 4k youtube with 3% reported cpu usage.
But, I mean, your expectation is not that unreasonable, computers were quite good by 2013. It is just an eye-opening framing.
I like how you padded this list by repeating the same thing thrice. Like, increased power usage is obviously going to lead to higher energy bills.
And it’s especially weird because it’s not true? The current SOTA codec AV1 is at a sweet spot for both compression and energy demand (https://arxiv.org/html/2402.09001v1). Consumers are not worse off!
I don’t remember ever watching a movie and wishing for a better codec, in the last 10 years
I do wish ATSC1 would adopt a newer codec (and maybe they will), most of the broadcasters cram too many subchannels in their 20mbps and a better codec would help for a while. ATSC3 has a better video codec and more efficient physical encoding, but it also DRM and a new proprietary audio codec, so it's not helpful for me.
And there’s no transfer of effort to the user. Compute complexity of video codecs is asymmetric. The decode is several order of magnitude cheaper to compute than the encode. And in every case, the principal barrier to codec adoption has been hardware acceleration. Pretty much every device on earth has a hardware-accelerated h264 decoder.
It feels like we’re losing something, a shared experience, in favor of an increasingly narcissistic attitude that everything needs to shapeable to individual preferences instead of accepting things as they are.
I’d be somewhat interested in something like a git that generates movies, that my friends can push to.
Extremely widespread mass media fiction broadcast are sort of an aberration of the last 75 years or so. I mean, you’d have works in ancient times—the Odyssey—that are shared across a culture. But, that was still a story customize by each teller, and those sorts of stories were rare. Canon was mainly a concern of religions.
It’s just for fun, we give it far too much weight nowadays.
I find the idea fun, kinda like using snapchat filters on characters, but in practice I'm sure it'll be used to cut corners and prevent the actual creative vision from being shown which saddens me.