Note: while I work at Netflix, I do not work on anything related to video encoding, these are just my semi-informed understandings :)
Note: while I work at Netflix, I do not work on anything related to video encoding, these are just my semi-informed understandings :)
The main enabler of more advanced compression strategies is the higher performance available to the encoder. Decoders are essentially deterministic state machines. Encoders however have to search a large space of transformations to find ones that captures the entropy for a given situation.
In the development of AV1 they called these transformation "tools" and the research encoder experimented with lots of them and only the most profitable ones made it into the standard.
The compression can thus still evolve even with the AV1 standard frozen, just like how x264 and LAME have gotten so much better over their lifetime.
That's quite thought provoking.
What do you see when the current generation codec is heavily bitrate constrained? First thing you'll notice are the ugly blocks with non-matching colors on the edges. Such primitive expression of lack of bitrate is showing quite clearly these codecs have very little understanding.
With a codec of the (probably far) future where both encoder and decoder have good "understanding", you wouldn't see any blocks, lower color depth, smudged details or so. If some detail is missing because of constrained bitrate, decoder will just fill it in intelligently.
It can go crazy far - when constrained with bit rate, the encoder can just send sort of a movie script - "Man stands on the beach and looks into the distance" and the decoder will "understand" it and render it in awesome detail including adding non-mentioned splashing waves, man's thinking expression and perhaps sunset (decoder effectively becomes little automated film director who takes missing information in movie script as artistic license). It will most probably look very different from the original movie, but it will be believable - if you didn't see the original then you won't really have a reason to suspect it's fake.
With more bitrate available you can add more detail to the movie script - the man is black and old, he wears this and that which the decoder would take into account when reconstructing the video. At some point you're always constrained on the bitrate, but "understanding decoder" can always go deeper and fill in more realistic details when needed.
It is true that for lossless compression there are constraints on the size of the input and the size of the compressed artifact+decoder, but I believe for lossy compression, the fourth element is the perceptive capabilities (or preferences) of human minds.