> Yet, despite a decade-long effort, full neural-video compression—using deep learning—has not beat the best configurations of conventional codec standards in normal conditions. Reviews from third parties show that when benchmarked with conventional distortion metrics as well as human opinion scores, conventional video encoders still outperform neural-network compression, especially when conventional encoders are enhanced with AI tools.

I don't know how much better AI can do in practice.

If you sit down and really think through how an intraframe compression technique works - something like JPEG or MPEG - You may come to the conclusion that we are already playing with the fringes of our reality.

Consider The DCT and quantization. When a piece of information is transformed by The DCT, we talk about "energy compaction". That is to say, we took a block of information and determined how to represent it as a perfect (lossless) arrangement of amplitudes at given frequencies. This results in most of the important information (in human terms) condensing around the DC component in the top-left corner. We then use something explicitly engineered (i.e. 'trained') on the human visual system to quantize these figures into levels the computer can tolerate (incurring loss). Zig-zag encoding is practically an emergent mathematical mandate at this point...

Then, take a long look at how we are doing interframe compression. Forward & backward difference frames, frames that are differences of differences, predicted vectors of information, etc. A lot of this is very obvious, but there are tradeoffs to be made that perhaps a human should be in charge of - I.e. subjective measures of latency in video streaming relative to group-of-picture size.

Consider this - How could an AI system improve on techniques that optimize for the Human Visual System? A very large part of our multimedia compression today comes from strategically sacrificing information that is subjectively worthless to humans.

All that said, I am sure you can find some low-hanging fruit. For example, if you go and ask ChatGPT to make JPEG work better, it may helpfully suggest swapping Huffman for arithmetic coding. Not exactly a groundbreaking discovery in my book, but it would make a big difference.