AI Poised to Transform Video Compression Landscape

The thing about video compression is that it's already very compute intensive. Most devices have dedicated hardware cores for encoding and decoding in order to save power. They work exclusively with integer values, chiefly unsigned 8 and 10 bit values with sequential and block memory transactions.

Humans not only design for compression quality, we also really focus on cost. And we design direct to what is cheap in hardware: integer adds, shifts, binary logic.

I do believe a different paradigm of ML is necessary for this particular application. The neural primitives used need to be more closely related to transistors than to matrix multiplications.

At a very (very!) abstract level AI is compression and compression is AI.

There's been rumblings about how Google's AI edge would lead to jumps in video and image coding but nothing big seems to have happened yet, though lots of micro-optimisations appear to be ML based.

Some suggested that the AV1 throw a bunch of tools at it spec design was built for AI to use.

Possibly VMAF is the biggest ML impact on codecs so far.

It's so amusing how AI became AGI and ML became AI. It used to be a marketing thing and now it's on the IEEE Web site.

> Yet, despite a decade-long effort, full neural-video compression—using deep learning—has not beat the best configurations of conventional codec standards in normal conditions. Reviews from third parties show that when benchmarked with conventional distortion metrics as well as human opinion scores, conventional video encoders still outperform neural-network compression, especially when conventional encoders are enhanced with AI tools.

I don't know how much better AI can do in practice.

If you sit down and really think through how an intraframe compression technique works - something like JPEG or MPEG - You may come to the conclusion that we are already playing with the fringes of our reality.

Consider The DCT and quantization. When a piece of information is transformed by The DCT, we talk about "energy compaction". That is to say, we took a block of information and determined how to represent it as a perfect (lossless) arrangement of amplitudes at given frequencies. This results in most of the important information (in human terms) condensing around the DC component in the top-left corner. We then use something explicitly engineered (i.e. 'trained') on the human visual system to quantize these figures into levels the computer can tolerate (incurring loss). Zig-zag encoding is practically an emergent mathematical mandate at this point...

Then, take a long look at how we are doing interframe compression. Forward & backward difference frames, frames that are differences of differences, predicted vectors of information, etc. A lot of this is very obvious, but there are tradeoffs to be made that perhaps a human should be in charge of - I.e. subjective measures of latency in video streaming relative to group-of-picture size.

Consider this - How could an AI system improve on techniques that optimize for the Human Visual System? A very large part of our multimedia compression today comes from strategically sacrificing information that is subjectively worthless to humans.

All that said, I am sure you can find some low-hanging fruit. For example, if you go and ask ChatGPT to make JPEG work better, it may helpfully suggest swapping Huffman for arithmetic coding. Not exactly a groundbreaking discovery in my book, but it would make a big difference.

I’m working on research for a journal article on this right now. Basically, get a bunch of videos you want compressed, use an auto encoder and just store the very small latent space variables instead of the video. Keep the decoder sitting around that was trained on those videos, and badaboom you can take a 512x512 movie and store it in 16x16x3 or another similarly sized latent space. However, this only works for fairly consistent video. Right now it works best for constantly monitoring video systems of physical phenomena. I.e. if you have multiple cameras pointed at the ocean to watch waves, you’re going to end up training a latent space that describes the wave physics. These small latent spaces are also the key to removing the black box. You can grab the latent spaces and connect the fields to physical quantities. For example, the wave reconstruction latent space has examples of velocity or wave height (variables needed to mathematically construct a wave field). Cool stuff!

I have yet to see a true AI ( or ML ) based Video Codec, or some potential Generative Video Codec like what Google did with Audio For Voice. And if it cant be done for Audio I doubt it will work on Video given the complexity. The current working prototype of H.267 ECM 8.0 aims to squeeze another 40% to 50% reduction in bitrate from H.266 / VVC Codec. There will likely be some ML usage on the Encoder side, but in terms of codec the exploration path hasn't provided much evidence how it will fundamentally change the codec efficiency.

I just realized there's inevitably going to be dedicated hardware to help speed things up with ai related tasks. The GPU helps significantly but I'd assume something even more suitable could be made.

At some point it'll become like the "Be kind, rewind" movie where an entire movie can be compressed to a text description of each scene then and entirely regenerated on demand by AI.

I believe Zoom already utilizes deep learning-based compression to some degree. I have seen some compression artifacts on video calls that look exactly like CNN hallucinations (e.g. https://thenewstack.io/deep-learning-neural-networks-google-...)

I remember reading about fractal-based image compression in the 1990’s. IIRC, there was some product out there that used it (I vaguely remember product ads with a paint can in them) but I don’t think it ever took off.

I wonder if coupling an AI to help find self-similar portions of the image could revive this idea?

It feels like if you’re willing to make computation >> storage, which is likely the bad optimization right now, you can use generative AI trained on the original video and key frames chosen to optimize size : fidelity and let the AI fill in the frames dynamically.

I'm going to draw a line with AI at what I call the "eyes on forehead" problem. With Ai Image rendering, there is the emergent trend of AI images being like a horror film, with faceless and extra-features on faces.

When AI is using a LLM dataset to fill in the gaps, it becomes a problem, because it's guessing what should be there when the "natural" noise would be better depicting nothing. Predictive vectors such as SVG use smooth images. i think the rasterization of a dataset turns single pixels into quasi-intelligible datapoints, but when treated indendently by a dumb AI, ends up rendering eye balls on a forehead. This is your problem, not mine. Please fix it.