ELI5?
Edit: Why the downvotes?
Edit: Why the downvotes?
At least, that is my understanding.
The harder part is the significantly increased compute use due to re-encoding things multiple times, to detect these cuts and to try to find the best encoding. Heuristics there can be arbitrarily complex and re-calculate any number of times. I imagine it hasn't been done earlier just due to cost, though maybe they've recently achieved a better heuristic.
edit: ah, great, they link to a "dynamic optimizer" post that goes into this in some detail: https://netflixtechblog.com/dynamic-optimizer-a-perceptual-v...
And for dynamic encoding like this: when it's wrong, it's not visually worse in that scene than choosing that sub-par encoding for the entire movie, which has the same "choose the best encoding" problem as individual chunks have. I assume it'd be relatively rare for it to result in anything worse than a one-shot strategy.
---
ffmpeg will let you easily do frame-to-frame-diff logic that lets you chop videos into scenes, for example: https://video.stackexchange.com/a/30701 I'm not sure how much it handles compressed-frame differences, but it shouldn't be too hard to build around it. Just might be a bit beyond bash-friendly.
In order for this to be the default, you'd need either humans, or pattern recognition algorithms to identify the chunks. You also need to quantify how by how much each chunk needs its encoding parameters tweaked.
Monetary costs aside, that's increasing complexity of your pipeline with relatively small gains. I'd bet that more companies will start looking at similar approaches now that 4K HDR (and 8K) are becoming more common. Probably not worth the R&D for 1080p, but we'll see tricks like this start to trickle down, I'm sure.
It's highly unlikely we'll see anything similar in FOSS tools in the near future.
It might sound trivial, but it is still extra computational work to figure out when the shot changes.
Ages ago they solved this by making some frames more important than others. Key frames being the top priority. If you are falling behind you can drop everything you are working on and skip to the next key frame. Then they added other priority levels so you can dump part of the work instead of all of it when things are only a little choppy.
With a zip file if you don’t have the middle of the file you can’t figure out the end. So clearly video is doing something different if they can remove chunks and still work.
Many of our compression techniques for video “hang” off the key frames. All the pictures around them are described as a set of changes to the key frame. Much smaller, but also why your screen occasionally looks like a horror show. Something went wrong and the changes were shown using the wrong starting point.
If the key frame is right before the scene changes completely, then you will struggle to compress the transition. There’s not enough space to describe all the changes. But if you have a program look for the transitions, you can delay or advance the key frame a bit so it lands exactly on the camera change. Bang. New picture. Then everything looks right.
Today it’s gone a lot farther. There are hints and patterns that span a whole movie or a chunk of it that for instance clues in that the whole movie is shot in sepia tone or a dark cave so you can guess the next scene will be sepia too. They’re talking about lining those attributes up with big shifts in the camera work, like someone coming out of a cave into bright light.