Netflix is now doing per-shot encoding for UHD content
netflixtechblog.com
netflixtechblog.com
Also, something that's kind of weird about all this is that encoders (x264, x265, etc) already have their own rate control algorithms that decide on a frame or scene basis how many bits to use. Netflix taking an approach like this is equivalent to claiming that they're capable of doing a better job than these codecs in an automated way - so why not just contribute the code needed to achieve these improvements back to the projects they're using?
Last, it's very weird to select individual frames from an encode and not a selection of frames. The comparisons purport to show that the lower bitrate encode is better than the higher bitrate encode, but in fact (if I understand how they're using rate control correctly) what they are showing is a single frame in the lower bitrate encode that uses more bits than the same frame in the higher bitrate encode. So it's arguably not a fair comparison. And even with that, depending on how sensitive you are to artifacts, some of the "optimized ladder" encodes still look worse.
Edit: here is an example of what I mean when I said that they may have made the quality at 1080p worse. If what you end up watching is the highest bitrate 1080p available, you would get higher quality on the old latter than on the newer one. https://i.imgur.com/rzPR7Sh.jpg Actually, on some of the examples, even 720p on the older ladder is better than 1080p on the newer one!
Maybe they will? But it matters most for 4K performance, so that's where they started?
> encoders (x264, x265, etc) already have their own rate control algorithms
But they simply operate on a stream without taking future frames into account. This does a first entire pass on the film holistically to determine where keyframes should go and settings per-shot. It can't be backported to codecs because they work linearly.
> And even with that, depending on how sensitive you are to artifacts, some of the "optimized ladder" encodes still look worse.
The artifacts look worse only because it's zoomed in so crazy far. On a cinema screen, the additional sharpness will be clear, the artifacts not so much.
Given that x264 already has a 2 pass mode, I don't see why that is necessarily the case. Even CRF mode uses mbtree by default, which is a pretty complicated rate control algorithm. x264 also has pretty intelligent keyframe determination, I almost always see I frames on scene transitions.
> On a cinema screen, the additional sharpness will be clear, the artifacts not so much.
I think it's probably the other way around. The closer you are to a screen, the more likely you are to notice increased resolution. But if there are large patches of the image full of artifacts, you'll be likely to see that even far away.
x264 will not align keyframes across resolutions and encodes. In addition the pershot encodes optimize more than just idr frame placement. They also optimize other encoder parameters such as aq.
> I think it's probably the other way around. The closer you are to a screen, the more likely you are to notice increased resolution. But if there are large patches of the image full of artifacts, you'll be likely to see that even far away.
That isn't how this works. It's not a strict tradeoff between more artifacts and less sharpness by changing resolution. Downscaling -> Upscaling is simply another form of lossy compression. It may look better or worse than using those bits in another spot.
Sure, that's true. (Though I don't know why it matters that keyframes aren't aligned.) But at the end of the day the point is that Netflix has a better rate control algorithm, and this could be built into x264, even if it might require a significant amount of work. (Which I'm sure the x264 developers would be willing to do for a substantial quality improvement.)
> That isn't how this works. It's not a strict tradeoff between more artifacts and less sharpness by changing resolution.
Of course it's not a strict tradeoff. It's a loose one. And yes, it may look better or worse. That's really my only point in that section of the comment: that bumping up the resolution earlier in the ladder as they're doing is not a pure win, and it may look worse to some people depending on their viewing conditions.
You can't ABR adapt without aligned GOP boundaries.
> Of course it's not a strict tradeoff. It's a loose one. And yes, it may look better or worse. That's really my only point in that section of the comment: that bumping up the resolution earlier in the ladder as they're doing is not a pure win, and it may look worse to some people depending on their viewing conditions.
Of course not, that's why the perform an analysis of both options and select the better one. That's what the algorithm does...
Yes, that's true of course, but not really relevant to whether you could port Netflix's work on scene-adaptive rate control to x264. Maybe you'd lose aligned GOPs... but for a lot of purposes (offline?) that doesn't matter.
> Of course not, that's why the perform an analysis of both options and select the better one. That's what the algorithm does...
The only point I've ever tried to make on this subject is that in some cases this approach fails. "Perform an analysis" is such a high level description that it misses the fact that this is being done according to some objective metric that may disagree with an individual viewer's personal preferences or viewing environment. In fact, just because an objective metric says 4k > 1080p doesn't mean the difference will be noticeable at the viewing distance the viewer is at, whereas the additional artifacts introduced by moving from 1080 -> 4k without a significant bitrate increase may very well be visible!
The best way to understand why this is used is to think of a movie — when there are shots that are totally, absolutely black, like scene changes, normal 1-pass CBR encoding uses the exact same amount of data to that part as it uses for complex action scene. But by using VBR and multipass, encoder “knows” that this piece is OK with lower bitrate and that bitrate can be then used for more complex scenes, thus creating better quality for those scenes that require more bitrate.
Also, you're making a slight mistake. About a decade ago (?), x264 was changed so that even 2-pass mode is really just using CRF under the hood. It's not "achieving the best quality", it's just figuring out what quantizers you need to hit your average bitrate target exactly. Some more information about that here: https://trac.ffmpeg.org/wiki/Encode/H.264
> normal 1-pass CBR encoding
You may be confusing constant bitrate (which no one should ever use) with CRF (constant rate factor), which varies the bitrate from frame to frame and scene to scene, without having to do a 2 pass encode.
This really is their core business and somewhere they can have a competitive edge, though. Even if you think it's unethical or that it might make business sense to contribute back to the upstream projects, you can see why they might want to keep some of that secret sauce in-house.
Is that still the case? They switched from x265 to beamr for HEVC. Not sure if they are still on x264 on recent encodes.
No they have not made it worse. 1080p encoded at lower bitrate can look better than 720p at a higher bitrate. Or vice versa. Even to the trained eye. It depends on the scene dynamics.* In the ladders they posted (called a RD chart) we can clearly see they have made a better selection of the resolution to encode at a given target bitrate increase the video quality. They have posted the BD rates for this stuff, the gains are substantial (30% or more visual quality at the same bitrate).
> Also, something that's kind of weird about all this is that encoders (x264, x265, etc) already have their own rate control algorithms that decide on a frame or scene basis how many bits to use. Netflix taking an approach like this is equivalent to claiming that they're capable of doing a better job than these codecs in an automated way - so why not just contribute the code needed to achieve these improvements back to the projects they're using?
Because they've built a split and stitch encoder on top of the codecs. That means the code is built around x264, not into it.
> Last, it's very weird to select individual frames from an encode and not a selection of frames. The comparisons purport to show that the lower bitrate encode is better than the higher bitrate encode, but in fact (if I understand how they're using rate control correctly) what they are showing is a single frame in the lower bitrate encode that uses more bits than the same frame in the higher bitrate encode. So it's arguably not a fair comparison. And even with that, depending on how sensitive you are to artifacts, some of the "optimized ladder" encodes still look worse.
This is extremely true.
* High action scenes tend to benefit from lower resolution with more bits spent on inter compression while slower scenes benefit from tighter intra compression.
Well, of course that's correct. Take 1080p encoded at 100 Mbps and 720 encoded at 300 Mbps, I guarantee you that 1080p wins every time. :-)
> In the ladders they posted (called a RD chart) we can clearly see they have made a better selection of the resolution to encode at a given target bitrate increase the video quality.
I understand how an RD chart works, but I disagree with your conclusions. Reread my original post carefully - my point is that if you're stuck with 1080p as the maximum resolution, than the quality you can get with the revised ladder is strictly worse, because Netflix will want to jump up to UHD resolutions, but your device won't support it. (And even assuming that the UHD will be better based on this chart is kind of a leap, because it depends on what artifacts you're most sensitive to. Objective metrics ≠ subjective experience. In several of their comparisons, I prefer the lower res version because the new one is simply too artifacted.)
> Because they've built a split and stitch encoder on top of the codecs. That means the code is built around x264, not into it.
I understand the technical difference, but I'm still not convinced. Fundamentally what you get at the end of the day is a single continuous h.264 stream. The quantizer is chosen based on some rate control algorithm, and IPB frame choices are made by another algorithm. Netflix's improvement amounts to having a better rate control, that understands scene differences and uses lower quantizers on more complex scenes. There's fundamentally no reason why you couldn't add such a rate control algorithm to x264. It might require using a 2-pass mode, but it could be done.
These are for their HEVC encodes. Apple (and most others) require H264 to have a separate ladder which will be 1080p limited and will be generated on different encode set and maximum video quality.* Almost any HEVC capable device can go to 4k, even if the screen is only 1080p. It will be used than down scaled.
> (And even assuming that the UHD will be better based on this chart is kind of a leap, because it depends on what artifacts you're most sensitive to. Objective metrics ≠ subjective experience. In several of their comparisons, I prefer the lower res version because the new one is simply too artifacted.)
Certainly objective and subjective metrics have differences, but VMAF does correlate pretty tightly with subjective metrics. This is a very detailed subject, I summed up many of my thoughts on the differences in my demuxed talk last year here: https://www.youtube.com/watch?v=nCUsXhSPyyw
* https://developer.apple.com/documentation/http_live_streamin... 1.23 iirc
Really? I wasn't aware Netflix even offered anything other than H.264 at <= 1080p. I know Youtube doesn't. If this is all HEVC maybe you're right and the resolution issue isn't such a big deal. If this is actually the case I wish they would have said so explicitly in the blog post.
> VMAF does correlate pretty tightly with subjective metrics. This is a very detailed subject, I summed up many of my thoughts on the differences in my demuxed talk last year here: https://www.youtube.com/watch?v=nCUsXhSPyyw
Thanks, I'll check that out when I have the time. I'd still emphasize that it's quite possible for objective metrics to be fooled (that's why codecs have psy tunings!), and I do prefer several "old-ladder" images over the newer versions in Netflix comparisons here. I suppose I've become rather cynical about claims made on the basis of metrics ("now 50% less bitrate for the same quality!"), when to my eye, web video has always been very bad and is getting worse as companies keep reducing the bitrate.
You can confirm this by loading a video, right clicking somewhere in the player and opening 'stats for nerds'
[youtube] dQw4w9WgXcQ: Downloading webpage
[info] Available formats for dQw4w9WgXcQ:
format code extension resolution note
249 webm audio only tiny 49k , opus @ 50k (48000Hz), 1.18MiB
250 webm audio only tiny 65k , opus @ 70k (48000Hz), 1.55MiB
140 m4a audio only tiny 130k , m4a_dash container, mp4a.40.2@128k (44100Hz), 3.27MiB
251 webm audio only tiny 136k , opus @160k (48000Hz), 3.28MiB
394 mp4 256x144 144p 73k , av01.0.00M.08, 25fps, video only, 1.72MiB
278 webm 256x144 144p 97k , webm container, vp9, 25fps, video only, 2.25MiB
160 mp4 256x144 144p 107k , avc1.4d400c, 25fps, video only, 2.05MiB
395 mp4 426x240 240p 159k , av01.0.00M.08, 25fps, video only, 3.42MiB
242 webm 426x240 240p 217k , vp9, 25fps, video only, 4.04MiB
133 mp4 426x240 240p 290k , avc1.4d4015, 25fps, video only, 4.48MiB
396 mp4 640x360 360p 340k , av01.0.01M.08, 25fps, video only, 6.68MiB
243 webm 640x360 360p 396k , vp9, 25fps, video only, 6.96MiB
134 mp4 640x360 360p 484k , avc1.4d401e, 25fps, video only, 8.27MiB
244 webm 854x480 480p 586k , vp9, 25fps, video only, 10.03MiB
397 mp4 854x480 480p 603k , av01.0.04M.08, 25fps, video only, 11.37MiB
135 mp4 854x480 480p 741k , avc1.4d401e, 25fps, video only, 11.56MiB
247 webm 1280x720 720p 1035k , vp9, 25fps, video only, 17.67MiB
136 mp4 1280x720 720p 1077k , avc1.4d401f, 25fps, video only, 16.72MiB
398 mp4 1280x720 720p 1133k , av01.0.05M.08, 25fps, video only, 22.07MiB
399 mp4 1920x1080 1080p 2106k , av01.0.08M.08, 25fps, video only, 40.74MiB
248 webm 1920x1080 1080p 2666k , vp9, 25fps, video only, 58.46MiB
137 mp4 1920x1080 1080p 4640k , avc1.640028, 25fps, video only, 78.96MiB
18 mp4 640x360 360p 601k , avc1.42001E, 25fps, mp4a.40.2@ 96k (44100Hz), 15.19MiB (best)"As a side note, we do have some additional points, not shown in the plots, that are used in resolution limited scenarios — such as a streaming session limited to 720p or 1080p highest encoding resolution. Such points lie under (or to the right of) the convex hull main ladder curve but allow quality to ramp up in resolution limited scenarios."
No it doesn't. Depending on the distance from the screen and indeed the screen resolution, you won't benefit from the extra 1080p resolution, but you could benefit from the extra bits. 1080p60 is 3.7GBit uncompressed with 10 bit 444, so that's 37:1 compression. 720p60 is 1.7gbit, or 5.5:1 compression, so far fewer compression artifacts.
And the graph you posted isn't quite showing what you're saying it is: these show the "4K" curves. When you're on a 1080-max device but have a strong connection, there are higher-bitrate 1080 streams available; they're just not on this chart because that's not what it's showing.
In other words, what the curves are showing is just that they're switching to higher resolutions at lower bitrates than before, when your device supports them. Really the way to read it would be drawing lines vertically: given a fixed bandwidth, the "optimized" encode is always giving higher quality and often higher resolution.
From just a cursory inspection using my Apple TV 4K, 12 Mbps is their target bitrate but their ceiling was ~16 Mbps.
That's still only a third of what AppleTV+ offers (36/48) and only 2/3 of Disney+ (18/24). Netflix is still higher than Prime Video (10/14). HBO Max was the lowest quality, allowing only 8/10 Mbps for their HD streams (which generously doubled HBO Now's infamously low 4/5 Mbps bitrate).
I've never cared too much about the quality though. The old school dvd rips that were compressed to 700mb to fit on a CD were usually fine by me.
I'm confused, how are you getting this information? Does Apple TV have a way to display the bitrate ceiling?
Also, do you find the bitrate differences noticeable? On my 4k tv, I find Netflix looks pretty good, but Amazon looks pretty bad and HBO Now is awful of course (no Max yet, since I use a Roku.) This might be enough to get me to check out AppleTV+.
Does your router report its current throughput?
Yeah, there's a clear difference, especially on Disney & Apple. Apple is very close to 4K blu-ray (there's some YouTube vids comparing them).
I'm using a low end 4K projector, so it's more apparent.
However, what has been driving me insane with all these services is how the bitrate is completely inconsistent throughout, depending on network congestion. Every service I subscribe to will "automagically" lower the bitrate if the network can't handle it.
Which is fine, I get it. The thing is, it's the only way to watch anything, and depending on the content, it can absolutely ruin it. I'm on Comcast, and Netflix will constantly ping pong between 480p and 1080p, and is very unpleasant to watch. It's ok for some shows, but certain ones like Planet Earth 2 become unwatchable (to me).
I really wish there was a setting to add a bit of wait time before playing to avoid this. ATV+ buffers a bit more than the rest, but has the same issues.
Edit: To clarify, this issue happens both wired and wireless. The main contributor to the inconsistency is the time of day I decide to watch.
I still use optical media myself.
Plus, with optical media you can easily make bitperfect copies!
On my desktop I always rent stuff through iTunes just cause I can download it in advance, and don’t have to deal with buffering or reduced quality.
Assuming x264 encoding, that sounds about right for 720p vs 1080p.
I keep thinking if their reason was to push for higher capacity iPhone sold.
For true videophiles there's Kaleidescape but it's less valuable now that there are so many exclusive shows.
I'm surprised that Apple TV apps don't allow you to download content, considering it is based on essentially iOS, but it is clearly a conscious choice.
A scary thought (they highjack NXDOMAIN, MiTM http to "bring you important messages", etc)
It’s kinda annoying though I get the business reasons and all that.
It’s just that it did used to be better and it’s gotten worse over time that really annoys me.
Though I guess there's a difference between allowing the user to switch to it vs making it default. Youtube lowered the default to 480 to help with congestion during the pandemic, but you can still manually still. I assume 99% of users use the default which is why they can get away with it.
But it would be kinda silly for Netflix to provide 48mbps but only to those who know to enable it.
I'm very suspicious of these numbers, I think they might be the maximum transient bitrate and not the average bitrate. Do you have an example of an AppleTV movie or show that actually has an average bitrate that high?
I was very surprised by Apple's numbers (and HBO), so I did some Googling to confirm.
Netflix is superior in every regard (easy to browse, quick to buffer, if I rewind it’s instant) but even to a naked eye Apple’s quality is way higher. Less smudgy, more definition.
It’s so bad that I will pay Apple $3 per episode than watch whatever garbage bitrate Netflix has decided is suitable for anime.
To my eyes, Apple TV+ consistently offers the highest quality images, with a 23 minute anime episode using 700MB+ total instead of 70-170MB I see on Netflix. Even Amazon is a huge drop in quality compared to Apple TV+, although it is better than Netflix.
If so, Netflix is on the wrong side of the CPU war with AMD ascending.
So why bother locking your paying customers out of 4K content? What's the upside?
Also: I'm reasonably convinced that NetFlix is violating the law, at least in Australia, by advertising that their content is 4K and then arbitrarily blocking access to the 4K streams. This is called "bait & switch", and the fines are eyewatering, even for large corporations.
I figured it was some backdoor Intel scheme where they paid Netflix to be exclusive.
Which is nuts, because you can buy HDCP stripper boxes for a few tens of dollars from several Chinese suppliers.
Nah, they have their asses covered. I went to the website and tried to sign up, and at the first mention of 4k (when you're choosing a plan), there's a disclaimer of:
>HD and Ultra HD availability subject to your Internet service and device capabilities. Not all content available in HD or Ultra HD. See Terms of Use for more details.
If you go to the terms of use, it says:
>4.7. The quality of the display of the Netflix content may vary from device to device, and may be affected by a variety of factors, such as your location, the bandwidth available through and/or speed of your Internet connection. HD, Ultra HD and HDR availability is subject to your Internet service and device capabilities
The phrasing makes it sound like external conditions, outside of Netflix's control are the reason 4K may not be available.
The reality is they sell two subscription levels: 4K and a HD-only one, but then after you've given them your money, they outright block access to 4K streams on a device that is physically capable of playing the stream.
I can play 3D games in 4K 60fps. I can stream YouTube at 8K. I can watch blu-ray at 4K with 2% CPU utilisation. For crying out loud, I have an NVIDIA RTX 2080 Ti and gigabit fibre!
But no. Sorry sir, your CPU is the wrong model, and Netflix cannot trust it. No 4K for you.
I agree that the phrasing isn't the clearest, but it's not like the actual requirements aren't available. A search for "netflix 4k" turns up this page as the first result: https://help.netflix.com/en/node/13444, which clearly lists what the requirements are.
Nowhere on that page does it mention HDCP. There is no such thing as a firmware update for 99.9% of monitors out there.
I meet 100% of the stated requirements on this page, yet NetFlix refuses to play 4K on my high-end computer. It does this silently, without explanation. It simply shows everything as HD, basically gaslighting me.
This kind of thing infuriates me.
I paid for this. I feel like a sucker.
And according to https://help.netflix.com/en/node/23931 you can use any cpu if you have an NVIDIA gpu starting with 3gb version of 1050.
Most people don't realize this but you can only watch Netflix in 4K in a browser if you're using Microsoft Edge on Windows or Safari in macOS 11. In fact, you can only get 1080p if you're running Chrome in Chrome OS, or IE, or Safari. All other browsers, including Chrome and FF on macOS, Linux, and Windows, are stuck at 720p.
Their blog post[0] from 2017 sounded hopeful for higher resolution video on Linux - they wrote "We... look forward to high-resolution video being available on more platforms soon" almost immediately after announcing FF on Linux was supported - but in hindsight it's clear that what they really meant was what you point out - support for things like Apple TV, Xbox, and TV Smart Apps.
[0]: https://netflixtechblog.com/update-on-html5-video-for-netfli...
Edge hasn't been around as long as Netflix has provided in-browser service, I'd have thought Chrome w/ Widevine would have been part of Netflix' success.
And of course because I’m on Linux I can’t get even 1080p encoded at a shitty bitrate unless I force it with a Firefox extension that they can gimp at any time (in a way, many thanks to the engineers who made the extension possible by relaxing checks on the browser).
I don’t understand why most platforms hate their paying users like this - I can and will pirate a Blu-ray quality rip if I can’t pay for it easily.
They don't. They're just very noisy.
Question for anyone here:
What do you use to play 4K/HDR? I have an Apple TV 4K, which can do 4K Dolby Vision playback and looks ok, but the Apple TV tends to have some jittering when streaming certain shows (very noticeable in panning shots of animation). The sound quality is also noticeably worse on my set with it (it doesn’t seem to be able to do direct pass through to my AV unit, always PCM/decoded on the Apple TV).
On the other hand I have a a Shield TV that does direct pass through of audio and sounds much better, and also seems to do video playback without that occasional jittering. It does not seem to support Dolby Vision though. Additionally the UI is very, very, very laggy after recent updates.
Does anyone use anything that doesn’t have either of these issues?
As it is, Apple has screwed up HDR on the 4K so badly that it's very difficult to make it work smoothly without also screwing up HDR on other devices.
My Apple TV 4k obviously runs much snappier but i find the HDR quality really dark and lower than native Android TV.
I don't have the best speakers or room acoustics, but I can't complain about the sound.
How don't more people complain about this? I avoid streaming on the Apple TV because it does some sort of bizarre framerate thunking that is just brutal for panning. Do so few people use the product that it just goes unnoticed?
My LG 4K TV has fantastic Netflix, Prime, and Disney+ clients. HDR, 4K, etc.
I wish it would just give the direct output to the TV and receiver (video audio). No idea why they the need to process it all on device.
That’s probably the ticket for it though. I find the lack of audio pass through to be the more annoying piece though. I have a machine that costs significantly more than the Apple TV and has knowledge of all of the attached speakers to do that decoding.
That‘s the correct behavior though and not specific to the Apple TV. Any device that actually does switch and match the content’s frame rate will cause the output display to resync/adjust with a short black screen. The alternative of not adjusting frame rate is much worse and it’s such an underrated problem in video playback in my opinion. The Apple TV handles this better than most devices.
Also, most high-end TVs are able to recover the original 24p (or other) frame rate and thus remove the judder by using specific settings: https://www.rtings.com/tv/reviews/lg/c9-oled/settings#judder
I tested with a high-speed camera that with correct settings my old Samsung UE75H6475 was able to recover the original frame rate perfectly from quite a few framerate mangling combinations. Haven't done similar testing with my current C9, though.
This is probably the case. DAZN took over NFL streaming in Canada and for the first two years seemed to use their existing European soccer processing chain (they might still --- I gave it a try to years straight and then gave up). So the 60/30 NFL stream was re-encoded to 25/50, and then on playback on my set would be displayed at 30/60. It was brutal, and even if displayed at 25 or 50 FPS was still brutal because they were seriously corrupting the NFL stream.
I tried it across a number of devices -- AppleTV, Chromecast, different TVs, pads, laptops -- and it was just unbelievably intolerable to me. Every panning pass was the horrendous juddering mess. Yet somehow no one seemed to have a problem with this! In discussions it seemed to be a non-issue.
Google simply "decided" for Chromecast users that 60 Hz is fine and that nobody needs 24 fps or 50 fps content.
Solution was to buy "premium" HDMI cables that are rated for 4K HDR
If that’s how it works that would be awesome as I bought an expensive 11.2 receiver not long ago, but when the new consoles are out I want to have VRR and ALLM, which means HDMI 2.1, which means it has to go straight into the TV, which means I would lose Dolby Atmos (assuming PS5 will support it of course, but that seems pretty likely)
I also have a second setup which is similar to the above in a dedicated cinema room with projector as well, but with even more speakers.
If you have a receiver, it should be where your video playback devices attach. Only the receiver attaches to the TV.
Even old XviD would reliably always insert a new I-frame / start a new GOP on a scenecut, and perform global rate optimization based on scene complexity within the target ABR parameters.
https://netflixtechblog.com/dynamic-optimizer-a-perceptual-v...
https://netflixtechblog.com/per-title-encode-optimization-7e...
A few relevant points:
- Their previous systems used fixed keyframes, so wouldn't be using scene-change detection at all (I presume this was to allow predictable chunking across different codecs)
- Since reliable streaming performance is a pretty big deal for Netflix, they probably have quite tight restrictions on VBR modes, which make them not work as effectively since they have less "room" to work with
Math, science, history, unraveling the mystery <as the pixels start to unravel>, it all started with a Big Bang. BANG (Bang, your Screen is a riot of random pixels. Oh wait, here’s the cast instead!)
Then I caught an episode on another service, maybe Netflix? The transition wasn’t awesome, but it wasn’t awful either. It only glitched for a frame or two instead of a few seconds. Clearly better heuristics were in play.
(I saw a similar problem with a famous monarch butterfly footage that I don’t have time to chase down)
Keep in mind this is on Youtube and uploaded by some shitty clip site, so it's a very biased example. In this case there are four lossy encode steps involved: master -> bluray -> clipsite's master -> Youtube encode. Youtube in particular has absolutely horrific, just inexcusably bad bitrate even at the highest quality they'll give you. I'd have to see what the original Bluray looks like in this case to see if there's really a problem worth worrying about here.
I fear people who grow up in the age of streaming might not realise that DVDs had good video quality because streaming services seem to hate SD content.
> As a side note, we do have some additional points, not shown in the plots, that are used in resolution limited scenarios — such as a streaming session limited to 720p or 1080p highest encoding resolution. Such points lie under (or to the right of) the convex hull main ladder curve but allow quality to ramp up in resolution limited scenarios.
Literally if they could just fix this I would have no problems with streaming quality.
(it's digital rights management and "Woo, PCs are scary!")
yes, because it saves you money - netflix could be increasing its subscription cost, or it could keep it down as the subscriber base grows, by using technique like this.
As long as you don't notice the difference, what's wrong with them saving some bandwidth?
Therefore, a technical solution is the next best option.
Here's hoping someone from Netflix is in the comments and can act on it, because their support system hasn't done anything in the 2 years since I brought it to their attention.
edit: Actually, I tried to track down the film used in the new encoding (I think it's The Dirt from the signs and dates seen in the frame captures) so I could screenshot for comparison. It actually worked in 21:9 fullscreen.
Was it the new encoding? I saw this problem up until this past week, most recently on Maniac but I checked that show and it's no longer an issue.
From the post I suspect it was this adding in extra blacks at top and bottom on 21:9 content in the old method:
> with fixed 4K resolution bitrates — 8, 10, 12 and 16 Mbps — regardless of content characteristics
But I truly don't know what it was. I just know that video no longer ignores 40% of my screen real estate by watching on a 21:9 monitor since sometime last week.
THANK YOU! That has been annoying me for years.
But Please add 3440x1440 to your testing. It's not shown in those charts.
also, what is the movie or show sampled in that blog?
Edit: Why the downvotes?
At least, that is my understanding.
The harder part is the significantly increased compute use due to re-encoding things multiple times, to detect these cuts and to try to find the best encoding. Heuristics there can be arbitrarily complex and re-calculate any number of times. I imagine it hasn't been done earlier just due to cost, though maybe they've recently achieved a better heuristic.
edit: ah, great, they link to a "dynamic optimizer" post that goes into this in some detail: https://netflixtechblog.com/dynamic-optimizer-a-perceptual-v...
And for dynamic encoding like this: when it's wrong, it's not visually worse in that scene than choosing that sub-par encoding for the entire movie, which has the same "choose the best encoding" problem as individual chunks have. I assume it'd be relatively rare for it to result in anything worse than a one-shot strategy.
---
ffmpeg will let you easily do frame-to-frame-diff logic that lets you chop videos into scenes, for example: https://video.stackexchange.com/a/30701 I'm not sure how much it handles compressed-frame differences, but it shouldn't be too hard to build around it. Just might be a bit beyond bash-friendly.
It might sound trivial, but it is still extra computational work to figure out when the shot changes.
In order for this to be the default, you'd need either humans, or pattern recognition algorithms to identify the chunks. You also need to quantify how by how much each chunk needs its encoding parameters tweaked.
Monetary costs aside, that's increasing complexity of your pipeline with relatively small gains. I'd bet that more companies will start looking at similar approaches now that 4K HDR (and 8K) are becoming more common. Probably not worth the R&D for 1080p, but we'll see tricks like this start to trickle down, I'm sure.
It's highly unlikely we'll see anything similar in FOSS tools in the near future.
Ages ago they solved this by making some frames more important than others. Key frames being the top priority. If you are falling behind you can drop everything you are working on and skip to the next key frame. Then they added other priority levels so you can dump part of the work instead of all of it when things are only a little choppy.
With a zip file if you don’t have the middle of the file you can’t figure out the end. So clearly video is doing something different if they can remove chunks and still work.
Many of our compression techniques for video “hang” off the key frames. All the pictures around them are described as a set of changes to the key frame. Much smaller, but also why your screen occasionally looks like a horror show. Something went wrong and the changes were shown using the wrong starting point.
If the key frame is right before the scene changes completely, then you will struggle to compress the transition. There’s not enough space to describe all the changes. But if you have a program look for the transitions, you can delay or advance the key frame a bit so it lands exactly on the camera change. Bang. New picture. Then everything looks right.
Today it’s gone a lot farther. There are hints and patterns that span a whole movie or a chunk of it that for instance clues in that the whole movie is shot in sepia tone or a dark cave so you can guess the next scene will be sepia too. They’re talking about lining those attributes up with big shifts in the camera work, like someone coming out of a cave into bright light.
This chart clearly shows multiple data points per resolution under the old fixed-ladder scheme, and under the new scheme the new data point for 1080p is lower on the video quality scale than any of the old 720p encodes. That's a pretty significant difference, and my question about where these new encodes fit into their existing price structure remains valid.
> As a side note, we do have some additional points, not shown in the plots, that are used in resolution limited scenarios — such as a streaming session limited to 720p or 1080p highest encoding resolution. Such points lie under (or to the right of) the convex hull main ladder curve but allow quality to ramp up in resolution limited scenarios.
Edit: Looking at their animated GIF comparisons, the newer encodes definitely give a much sharper image, but also have a lot more in the way of ugly artifacts:
On the left, under the tree branch: https://miro.medium.com/max/2000/1*SQQkYltbVC-HT-l8GpN-7Q.gi...
The right side of the frame: https://miro.medium.com/max/2000/1*A37BBzK4Ap8JPhmiUiFKcw.gi...
Many of the areas that should be fairly flat looking: https://miro.medium.com/max/2000/1*qI3EY7HDp3p0gwK9YGqchQ.gi...
There's definitely a lot more ringing and similar artifacts in the newer, lower-bitrate samples despite their overall increase in effective resolution.
They're showing the curve that is used for connections that want to be streaming 4k. So if you're on the 1080 stream in the charted scenario, it's because you're bandwidth-limited. They have higher-quality, higher-bitrate encoding curves for lower-resolution connections. However, it doesn't make sense to use that 1080 encoding for a 4k connection. If you could afford the bitrate, you'd rather be streaming the higher resolution from the displayed curve.
Sure, the first graph shows 1080p at higher quality for lower bitrate to the 720p on the optimized ladder, but if I'm resolution-limited (on a phone) and Netflix isn't streaming higher quality and downsampling on the device, the changes here show that my perception of quality will suffer. (I feel for folks stuck with a 720p screen, if they're just streaming native resolution it looks like they'll be getting worse than previous 480p quality levels.)