Film grain synthesis in AV1 (2019)
norkin.org
norkin.org
Agreed though, it does feel weird as a vfx artist to be bringing down the quality of cg to that of film stock.
The very first codecs from On2 (then called DuckCorp) -- TrueMotion and TrueMotion2 (used in games such as FF7) -- used a compression technique that did a good job of filtering, then on playback simulating film grain. We used this effect to good advantage, demoing exclusively 24 fps film-sourced material and properly dealing with frame rate using 3/2 pulldown.
Everything old is new again...
What would undermine it is imperfect removal of real noise or grain from the source material: Then you'd have some remnants of the original grain, plus the synthesized grain... an incorrect result. It seems that this scheme requires perfectly clean input images.
I hope this idea is taken further, and it becomes the norm among codecs or wrappers to provide room for an author-defined post-processing shader to be applied after decompression.
If you removed the noise, it would sound a bit weird. For telephone calls in particular, the background noise lets you know that the phone call is connected. If you filter it out, people think that the call was dropped.
> and estimating the film grain parameters from flat regions of the difference between the noisy and de-noised versions of the video sequence.
Storage and bandwidth are only getting cheaper.
These noise filter will sometimes be temporal, rarely will be motion compensating (because that's computationally expensive) and as a result can't get as good a result as I can.
So these grain removal techniques are in many ways a political compromise between people who just want the smaller video, and the people who want the grain.
I'd be fascinated to see how codecs with and without this feature diverge in terms of encoding changes. Presumably there were some big gains just sitting there but the minority that preferred grain stopped then being exploited.
Much of the drama around encoder quality has came down to this dichotomy. Choosing to lose the grain/detail was attacked as plastic or fake by the people who care about it, while people who didn't could point to better test scores. Which then caused more arguments about the validity of the measures.
Sure, for the typical Hollywood blockbuster it has not much value and can break the immersion, but for everything else grain can make you look on a picture, rather than into it. This could be the difference between looking through a window and looking at a painting for example. Additionally grain can emphasize the passing of time. A shot of a still life without grain is something different from the very same shot with grain.
Today it is just another tool in the aesthetical toolbox. And someone somewhere decides when to have it and when not to have it.
People today miss the point that things like grain can be an intentional directorial choice and not an artifact to be removed.
However, for a lot of film shot pre-2010, grain is an artifact that's not there by choice or artistically. Other than perhaps the film 300.
It just so happens that most of my personal media fits in that box of being pre-2010 which is why I generally denoise.
Something like 30 rock, or The Office, for example, don't have film grain because of some artistic choice.
Interestingly, though, "That 70s' show" does in a few cases even though it's somewhat noisy be default. That's a more tough call.
However, for a bunch of film and especially tv shows, the choice had FAR less to do with aesthetics and was more related to cost. Big budget films certainly could pick any sort of equipment/stock they wanted. Lower budget productions didn't necessarily have that luxury.
I'm not saying it didn't happen, rather that the choice in stock was more often than not "cheap while being as clear as we can afford".
Again, you see this with modern film where grain is almost never added (except for specific scenes trying to give more of a dated effect). That, to me, says cinematographers aren't generally trying to pick their stock to add grain. Some do, but that's the exception and not the rule.
Well it says the noise is specified exactly by the bitstream. I'm not sure how much lenience there is for the decoder in the postprocess option.
Today, Netflix doesn't need to use less bandwidth just to make room for Hobomax and Paramount+.
You could do things like 480p training videos with ultra-crisp text, and snow/fire/etc effects.
You'd probably have to prevent it from having network access or else you'd just be asking for the videos to rot over time, but you could give them an embedding API or even access to a few small bits of data, so you could have a YouTube video that changes with seasons and location.
If you were strict about the no network access thing you could give access to all sensors, since the data would never leave the device.
YouTube's already got some limited interactivity features, DVDs had them, why not all videos? It would mostly be a novelty, but you could also do really cool stuff like use Bluetooth beacons to skip around to certain parts for an interactive tour, although you might have to enforce downloading the whole thing before playing if you wanted 100% no privacy leaks.
But there is definitely a trend towards more customizable decoders, with a larger set of primitive operations (DCT, motion estimation/compensation, convolution filters) and parameters. The codec space is slowly moving towards a model where the decoders are VM highly optimized for specific tasks.
Here's my attempt at creating a grainy video effect in the browser using a media stream (from the device's camera - the page should ask for permission first). Very much an MVP; I'm already working on making the dithering algorithm more efficient. https://codepen.io/kaliedarik/pen/OJOaOZz
> YouTube's already got some limited interactivity features, DVDs had them, why not all videos?
I don't know about videos, but again this is doable in the browser using the canvas. For instance, this example: https://scrawl-v8.rikweb.org.uk/demo/canvas-027.html - the swans, and both geese, are clickable (to their respective Wikipedia pages). The navigation around each goose is a clickable box, while the navigation on the swans uses a chroma-key effect to limit the clickable area.
The answer always is that the technique allows better use of bandwidth, so you can have a better image without increasing bandwidth. Or if you're able to increase the bandwidth, you can have even better picture with the technique than without it (until the bandwidth is so high that you can send the video uncompressed, but that's not happening anytime soon for video on the web).
I still buy blu-rays though so I am a firm believer in the "just throw more bits at it" solution.
> The correct answer to this question is that often the choice is not between the original film grain and the synthesized one. When video is transmitted over a channel with limited bandwidth, the choice is often between not having the film grain at all (or having the grain significantly distorted by compression) and having synthesized grain that looks subjectively similar to the original one.
Film grain doesn’t compress well. If you buy a 1080p Blu-ray disk, you can get a bit rate of something like 40 Mbit/s. Go watch 1080p video on Netflix, and you’re going to get something closer to 5 Mbit/s. Yes, the codecs are different—yes, you can talk all you want about how everything is going to be fast when we all get 5G—but this is still the ground reality that most people are working with. The bit rate is more of a constraint you have to work within than a variable you can just tweak to get the results you want.
By comparison, ProRes 422 has a target bit rate of 147 Mbit/s for 1920x1080@29.97.
Would you take a 1080p BluRay over 4K Netflix stream? I'm legitimately curious. I feel like we sometimes prioritize higher resolutions over overall detail.
Note that plenty of movies, in the theater, were never done in 4K to begin with. 4K is great for selling TVs, but cinematographers and directors seem to be more lukewarm on using 4K. I think most movies are still done at 2K. Your UHD Blu-ray Disc might just be an upscale.
Disney notably dragged their feet, but starting this year all their new MCU movies are finally being mastered with a 4k intermediate.
oof, every single day
> I feel like we sometimes prioritize higher resolutions over overall detail.
Yes, resolution over detail (/bitrate). Netflix 4k has half the bitrate or blueray 1080p
Netflix 4k is <20mbps, blueray 1080 is up to 40mbps, bluray 4k is up to 128mbps
Gloats in New Zealand.
Obviously a streaming service can't expect all their customers to have this kind of high-speed internet, but dynamically changing video quality on the fly to meet bandwidth limitations is a problem they all solved a very long time ago. It is not unreasonable for Netflix to offer such high bitrate streams to customers with the bandwidth for it.
As for hardware, pretty much any hardware h.264 decoder made in the last decade should have no problem with blu-ray spec AVC streams.