AV1@Scale: Film Grain Synthesis, The Awakening

Regarding aesthetics, I don't think AV1 synthesized grain takes into account the size of the grains in the source video, so chunky grain from an old film source, with its big silver halide crystals, will appear as fine grain in the synthesis, which looks wrong (this might be mitigated by a good film denoiser). It also doesn't model film's separate color components properly, but supposedly that doesn't matter because Netflix's video sources are often chroma subsampled to begin with: https://norkin.org/pdf/DCC_2018_AV1_film_grain.pdf

Disclaimer: I just read about this stuff casually so I could be wrong.

If you have a few static frames and average them, you improve SNR by retaining the unchanged signal and having the purely random noise cancel itself out. Retaining noise itself is not useful.

I suspect the effect you might be seeing is either just an aesthetic preference for the original grain behavior, or that you are comparing low bandwidth content with heavy compression artifacts like smoothing/low pass filtering (not storing fine detail saves significant bandwidth) to high bandwidth versions that maintain full detail, entirely unrelated to the grain overlaid on top.

To illustrate the temporal aspect: consider a traditional film projector. Between every frame, we actually see complete darkness for a short time. We could call that darkness "noise", and if we were to linger on that moment, we'd see nothing of the original signal. But since our visual systems tend to temporally average things out to a degree, we barely even notice that flicker (https://en.wikipedia.org/wiki/Flicker_fusion_threshold). I suspect noise and grain are perceived in a similar way, where they become less pronounced compared to the stable parts of the signal/image.

Astrophotographers stack noisy images to obtain images with higher SNR. I think our brains do a bit of that too, and it doesn't mean we're hallucinating detail that isn't there; the recorded noise - over time - returns to the mean, and that mean represents a clearer representation of the actual signal (though not entirely, due to systematic/non-random noise, but that's often less significant).

Denoising algorithms that operate on individual frames don't have that context, so they will lose detail (or will try to compensate by guessing). AV1 doesn't specify a specific algorithm to use, so I suppose in theory, a smart algorithm could use the temporal context to preserve some additional detail.

Together with external noise generation (something like https://old.reddit.com/r/AV1/comments/r86nsb/custom_photonno...) that'd support external denoising (pass the reference and denoised video to get the grain spec), the whole FGS thing would be much more interesting.

I would love for them to provide an option to view it with film simulation vs without.

One of my favorite movies of all time, The Holdovers, did film simulation extremely well. It's set in the '70s so it attempts to look like a movie of that era.

It looked great to me, but if you're an actual film nerd you're going to notice a lot of things aren't exactly accurate.

Maybe in the near future we'll see Netflix being able to process some post effects on the client. So if you're color blind, you get a mode for that. If you don't want fake grain you can turn it off.

That's true, but at a given bitrate (until you get to very high bitrates), the compressed original will usually look worse and less sharp because so many bits are spent trying to encode the original grain. As a result, that original grain tends to get "smeared" over larger areas, making it look muddy. You lose sharpness in areas of the actual scene because it's trying (and often failing) to encode sharp grains.

Film Grain Synthesis makes sense for streaming where bandwidth is limited, but I'll agree that in the examples, the synthesized grain doesn't look very grain-like. And, depending on the amount and method of denoising, it can definitely blur details from the scene.

that's an understatement. it just looks like RGB noise effect was added. film grain does not look like RGB noise. to me, film grain is only one part of what gave film the film look. the way the highlights bloom rather than clip. it also was more natural/organic/some descriptive other than the ultrasharp of modern digital acquisition. using some SoftFX or Black Mist type filters help, but it's just not the same as it is a digital vs analog type of acquisition. all of these attempts at making something look like it's not just keep falling down in the same ways. but hey, there's a cool tech blog about it this time. film grain filters have been around for a long time, yet people just don't care for them. even in Blu-ray time frame, there was attempts at removing the grain in the encode and applying it in playback. Netflix isn't coming up with anything new, and apparently nothing exciting either based on the results.

A few things to note:

- still-frames are also a mediocre way to evaluate video quality.

- a theoretically perfect[1] noise-removal filter will always look less detailed than the original source, since your brain/eye system will invent more detail for a noisy image than for a blurry image.

1: By which I mean a filter that preserves 100% of the non-grain detail present, not one that magically recovers detail lost due to noise.

I don't know the psychovisuals behind that. Maybe it adds some high frequencies that compression often washes out, or maybe acts like some kind of dithering.

As for your eyes, I am pretty sure that they have grain, that's how quantum physics work, you just don't perceive it because your brain filters it out. But again, I don't know how it interacts with film grain.

A child watching a Buster Keaton skit and gasping and giggling and enjoying it is going to have a different subjective aesthetic experience of the media than a film critic who knows exactly what type of film and camera were used, and what the meaning of all the different abstractions imply about the scene, and the fabric of Keaton's costume, and so on, and so forth.

Subjective aesthetic preferences are in the realm of cognition - we need a formal theory of intelligence mapped to the human brain, and all of these subjective phenomena collapse into individualized data processing and initial conditions.

There's something about film grain contrasted against clean cel animation which might make it easier for people to suspend disbelief. They are conditioned to think that absence of grain is associated with unreal animation, particular types of media, and CGI. Home video and news and so forth had grain and low quality, so grain gets correlated with "real". In my view, there's nothing deeper than that - we're the product of our times. In 40 years, media will have changed, and it may be that film grain is associated with surrealism, or edited out completely, as it's fundamentally noise.

I have to imagine past glassmakers would have been absolutely enthralled by the ability we now have to make uniform, large sheets of glass, but here we are emulating the compromises they had to make because we are used to how it looks.

this is likely the result of ~100 years of film-based filmmaking and projection. hell, we still call it filmmaking.

Look around you: nearly all surfaces have some kind of fine texture and are not visually uniform. When this is recorded as video, the fine texture is diminished due to things like camera optics, limited resolution, and compression smoothing. Film grain supplies some of the high frequency visual stimulus that was lost.

Our eyes and brains like that high frequency stimulation and aren't choosy about whether the exact noise pattern from the original scene is reproduced. That's why the x265 video encoder (which doesn't have grain synthesis since it produces H.265 video) has a psy-rd parameter that basically says, "try to keep the compressed video as 'energetic' as the original, even if the energy isn't in the exact same spot", and even a psy-rdoq parameter that says, "prefer higher 'energy' in general". These parameters can be adjusted to make a compressed video look better without needing to store more data.

It might be that there is a large part of the population that still has that association.

Cinephiles are also more likely to watch older (i.e. with grain) movies that ARE well shot and beautiful (which is why they are classics and watched by cinephiles) and not see bad film movies, only the cream of the crop, while being exposed to the whole gamut of quality when watching todays movies shot digitally. Would reinforce that grain = good while not being necessarily the case - and their opinion might be heard more than gen pop.

At any rate, it can be a neat tool to lower sharpness!

They definitely do at night when it's dark out. There's a kind of "sparkling" or "static" that comes in faint light.

Fortunately, our eyes have way better sensitivity than cameras. But the "realism" just comes from how it was captured using the technology of the day. It's no different from phonograph hiss or the way a CRT signal blurs. The idea is to be "real" to the technology that the filmmaker used, and the way they knew their movie would be seen.

It's the same way Van Gogh's brush strokes were real to his paintings. You wouldn't want his oil paintings sanded down to become flat. It's the reality of the original medium. And so even when we have a digital print of the film, we want to retain as much of the reality of the original as we can.

I've since learned that not everybody sees the world like I do, but I still do love to see grain and noise in pictures. Only RGB noise I often find dreadfully ugly when looked at up close, which is a shame, since that is exactly what most color cameras include.

Does it add any more than modern video compression techniques? What constitutes noise in cinema, is somewhat subjective.

Well ackchually -- illumination is inherently random, so all time-bounded captures of a scene (including what your eyes do) are subject to shot noise: https://en.wikipedia.org/wiki/Shot_noise

The author did not do a particularly good job of selling the merits of film grain. I am not sure what he meant by that but I will say that it does increase the perceived sharpness of the image as well as perceived detail, even if it is just an illusion. The cinematographer Steve Yedlin describes it as giving the viewer’s eyes something to “latch” onto.

Fake lights, fake shadows, fake sky, ...

What matters to Netflix is the bandwith/quality ratio, that’s it. Removing noise before and adding it in software at playback time will produces a significantly better result, at similar bit rate. Because compressing noise produces horrible results.

People remember the emotions the artwork engendered, and thus the whole work is associated with the feelings, flaws and all. If the work is particularly widely known, the flaws can become a stand-in for the work itself.

I see this in video games - I'm fond of the NES-era "flaws" and limitations (palette limits, sprite limits, sound channel limits), but less connected to the Atari 2600 or SNES/PS1/NDS/etc flaws. Shovel Knight is charming; A Short Hike, while great, doesn't resonate on a style level.

I've read that a 15-perf 65mm IMAX negative shot with slower film stocks is "virtually grainless", even when viewed on a 70ft screen. Grain is apparently noticeable in IMAX films when large/fast stocks are used and pushed toward their limits, and (of course) when smaller-format film stocks have been blown up.

There are two possible advantages for this kind of grain synthesis. For Netflix, they could produce the same perceived quality at lower bitrates, which reduces costs per view and allows customers with marginally slow connections to get a higher quality version. For a consumer, the advantage would be getting more non-grain detail for a fixed bitrate.

You are right that if you subtract the dentists frame from the raw one, showing only the estimated noise, you would get some impression of the scene. I think there’s two reasons for this. Firstly, the places where the denoiser produced a blurry line that should be sharp may show up as faint lines. I don’t think this is ‘hidden information’ so much as it is information lost to lossy compression. In the same way, if you look at the difference between a raw image and one with compression, you may see some emphasized edges due to compression artefacts. Secondly, the less exposed regions of the film will have more noise so noisiness becomes a proxy for darkness, allowing some reproduction of the scene. I would expect this detail to be lost after adjusting for the piecewise linear function for grain intensity at different brightness levels.

Perhaps a third thing is the level of noise in the blacks and the ‘grain size’ or other statistical properties tell you about the kind of film being used, but I think those things are captured in the film grain simulation model.

Possibly there are some other artefacts like evidence of special effects, post processing, etc.

I just want to say this is brilliant suggestion. Shit tier DVD rips of Newsradio really be elevated on shaderglass.

Film grain simulation can be an effect, but that's not the primary application being described here. As @ricardobeat succinctly put it elsewhere in this post, this process usually supports "a reproduction of the original grain pattern, compressed separately from the underlying image content. The result is closer to the original picture than the denoised/compressed version".

> If kids grew up without seeing film grain, how would they feel about it?

Film isn't quite dead yet, with ~7% of features in 2025 still being shot on film and maybe 1% more taking the digital→film→digital (a.k.a. "analog intermediate") route. Unless their media diet is all super-recent and low-quality, even today's kids will have most likely seen many movies shot on film with visible grain. Popular candidates would include the Harry Potter franchise, most episodes of the Star Wars franchise, early Marvel movies, many animated Disney and Studio Ghibli features, etc.

1. You prefer Betamax or VHS to digital media (highly unlikely)

2. You own laserdiscs (limited to 480i)

3. You own 35mm prints of film.

Since all other formats film has been made available on are both digital media and compressed.

All that is 24fps.

That's without audio, which I assume you also want to be uncompressed.