Adventures with ffmpeg and color ranges
facebook.com
facebook.com
If the story was hosted on a personal blog I recognised (like Julia Evans to take a random example), then perhaps the domain would be enough for me to choose to read the article to see if the content is interesting.
If the link was to a broader domain that hosts many writers, I might use the parts after the domain to filter my interest: example.com/news-opinion vs example.com/science-blog gives me more information that might not be clear from the post title.
With a site like Facebook, which hosts so much different value levels of content, it is harder for me to visit facebook.com/233555323 rather than facebook.com/ffmpeg-interest-group/post6665443. For whatever reason, the URL remains something I use to filter my internet consumption.
Perhaps it has something to do with the amount of scams or other general reasons not to trust non-human readable URLs, or perhaps it is more due to my distrust of article titles (sensationalism) and seeking a second source to verify my interest. Perhaps I simply bias against Facebook - I don't know.
Regardless, I visited the story after thinking about this, and was very surprised to see it was an article by John Carmack - an author I certainly respect. It is a shame for me that both the URL and the title don't reflect that.
https://www.ft.com/content/13be9132-8149-11e9-b592-5fe435b57...
Not if done right, like e.g. stack overflow, see https://stackoverflow.com/questions/517355/string-formatting...
The human readable text is actually meaningless, and anything after the id will redirect there, e.g. https://stackoverflow.com/questions/517355/this-could-be-any...
meta.stackexchange.com/questions/148454/why-does-stack-overflow-use-title-slugs/
John Carmack works for Oculus/Facebook, so that's probably part of the reason he posts on Facebook, but you might mean you were surprised because the URL gave no indication.
If you have enough control over the decoder and renderer to make that hack work, you should just be producing 10-bit HEVC/AV1 video – which does not suffer from these 8-bit banding issues – with a modern color space (Rec 2020, PQ, etc.).
And if you don't have choice or control over the decoder and renderer, then it's a good sign that this would be a fragile hack.
From Rec. 2020 wikipedia article: Since a larger color space increases the difference between colors an increase of 1-bit per sample is needed for Rec. 2020 to equal or exceed the color precision of Rec. 709.
The "correct" solution is to use a modern colorspace (rec 2020) with 10 bits encoding (or 12 bits, if you can). UHD standard mandates at least 10 bits.
On the '821 you could gain 1 bit by degrading to an older color space - 709.
Full 0-255 range should never be used - they are invalid according to the standard. None of the hardware codecs are validated for 0-255 - you're asking for trouble here. Expect ghosting and weird motion artifacts accumulating from one I-frame to the next I-frame.
YT and other platforms have no problem ingesting these videos. Also the RPi Broadcom chipset does not seem to care.
Let's say I have a video with full RGB range 0-255 video. Why would 10-bit be a better option than 8-bit?
I was just complaining about this a couple weeks ago with some friends after the “Dark” episode of GoT. I was saying the codecs don’t seem to have caught up or they are just being fed at too low a bit rate to really excel at dark scenes.
It’s hard to imagine this is just a scaling issue with the color ranges, and if so, how could it possibly not have been fixed?!
I have a high end OLED screen in a very dark room, and I was very disappointed by the picture quality. Enough to wish I hadn’t watched it through HBO Go and that I had a better source.
I would love to see some before/after scenes showing the effect of the patch.
I think he meant to say that the limited dynamic range of 8-bit video manifests most obviously in dark scenes, since that's where standard gamma curves are the most imprecise relative to our vision (which was fine back when TVs couldn't reproduce that range anyway.) The 14% from full-range doesn't help that much; really you need the 4x finer precision you get from 10bit.
Unfortunately for you, high end OLED in a very dark room is literally the most likely conditions under which you'd be annoyed by these problems... :-)
HDR is the way to go.
Perhaps bitrate compromises have to be made when you’re streaming to so many people at once, but it’s not usually so glaring.
It will be interesting to see if it looks better when a higher-quality Bluray version comes out.
https://news.avclub.com/was-game-of-thrones-the-long-night-t...
https://pbs.twimg.com/media/D5SmDG7WwAAMhdf.jpg:large
I think that particular screenshot makes it look even worse than it was, but it was definitely sub-par. I assume that a hardcopy 4K Bluray would not exhibit the same effects?
B) The issues you're seeing with background banding are _not_, primarily colorspace issues. While having a larger colorspace helps, these are largely related to the way quantization is done for the macro blocks in h.264.
The original h.264 spec (which is baseline and main profiles) only had a exponential quantization matrix.
During quantization we are "rounding" to the nearest integer for pixels. If we imagine each macroblock only had _one_ frequency coefficient. Dark scenes would have a low coefficient, say 1.4 If we round to 1 that's a 28.5% error in the rounding. A brighter scene which is at 9.4 has a rounding error of only 4.3%.
Custom quantization matrices and non linear quantization matrices(which are available in newer codecs) help deal with this.
Advanced h264 encoders have variance adaptive quantization technologies to help choose what level of quantization to apply to a block.
In short: h264 has technological limitations that are not caused by colorspace and the encoder doing this encode doesn't have strong AQ.
I have encoded video for over a decade, and I don't think a two month period has gone by where I haven't learned something that convinces me to completely change how or what flags I'm using.
This is way beyond that but still.
Maybe I'm just dumb and not in the domain enough to get a decent handle but everytime I use ffmpeg I feel like a Norse Shaman trying through trial and error to locate the correct orientation of runes for the blessing.
Disclosure: FFmpeg contributor and Vimeo employee.
the second problem is that ffmpeg has a lot of features, and nobody wants to go through and fully document hundreds of formats, codecs, and filters.
Ffmpeg does a great job of guessing _what_ you want. But as soon as you're outside of those guesses you need to know _a lot_ to get things working.
It used scripts, and it required external tools to do the encoding but despite the complexity, I found it less arcane than ffmpeg command lines for similar operations.
Once I got better at understanding FFmpeg's filters there really was no looking back.
> For arcane reasons related to TV broadcast limitations, many video formats restrict the YUV color components to be in the 16..235 or 16..240 range instead of the full 0..255 range. Losing 14% of the already-barely-enough 8 bit dynamic range is bad enough, but it also often results in the black range starting at a quite visible grey value because most players don’t rescale the range. This is usually visible as ugly banding or blocking in dark scenes.
Of course players "rescale" the range. (It's not exactly rescaling, it's just that a value of 235 IS reference white in Rec. 709.) In fact, the thought if them not doing this is kind of crazy, as Rec. 709 (used in Blurays and elsewhere) as well as Rec. 601 explicitly use limited range YUV. Players that don't use limited range correctly would be out of spec and couldn't play a single Bluray disc correctly. So for the overwhelming majority of use cases, you want regular old yuv420p.
The author is picking up on something that actually happens, however. There are Blurays out there that have incorrectly converted YUV values. This happens when for example something is in limited range YUV, and then that data is passed to a converter that treats the data as full range YUV, and compresses it again to fit in limited range YUV. This "doubly compressed" data is embarrassingly common (maybe 5% or so of Blurays have this problem). If you get a Bluray like this, or any video with the same issue, the blacks will be bright grays and the whites will be dull (similar to how the author describes limited range YUV). The player on the end is reading the YUV values correctly, but the data has been doubly compressed so it only undoes one round of this.
The issue is well known in the pirate communities as an example. Reliable release groups always fix these broken Blurays in their encodes. This is usually described as "fixing levels" or something like that.
I have one question about terminology though: when you refer to “players“, do you mean all kinds of players or just physical hardware devices like Blu-ray players?
I ask because I haven’t had/observed any issues or defects with colour in software/PC-based desktop serious at all.
Disclaimer: not in the Blu-ray-authoring business.
ffmpeg -i input.png -pix_fmt yuv420p movie.mp4
Open in VLC or your favourite player and take a screenshot. Now check with a color picker whether you have white as 255 or 235.
Update: checked with VLC and QuickTime Player, both handled it correctly and displayed white as 255.
Carmack is in the wrong here.
Sure, Google has a horrible track record when it comes to service durability, but I wouldn't trust Facebook as a reliable host either. Anyone having relied on their @facebook.com email address (2010-2016) may agree.
"Years ago, I felt burned when I wrote several articles for #AltDevBlogADay, and they vanished. I have much more confidence that what I write on FB won't vanish. [...]"
Adventures with ffmpeg and color ranges.
A video legacy issue that can cause a lot of problems is the issue of “limited” versus “full” component range. For arcane reasons related to TV broadcast limitations, many video formats restrict the YUV color components to be in the 16..235 or 16..240 range instead of the full 0..255 range. Losing 14% of the already-barely-enough 8 bit dynamic range is bad enough, but it also often results in the black range starting at a quite visible grey value because most players don’t rescale the range. This is usually visible as ugly banding or blocking in dark scenes.
For ffmpeg, the trick to avoid this is to use ‘-pix_fmt yuvj420p’, which says to use the j-for-jpeg full range in a 420 YUV subsampled p-for-planar format.
If you are starting with either RGB images, a 10/12 bit format, or a yuvj420p format video as input, then with the libx264 codec, you would get a full range output. Note that any video processing tools used along the way could also limit the range, and once it is gone, there is no getting it back, so you must be very careful and check your entire pipeline!
When you us this format, ffmpeg complains about ‘deprecated pixel format used, make sure you did set range correctly’, but you should ignore this warning.
Ffmpeg would like the world to move to specifying the range independently from the YUV channel subsampling and layout:
Setting ‘-color_range 0 -pix_fmt yuv420p’ makes the output format yuv420p Setting ‘-color_range 1 -pix_fmt yuv420p’ makes the output format yuv420p(tv) Setting ‘-color_range 2 -pix_fmt yuv420p’ makes the output format yuv420p(pc)
Unfortunately, this isn’t yet uniformly handled throughout all the internal format tests.
The libx265 integration in ffmpeg didn’t support the deprecated yuvj420p pixel format, only the basic yuv420p one, and no matter what I did, my test videos were always coming out limited range. It didn’t matter if you add a ’-color_range 2’, or ‘-x265-params range=full’. Those will change the settings in the VUI (Video Usability Information) section of the output, but the values are still compressed to the limited range.
Regardless of the input data, any 8 bit h265 video coming out of ffmpeg was limited range!
I walked through the libx265 code looking for range compression, but it turned out that all the damage was being done by ffmpeg before it got to x265. Ffmpeg will automatically convert formats when the output differs from the input, and since x265 only supported yuv420p, any full range input will be processed.
Adding ‘-v 48’ to ffmpeg will dump more information, including this auto_scaler invocation, which is what is killing the full color range: [auto_scaler_0 @ 000001f55ecd1a40] w:2048 h:2048 fmt:bgr24 sar:0/1 -> w:2048 h:2048 fmt:yuv420p sar:0/1 flags:0x4
I was about to start hacking the code to at least do what I wanted for my use case, but I tried an appeal to Twitter:
https://twitter.com/ID_AA_Carmack/status/1131715388067274753
My suspicions were confirmed, but Derek Buitenhuis went ahead and submitted an official patch to get yuvj420p accepted by libx265, and windows builds are already available at https://ffmpeg.zeranoe.com/builds/.
I suspect there was probably some way of working around this involving explicit format conversion filters with -src_range and -dst_range overrides, but this is now working as you would expect it:
ffmpeg -i source.mp4 -c:v libx265 -pix_fmt yuvj420p dest.mp4
Bravo to ffmpeg!
As far as I can tell in XYZ black is (0,0,0) and the only sRGB value that maps to that is (0,0,0)
If sRGB (1,1,1) is XYZ (1,1,1), then sRGB (0,0,0) is XYZ (0.0025,0.0025,0.0025).
Or if you are using absolute XYZ values, sRGB (0,0,0) is XYZ (0.1901,0.2,0.2178).
Ignoring it will give you nonsensical values when you try to do conversions into other color spaces or when you try to do color comparisons.
Otherwise, why would he move away from .plan?
Facebook may be a big conglomerate these days, and they may not be the cool kid on the block anymore, but that doesn't mean that they're idiots. If anything, they treasure the presence of someone like Carmack all the more, because it's no longer reputionally-free for people like him to come on board. I'm sure that Facebook is appreciative when he blogs on their platform, but it's patently ridiculous to pretend that anyone would -- or could -- force it.
In fact, Facebook has already been forced to pay at least dozens of millions of dollars essentially as a direct consequence of employing Carmack [0], and I'd be willing to bet they'd happily pay (at least) dozens of millions more.
If you want your mind blown even harder, look up some of the paperwork around the Waymo / Uber debacle [1]. SV companies pay big bucks for this type of talent, even the ones you haven't heard of, like Anthony Levandowski.
If the roll of the dice is in your favor, SV stardom is at least as valuable as Hollywood stardom (and, for the most part anyway, you still get to walk around in public unharrassed).
Disclaimer: I have no special connection with any of the companies or people mentioned, I don't know the inside baseball, and this is all conjecture. But at least it's better supported than yours. ;)
[0] https://en.wikipedia.org/wiki/ZeniMax_v._Oculus
[1] https://www.cnbc.com/2017/04/03/waymos-uber-lawsuit-reveals-...
"This fist holds the Ego. This fist holds the Id. I bring you a new existence: the GlobalCoin. I came up with that myself. As Super-Ego."
[not the person you responded to]
In exchange for their patronage, employers to the world's technical stars get to use their name in recruitment and other types of marketing, and they're well-positioned to profit from the star's next great achievement, if indeed one ever occurs. This is very similar to the benefit to a record label or a movie studio in signing a deal with the current one-hit wonder or starlet of the hour.
As in any employment transaction, there can be no true guarantees. None of us can promise that we'll be here tomorrow. But the value of investments in strong personal brands has proven profitable enough over the last several decades that many companies are eager and willing to make them. Entire industries are built around them. I don't think it's going to stop any time soon. cf. the Lindy Effect [0]
There is surely some limit to Facebook's tolerance of a single man's foibles, regardless of star power. I would bet my bottom dollar that "blogged off-Facebook" is nowhere near it.
No one is nearly as interested in Carmack's thoughts on codecs as they are about the domain name of the server he's posting to.
And yet, we'd all read his stuff no matter where he puts it. But maybe he reaches more outsiders this way? I doubt it.