John Carmack on JPEG

You really don't want compression artifacts in any image, which is why you should set the compression ratio wisely when encoding. I don't see how X-rays are any different.

There's one big win though. Digital cinema projectors, in now pretty much all theatres, run DCPs which have movies encoded with it.

JPEG2000 is such a cool choice for Second Life. The fact any truncation of a JPEG2000 bitstream is just a lower-resolution version of the image makes for convenient progressive enhancement when loading textures over the network, right?

(And Second Life even used JPEG2000 for geometry, sort-of. I guess with the advent of proper mesh support, that may be less common now though.)

I work in the library/archiving space where people spent years trying to make this happen due to the compression wins, support for a good range of color depths and spaces, and the progressive decoding is perfect for browsing galleries of high-res images before zooming way into the one you wanted.

The frictional cost largely canceled that out: people don’t trust unreliable formats and the JP2 files which only opened in one of {Kakadu, Aware, Adobe} and couldn’t be used with any open source tools live long in the memory. Performance in Jasper was wretched … and while the vendors thought that’d boost sales, it seemed far more effective to me at getting people to use other formats instead.

These days, if anyone is talking about new image formats the first question is what they’re doing for open source (especially things like ImageMagick which half the world uses) and specifically browsers. A WASM polyfill for <picture> or <img srcset> is really critical because it means people don’t have to transcode everything for a handful of users.

Original image: https://www.fnordware.com/j2k/jp2samples.html

What it looks like in GIMP: https://imgur.com/a/HCNz7ga

What possesed someone to actually attempt to use JPEG2000 I do not understand.

JPEG has introduced multiple new formats since JPEG 2000, all of which are better, such as JPEG XR and JPEG XL.

Apple has decided it shouldn't be by not allowing its use on iOS.

Namely, VK_KHR_sampler_ycbcr_conversion. I'm actually not sure if other APIs provide the equivalent yet, the HW capability is pretty recent...

while not standard, writing a shader to decode is not hard(admittedly if you know shaders)

The time I spent dealing with video decode/encode was not pleasurable but interesting.

The really hard part is the actual encoding/decoding algorhtims

Carmack is giving an extremely simplified version of the world as it exists today. Your hardware can already use YUV formats, maybe it is even outputting YUV because you chose a resolution and bit depth that forced it to for bandwidth or clock constraints. Complex apps like browsers that do their own compositing can already choose to use YCbCr formats when supported for textures.

You are correct, and you can find details on that by reading the card documentation (as far as it is available). That was in particular the case when cards had a fixed-function rendering pipeline.

With the advent of more general purpose cores on the GPU for the rendering pipeline, YUV decoding has been offloaded to them.

So since you already have to process the image, it doesn't seem like a big ask to convert from JPEG-flavored YUV to GPU-flavored YUV. But I'm not an expert, so maybe this is hard/lossy?

It's been done. There are even GPUs that support this operation natively, so there's no additional overhead.

It's a bit tricky because rendering pipelines composite the final image through many layers of offscreen compositing before the pixel hits the screen.

The core issue is that the offscreen composited layers would still be 32bit textures which is a bigger issue. I would imagine a Skia-based draw list to encode this through the pipeline which could help preserve this perhaps.

Your display uses 3 bytes per pixel. 8 bits for each of the R, G, and B channels. This is known as RGB888. (Ignoring the A or alpha transparency channel for now).

YUV420 uses chroma subsampling, which means the color information is stored at a lower resolution than the brightness information. Groups of 4 pixels will have the same color, but each pixel can have a different brightness. Our eyes are more sensitive to brightness changes than color changes, so this is usually unnoticeable.

This is very advantageous for compression because YUV420 requires 6 bytes per 4 pixels, or 1.5 bytes per pixel, because groups of pixels share a single color value. That's half as many bytes as RGB888.

When you decompress a JPEG, you first get a YUV420 output. Converting from YUV420 to RGB888 doesn't add any information, but it doubles the number of bits used to represent the image because it stores the color value for every individual pixel instead of groups of pixels. This is easier to manipulate in software, but it takes twice as much memory to store and twice as much bandwidth to move around relative to YUV420.

The idea is that if your application can work with YUV420 through the render pipeline and then let a GPU shader do the final conversion to RGB888 within the GPU, you cut your memory and bandwidth requirements in half at the expense of additional code complexity.

Wikipedia is a good source of diagrams and details that explain this further: https://en.wikipedia.org/wiki/YUV#Y%E2%80%B2UV420p_(and_Y%E2...

There are alternatives. Google optionally uses webp, because it saves them bandwidth and they control the entire chain. And specialized applications like maps sometimes use better suited formats, but if you want to share a picture, that's JPEG.

The same thing happens with audio. The go to format is still MP3 even though it is well outdated. In fact, we had what is close to the perfect lossy audio codec (Opus) since 2012 and support it is pretty much only used when you control the entire chain, like in video games.

So maybe JPEG XL is the perfect image format, but it is still a tough sell when you have something universal that is good enough against it.

The only way I can see it gain traction is if tech giants get together and force it on us. Like they have been doing for web standards. And if there are patents, even that may not work.

Video is different because it uses a huge amount of bandwidth and no formats are "good enough" yet.

You can set a time range in Google; "Tools" link under the search bar. DuckDuckGo has this as well. Super useful, especially if there's some recent/prominent news and you want to find things other than this news, or if you're looking specifically for older stuff.

I think you can use "word" (with quotes) to prevent the synonym thing; not entirely sure about that.

and it was in IE11 even!

I'm not sure I understand where Carmack is coming from here though (am I missing some context? I don't use twitter and these threads are always a huge pain for me to follow especially since Carmack doesn't even bother breaking on full sentences). I don't get how processing in YUV instead of RGB has anything to do with 10bit components for instance.

Also, in my experience most video software deals with YUV natively and only converts as needed. It's probably different in the gaming and image processing world but that's because everything else is RGB and it seems to be a big ask to just tell everybody to convert to YUV.

Besides if quality is of the essence, you will typically store more that 10 bits for internal processing, probably 16 and maybe even floats if you want to have as much range as possible.

I dunno, I won't pretend that I'm smarter than Carmack, but I wish there was a bit more context because it's a bit opaque for me at the moment.

I know several imaging apps have the ability to select which subsampling type to use as well.

Seriously though chroma subsampling is not kind on any kind of red shape, it's especially bad against white backgrounds.

Most of them. Pretty much the default subsampling for libjpeg(-turbo).

Yes, when they can pay the computational price for compression beforehand. Not going to work, if you had to compress in realtime.

However, I've heard that Netflix was one of the first sponsors of https://github.com/BinomialLLC/basis_universal They are shipping innumerable images to a huge variety of underpowered set top boxes. Totally worth investing in a giant-leap memory optimization.

Everyone is currently approaching AGI in what you might call a traditional way. Carmack's way is completely different, which was refreshing.

I think Carmack's technique has the highest chance of reaching AGI. The tweet chain isn't as unrelated as it seems.

(b) Can you use curl? Maybe hit the API.

(c) The modern web DOES require JavaScript. Sure, I'm not clear on what that has to do with reading three tweets available on a public API, but it's pretty weird to object to a declarative statement of truth.

4:1:1 is half of RGB but I've never seen anyone ever use it. Pretty much everything in use today (at the consumer level) uses 4:2:0. It's good enough for today's standards (AV1 Main profile only supports 4:0:0 and 4:2:0).

Essentially, rendering JPEGs directly as a GPU texture (since GPUs can already natively use 420 encodings for textures). His point is that this would improve performance, very slightly improve quality, and reduce power usage as there's no colour space conversion.