Using Saliency in progressive JPEG XL images | Hacker News Reader

Using Saliency in progressive JPEG XL images

opensource.googleblog.com

89 pointssigna1144 comments

I was honestly skeptical of another format beyond AVIF for images. I thought the next generation coded would be based on neural net encoding/decoding for lossy images (as we should be able to get "easy" 2x wins). However, I think progressive loading in JPEG XL is an amazing feature, and a significant update, and is perhaps more important that raw compression ability.

My understanding is also that you can also use the progressive nature to have 1 "high res" version, and potentially render different sizes for different connections, and do some interesting things. For example, I believe you could (in principal) do something like stop loading after 1s so long as a minimum size level was achieved) - to make things fast on all devices, but only the quality would vary. So if you are on a 4k monitor and gigabit download? You can get massive images. On a 3g phone? Get the basics.

It's a really great feature, and does seem like it would likely have the most real world impact on actual website perceived performance.

Another strong feature is the great upgrade path from existing jpeg files, which can be losslessly recompressed into jpeg xl with an additional ~20% compression.

At this point I don't really see a reason for AVIF, hardware accelerated decoding could be a benefit, but AFAIK there's no device supporting it.

AVIF should be completely cancelled in favor of JXL. A native designed-for-images codec is so much better than "use video codec for one frame" in countless ways. Indeed progressive being probably the biggest one.

Oh, and Apple might be less hostile to it, because it's not based on an "anti-MPEG" video codec.

I think AVIF is a great replacement for GIF (the acronyms even rhyme and all!). For short-video-without-audio, it's as good as it gets.

For still images though, I don't really see any major advantage of AVIF over JXL except AVIF being a bit older and already enabled by default in Chrome. I do see several advantages of JXL over AVIF though.

So the way I see it, the future is:

AVIF instead of GIF (and APNG and animated WebP)

JPEG XL instead of JPEG (and PNG and still WebP)

Disclaimer: I am probably biased since I am a jxl dev. Please do your own independent evaluation (and feel free to share your opinions/conclusions with me).

<video autoplay> instead of GIF though.

It's very surprising to me that progressive loading techniques have been around for decades, but have barely been used on the web. Images often make for most of a websites content, size wise, and are frequently far larger than they need to be. Even those with generally good internet experience waiting for images to load for a noticeable amount of time. A 95% compressed version of an image is almost always good enough for a first glance and then the rest of the image can be seamlessly, progressively loaded. It's apparently been a ubiquitous problem for decades, with a known solution, yet it has not been applied. I'm curious as to why.

Decoding a progressive JPEG when you have the whole stream is more CPU intensive and the decoder itself is more complex.

Decoding JPEG images is not very CPU-intensive at all. Back in 1995, an ARM chip was fast enough to render JPEGs on the fly - i.e. apps didn't need to buffer the decoded versions, it was fast enough to draw them straight from the decoder when redrawing windows. (This was on RISC OS 3.6, IIRC)

It is more CPU intensive than a baseline JPEG.

rule of the thumb: progressive JPEG costs ~3x more resources (CPU, memory) than baseline JPEG.

With baseline jpeg, you can blit the decoded blocks to screen directly and forget about it.

For progressive, you have to buffer the whole image (in DCT format!) and do the idct / color-space conversion on all passes.

But, provided you don't need the intermediate results, you can rearrange the data back to progressive and then render it the simple way, for those times when CPU power is constrained more than the network. Total CPU use ends up being pretty much the same (the data rearrange step is rather cheap compared to the idct's)

if you re-arrange you have to wait for the bitstream to finish downloading, and thus lose the 'display something early on' feature.

No... After all but the final pass of the image has been delivered you have enough data to start doing progressive rendering. The final pass contains most of the bytes, so you still get a pretty decent progressive render too.

To minimize additional computation JPEGs often use a limited number of scans, for example 5.

By default JPEG XL's progressive uses only two scans, 8x8 DC, and the transforms in the second pass using a 256x256 tiles in encode-time chosen priority order. This choise allows JPEG XL to do only one round of DCTs even in the progressive case. The 8x8 DC is interpolated using cheaper methods.

Because of the design choices, every JPEG XL image is guaranteed to be at least minimally progressive in the same manner, i.e., 8x8 DC first. Having a guarantee will make it more rewarding for system designers to focus on extracting some user-experience benefit from that feature.

For the last 10 years progressive JPEG has been the fastest growing image format in the internet. Today, about 20-25 % of JPEGs in the internet are progressive. Mozjpeg outputs progressive JPEGs by default. JPEG decoding times are usually in the ballpark of 10 milliseconds, often considered an insignificant performance factor in the system analysis.

With battery-powered devices being so common, "fast enough" is no longer the only criterion for performance. Every cycle that the CPU doesn't spend idle subtracts from battery life.

Anything you could do live 26 years ago can be done today with an infinitesimal fraction of a modern mobile chip.

>Every cycle that the CPU doesn't spend idle

Such as time spent networking?

Network latency will dwarf CPU time in that case; the CPU can idle while waiting for responses.

On the other hand, it is ~infinitely less CPU intensive than decoding HEIF and AVIF with no hardware acceleration, and they were supposed to be the future of the web.

AVIF is equivalent in decoding complexity, even in software. It's not order of magnitude different. Encoders OTOH can be as slow as you want them to be.

Lazyness, indifference, time/budget constraints, and software that didn't bother to optimize for this. In other words: Hanlon's razor (https://en.wikipedia.org/wiki/Hanlon%27s_razor), more or less.

Progressive JPEG would also be cool if bandwidth costs matter, nowadays a lot of images on the web have more pixels than the screen is able to display, so they are downloaded and downscaled. Or you have the HTML "imgsrc" attribute for "retina" images.

Imagine a file format that says "Byte-ranges 0-n will give you the image for a resolution of 640x480*. Bytes n+1 to m will give you the rest of the pixels for a resolution of 1440x1080, bytes m+1 to the end will give you the pixels you need to add to the 1080p version to get a 2160p version".

So the browser would decide what it needs and just requests the file up to that byte...

* The low-res 640x480 version might be all the browser needs if the person who designed the web page just wants the image e.g. as a small headline image in a news site column that is 640px wide

Basically that's what you can do with a progressive jxl: the frame header includes a table of contents that can be used to infer the byte ranges needed for a progressive preview corresponding to 1:8, 1:4, 1:2, and 1:1 resolutions.

Yup. Interlaced GIF (for progressive rendering) was the norm during dialup BBS stuff. As far as I know, web browsers always supported it. Dunno why the custom didn't make the jump from BBS to web.

20-25 % of JPEGs in the internet are progressive. MozJPEG outputs progressive JPEGs by default.

Safari chooses not to show intermediate results, but they are available with Chrome and Firefox.

The impact of progression on user experience is a complex question. It is for minimizing user experienced latency vs. (over)activating preattentive processing by images changing on the screen. Last year, Moritz Firsching (also the author of the blog post in question) made several improvements to the progression of traditional JPEG (libjpeg-turbo, Chrome, Firefox). Those changes -- in my opinion -- dramatically reduce the flicker and poor renderings that could occur from traditional JPEG progression, making progressive JPEGs more attractive overall.

https://jpegxl.info/

https://news.ycombinator.com/item?id=27577328

Chrome status: behind a flag `enable-jxl`

FireFox status: only in Nightly and behind a flag `image.jxl.enabled`

Edge status: behind a runtime flag `--enable-features=JXL`

Safari status?

Safari is the new IE6; you can basically not expect anything to happen in a long time.

The webkit core of safari is opensource right? Do they accept outside contributions for stuff like this?

See https://lists.webkit.org/pipermail/webkit-dev/2021-May/03184... and https://lists.webkit.org/pipermail/webkit-dev/2021-May/03185...

> image format support depends on OS library support, rather than anything in WebKit. [...] to be clear, Apple’s OSes do not currently support JPEG XL.

I even got `jxl` to work in Electron!

https://github.com/whyboris/Video-Hub-App/compare/jxl

Reminds me of the mechanism twitter used to crop images based on gaze prediction [1]. If their model is anything to go by we know which parts of a bikini picture we will see in high-res first [2]

[1] https://blog.twitter.com/engineering/en_us/topics/infrastruc...

[2] https://www.reportr.world/news/twitter-drops-image-cropping-...

Can I use JPEG XL Yet [1] ?

Again Image / Video / Audio Codec Support on Safari requires OS update. Unless this is on their agenda for iOS 16 and next macOS, which I doubt, it will likely be a few years before support of it becomes widely available.

[1] https://caniuse.com/jpegxl

I'm surprised that there seem to be no polyfills.

Polyfills for image and video formats are pretty much never going to be worth it: they’ll be a fairly hefty chunk of code (so you’ll need a lot of shrunk-by-using-the-polyfilled-format images to balance it out), and they’ll load much more slowly than the native (so that lot of images is now a liability), while not being able to smooth loading the way stuff native to the browser does. I don’t think a compelling reason to do it that way has ever or is ever likely to exist; it’s an all-round bad idea in practice. Just fall back to a different format, that’s what <picture> is designed for.

I'm not sure how big a jxl decoder has to be, but there's a wasm webp decoder that zips to about 50KB. That kind of cost pays for itself very quickly.

And whether it loads faster or slower depends on your connection. But yes it hurts to not get the smoothness benefits.

Hmm… https://webmproject.github.io/libwebp-demo/webp_wasm/index.h... ships 141KB/54KB (bytes/transferred) of WASM and 116KB/34KB of JavaScript, including lots of bogus SDL stuff. https://kenchris.github.io/wasm-webp/ is 120KB/44KB of WASM and 30KB/9KB of JavaScript, much more reasonable. (Incidentally, I never like the JavaScript such tools produce; it’s always hideously unoptimised. Experience suggests to me that I could reduce that 30KB to under 10KB with a little effort, and possibly to under 5KB; and if I could touch the WASM as well, still further.)

Definitely smaller than I thought I recalled, and WASM bytes are no dearer than image bytes, though JS bytes are much more expensive, so it’s closer to just the more expensive decoding of the actual images that’ll bite you. Plus the not-integrated loading experience, of course.

It doesn't matter too much (for the web) as you can use <picture> and <source> and specify multiple jxl/avif/webp/jpeg sources, and the browser will choose the first type it understands. These elements are widely supported.

Hopefully JXL doesn't become yet another image format (927)... some features of it are incredible. For example, you can have mixed lossless/lossy content in one image. No longer will you need to choose between a lossy or lossless format: the encoder would intelligently segment the image.

> you can have mixed lossless/lossy content in one image. No longer will you need to choose between a lossy or lossless format: the encoder would intelligently segment the image.

What's the advantage of that? If I'm dissatisfied with the compression quality in some areas, I can tweak the quality parameter already.

See https://github.com/libjxl/libjxl/pull/466 for more details.

As far as I understand it, the synthetic mode compresses better for (as you can guess) synthetic content. So if you're encoding a mixed content screenshot, you'll get better quality and size by encoding the photos and UI separately.

Oh I see, that makes sense.

Nice discussion of progressive loading and other features in JPEG XL (albeit not relying on saliency) https://www.slideshare.net/cloudinarymarketing/imagecon-2019...

Previous HN discussion https://news.ycombinator.com/item?id=21612708

I'd be worried about a user experience where the image looked good enough that a user thought it was done downloading but then didn't realize it was getting better or still downloading and moved on. A nice indicator would be helpful.

Reply on news.ycombinator.com