JPEG Committee releases a call for evidence for image compression based on AI
jpeg.org
jpeg.org
The JPEG XL Image Coding System (ISO/IEC 18181) has produced an open source reference implementation available on the JPEG XL Gitlab repository. The software is available under Apache 2, which includes a royalty-free patent grant. Speed tests indicate the multithreaded encoder and decoder outperforms libjpeg-turbo. Independent subjective and objective evaluation experiments have indicated competitive performance with commonly used image coding solutions while offering new functionalities such as lossless transcoding from legacy JPEG format to JPEG XL. The standardization process has reached the Draft International Standard stage.
That to me is the biggest news. I remember reading a presentation where it had even better quality at medium bitrate than AVIF. Since JPEG does not belong to Google or Apple, hopefully it would be a format easier to swallow from both parties.
A bigger problem is that various companies just did a lot of work to adopt HEIF so they're not going to be eager to churn onto JPEG XL. Generally people wait 5-10 years before they're open to upgrading.
Yes but it is submitted as an open standard to an Organisation that doesn't run or belong to Google.
Is HEIF used anywhere other than on Apple devices?
- Patent situation. If you're a camera or phone maker, it is quite a risk to use HEIF. Even if you pay all the major patent pools, you only cover part of the patent claims. You'll need dedicated hardware, so once you've produced the devices, the patent trolls have quite some power since you can't just switch to another codec with a software update.
- The codec is not optimized for high-quality still images – it's a video codec, and has its sweet spot in the lower bitrate regions. Photographers care about subtle texture, high bit depth, etc.
- There's no progressive decoding – again, it's a video codec, where progressive decode of a single frame is not useful, but for still images on the web, progressive can be a desirable option.
FWIW, AVIF is a project of the Alliance for Open Media, which all the major browser vendors are founding members.[0] AVIF should get browser support soon.
Taken to the extreme, a sufficiently advanced AI compression system could theoretically end up storing representations of high-level features like faces in the compressed data; which could lead to some really bizarre looking compression artifacts when the system breaks down.
Or, you could have a compression system created specifically for photographic images (as opposed to a “general compressor” like LZ77), where these high-level features are part of the (de)compressor itself.
Now, lossy domain-specific compressors are pretty common; one good example is audio “voice codecs” for telecom, in which every snippet of audio is encoded as a phoneme-like symbol, such that noise comes out the other end as a gobbledygook that sounds a lot like words.
But much more interesting to me is a lossless domain-specific compressor, which I’m not aware of any examples of. (Please tell me if about some if you know them!) In such a compressor, the input would be “rebased” against the model, such that the output looks like an instruction stream ala “start with [this synthesized template with these parameters]; and then add point-modifications {A, B, C, ...} to get from there back to the original input.” So inputs that were more like something it “understood” would compress well (i.e. just be described by a template) while inputs that were less-well-understood wouldn’t get much from the procedural-generation phase, and would end up being specified entirely as point-corrections, and so not really compress at all.
https://papers.nips.cc/paper/9383-integer-discrete-flows-and...
https://xiph.org/flac/format.html#interchannel https://xiph.org/flac/format.html#prediction
The model should make the errors small, and including the entropy measure of the error in the loss function means that the model produces "cheaper to correct" errors.
A nice keyword on this front is "bits back argument".
Specifically, see https://arxiv.org/abs/1609.07959
A bit academic, but if you check out this paper from 2015 https://arxiv.org/pdf/1502.04623.pdf on a neural network trained on binarised MNIST (28x28 pixel images), Table 2 quotes entropy figures of 116.815 after converting to bits, which would be 28*28/116.815=6.71x compression (if used as a compressor)
The success of a compression scheme ultimately comes down to its ability to estimate p(x), for some sample x. And estimating p(x) is precisely the problem the field of ML addresses.
I think that's the major risk for straightforward "AI" compression. You'll end up with some high level description of the scene and the decompressed image will then be an "artist's impression" which contains details that may not have been in the original image. You'll take a photo of a cheque, compress it, decompress it, and you'll get a picture of a cheque, but it might not be the same cheque.
https://www.zdnet.com/article/xerox-scanners-alter-numbers-i...
--Edit-- I think there are differences between the case of using this for compression and how Xerox implemented (which they claim was due to compression).
It can be argued that most advanced codecs of today already handle compression via very complex and advanced heuristics already comparable to what a lot of people would confuse for AI and/or ML today, so I think it's not as spooky as it may sound at first.
The Xerox implementation actually changed values on the page from one to another innocuously. I would hope that the AI being talked about is not "This 6 looks more like it should be an 8, so I'm going to put one there upon decode" and more like, "this block or unit of the image has these characteristics, and theses methods work very well to reproduce these characteristics when next to these other blocks..." etc.
But that's the best way I can phrase it.
Movies could accidentally "transpose characters" too - meaning Mark Walhberg instead of Matt Damon
You've gotta be kidding me.
- Call for Evidence launched for AI-based image codecs
- Call for Evidence launched for point cloud codecs
- Next-gen image codec JPEG XL reaches Draft International Standard stage
- Light-weight JPEG XS codec is looking at compression of camera raws
First: Use AI to generate the text description of an image. http://homepages.inf.ed.ac.uk/keller/publications/jair16.pdf https://www.captionbot.ai/
Second: Use AI to generate an image from the text description. https://news.developer.nvidia.com/ai-model-can-generate-imag...
https://twitter.com/cassmarketos/status/1229473344480673792?...
Of course, an actual working compression algorithm is quite likely. I recall that recently someone was upscaling and colorizing ancient movie reels to modern standards, and there is plenty of low-hanging fruit in fine-tuning parameters to suit specific images, so I actually look forward to this.
I just hope they don’t name it JPEG2020, JPEG2000 was a bit of a dud.
There are lots of scanned ebooks on archive.org that use J2K compression, and the sluggishness when turning pages is extremely noticeable compared to others which use normal JPEG.
Then I realised what must've happened. There were tons of open air carts on the street selling eggplant. My guess is that the NN was either shown that or recognized that part of the image and colored almost everything else it saw using that as its base point. Not the best technique.
At a high enough level, you are using a model of some sort to estimate (or guesstimate) changes in the image but the models for "X type of noise from the sensor" and "artifact due to block size choice" or "artifact due to aliasing in the RF domain" or whatever are quite different.
An AI that “fixes” JPEG artifacts would just be a regular upscaling/supersampling AI—just one trained on JPEG sample-matrices instead of BMP sample-matrices, so that it can take advantage of the extra information each JPEG sample-point carries.
You might have been exposed to the singular-value decomposition to compress a matrix, and representing an image as a matrix e.g. [1].
JPEG encoding is about transforming the signal into a frequency domain (think FFTs) where it is less detrimental to throw away information.
If there is a connection to what you're saying, I expect you should try to make some statement about the eigenvalues of the DCT matrix [2]
[1] http://www.math.utah.edu/~goller/F15_M2270/BradyMathews_SVDI...
[2] https://math.stackexchange.com/questions/962533/why-does-the...
I don't know enough about JPEG or about AI to know whether that would be doable.
Reminds me of a "lightweight" zip compression library that I recently used that was literally 98% overhead and 2% actual compression code. It had its own definitions of everything: integers, integer properties, memory allocation, streams (as in sequences of memory blocks, not even real I/O!), error handling, progress notifications, etc... Pages and pages of conditional macros and compiler-specific hacks. Just to call a single a function that took an array and returned an array. Ridiculous.
https://gitlab.com/wg1/jpeg-xl/-/blob/master/include/jpegxl/...
https://gitlab.com/wg1/jpeg-xl/-/blob/master/include/jpegxl/...
https://gitlab.com/wg1/jpeg-xl/-/blob/master/jxl/base/compil...
https://gitlab.com/wg1/jpeg-xl/-/blob/master/jxl/base/os_spe...
https://gitlab.com/wg1/jpeg-xl/-/blob/master/jxl/base/arch_s...
That last one is trying to work out the NUMA topology in a portable way... good luck with that! That's not something I expected from a JPEG library. Like memory management, they'd actually be better off requesting thread-management function pointers from the hosting application. Otherwise, this would not be a "good citizen" in constrained environments such as embedded in a web server or database engine.
Random examples of silliness from just this one file:
They ignore the first 2 cores on 3+ core systems: https://gitlab.com/wg1/jpeg-xl/-/blob/master/jxl/base/os_spe...
They'll be limited max 64 threads (Half the threads on the larger single-socket AMD EPYC 2 CPUs!): https://gitlab.com/wg1/jpeg-xl/-/blob/master/jxl/base/os_spe... because they're not using the Processor Group Aware API: https://docs.microsoft.com/en-us/windows/win32/procthread/pr...
Sure, 64-core monsters are still somewhat rare today, but 16-core/32-thread desktops CPUs are now affordable and the TSMC 5nm process will likely result in 96-128 core monstrosities this year. That's 384-512 threads for a dual-socket server or workstation! If you're making an image codec library "for the next decade", a little future-proofing might go a long way. Look at it this way: Image decoding throughput is one of the main performance issues I have right now with Lightroom and I work with "only" 36 Mpx images. There are 80 Mpx cameras just around the corner...
For example, there was a blog article a couple of years ago basically saying that the major Amdahl's law limitation to decoding formats like JPEG is the Huffman bitstream decoding, which is Step #1 in the process. The author pointed out that simply restarting the Huffman bitstream a few times for a typical PNG or JPEG file would increase the size less than 1%, but would allow truly parallel decoding for every step.
For all the apparent effort to make JPEG-XL parallelised by default, it seems odd that there doesn't appear to be any mention of parallelising the ANS bitstream decoding...
FYI we actually do support hooks for malloc (JpegxlMemoryManagerStruct) and threading (JpegxlParallelRunner) precisely in order to be a good citizen :)
The topology is more about #cores/sockets, used by the benchmark_xl app to decide how many threads to spawn.
I agree adding Processor Groups makes sense, but Windows hasn't been our focus yet.
FYI it is indeed critical to parallelize ANS streams. Each 256x256 "group" can be decoded in parallel.
Just to clarify: I'm not criticising the JPEG XL code, it seems to be quite clean at first glance! I was merely making a general observation that would apply to practically all portable C-interface libraries, such as ZStandard, libpng, etc...
PS: I wonder if it would be possible to make a "portable library header" that all such similar libraries could share to avoid the rework on both ends (library authors and library consumers)...
> Windows hasn't been our focus yet.
That's a real worry. Not just because of the word "Windows", but what else it implies.
To provide some constructive feedback: JPEG XL will be yet another failure thrown on the trash heap of similarly well-intentioned formats. It'll die along with JPEG 2000, JPEG EXR, and all of the others. Don't be so arrogant to think that somehow your format will be special and adopted by the masses. The others were just as technically advanced for their time and they all failed.
Please, please, please learn this lesson, because this history of failure saddens me. The lack of a capable image formats compatible with all client platforms is what stops me as a photographer from sharing my images in full quality. In 2020, still, it is effectively impossible to share images in any other format than 8-bit sRGB JPEG across platforms. You can't email a HDR, 10-bit, or wide gamut picture to anyone and expect them to be able to open it and have it look even vaguely right. You can't post such an image on any website you don't control yourself. Even if you control the website, it will look wrong for 99.9% of users OR they'll just see the 8-bit sRGB version anyway, even if they have 10-bit HDR display.
It is literally only the walled garden of the Apple ecosystem that gets this right. Microsoft and Google most certainly doesn't. As an iPhone user, I can share a wide-gamut HEIF picture and 100% of the time it will look correct for other iPhone users. However, Apple "cheated" a little bit. When you send one of their HEIFs to any non-Apple-controlled app, it's silently converted to... 8-bit sRGB JPEG. It's just sad.
What everyone gets wrong with new image formats is this: They think that the problem that needs solving is the encoder/decoder. That's step one of many problems that users need solved. Image format developers assume their format will be "adopted", but there's no network effect. Why would anyone use a format they can't email, can't open on Windows 7, and can't edit in Photoshop?
If you stop with the JPEG XL library, with Windows compilation an afterthought, I absolutely guarantee that you will fail to gain traction and the entire effort will be essentially wasted.
To get actual adoption, you would need to:
* Most importantly, develop an image decoder plugin for Windows.
* Help Firefox, Safari, and other less common browsers merge support.
* Ensure that Chromium gets support, but I assume that's a given.
* Send pull requests to add JPEG XL support for all major open source libraries that perform image processing, such as WordPress plugins and server-side image processing utilities for websites.
* Develop a plugin for all major image editing tools such as Photoshop.
* Work with Adobe on Lightroom export support.
* Contact Sony, Nikon, and Canon and help them with in-body encoding support.
But even that's not enough. If you want 10-bit and the wide gamut (let alone HDR), you have to take steps to force support, or it won't magically materialise. You'll end up with two JPEG XL formats: The 8-bit SDR sRGB variety with 99.9999% adoption and the "fancy" JPEG XL that nobody uses and looks wrong if they try.
IMHO the only way to do this is to forcibly break applications that aren't processing the ICC profiles properly. Much like Google's Chrome experiment with randomising some TLS header fields to occasionally break middle boxes written by idiots that think version numbers are checked with equality, not inequality operators.
You'd have to do something drastic, like leaving the image planes unnamed and randomised so that literally the only way to decode the image correctly is via a compliant colour space transformation.
It sounds drastic, but other people have tried the easy way and failed. As I said: It's 2020!!! Email me a non-sRGB image that I can view in my mail client...
> That's a real worry. Not just because of the word "Windows", but what else it implies. We have put a great deal of thought into an adoption plan. "This time is different" because we actually provide value to existing clients/servers with JPEG, rather than giving them yet another bitstream to store, or lossy transcoding that just adds more artifacts. The lossless JPEG transcoding is a game-changer, as is the feasibility nowadays of decoding via WebAssembly - even with SIMD for speed.
> If you stop with the JPEG XL library, with Windows compilation an afterthought Good news, we are absolutely not stopping there. The project started in 2015. We're now moving out of the research phase and into productionizing, with integrations and plugins underway. Investing in such things too early would have meant less R&D.
> You'd have to do something drastic, like leaving the image planes unnamed and randomised Interesting idea, thanks for this suggestion.
The True Cost in this specific case is (0..7 bits to fill a byte) + 2 bytes for the RST marker for each interval. 320x240 image and RST interval for each 240/8 block means 30 RST's so something close to 75 bytes "wasted", or, other way to look at it is ~3 bits per row.
The reason is that for the JPEG the Huffman tree is fixed and is transmitted with the image data so restarting is cheap; it does not start from scratch and build a new tree each time; each decoded symbol is also independent so you can stop and restart at any time you want. I don't have any ANS experience at all so won't say anything about that.. but it would be nice if some sort of way would be possible to isolate the ranges.
Good observations, btw. :)
Some of those may be more popular than others:
JPEG XR is also known as "Windows Media Photo" and "HD Photo" and has been supported in the Microsoft ecosystem since Windows Vista and Internet Explorer 9.
JPEG XS, XL and Pleno are of course much newer and will still have to be battle-tested.
But yes, none of the newer JPEG standards have reached the popularity of the first JPEG codec, which is by far the most popular image format ever.
They also use very expensive dedicated hardware to decode Motion JPEG2000 at movie framerate, because it's far more computationally intensive than regular JPEG or even the newer video codecs like H264/H265.