Introducing the ‘mozjpeg’ Project
blog.mozilla.org
blog.mozilla.org
I remember what it was like to watch a 320*200 JPEG image slowly build up on a 386SX PC with a VGA card. Today, a HD frame compressed with JPEG can be decoded in milliseconds. This highlights the secret to JPEG's success: it was designed with enough foresight and a sufficiently well-bounded scope that it keeps hitting a sweet spot between computing power and bandwidth.
Did you know that most browsers support JPEG video streaming using a plain old <img> tag? It works also on iOS and Android, but not IE unfortunately.
It's triggered by the "multipart/x-mixed-replace" content type header [0]. The HTTP server leaves the connection open after sending the first image, and then simply writes new images as they come in like it were a multipart file download. A compliant browser will update the image element's contents in place.
https://code.google.com/p/chromium/issues/detail?id=249132 http://blog.chromium.org/2013/07/chrome-29-beta-web-audio-an...
It's too bad IE never supported multipart/x-mixed-replace or we might have seen more live updating websites earlier. Now that we have WebSockets and well understood long polling approaches it doesn't matter any more, since x-mixed-replace would keep the download spinner spinning forever and the newer approaches don't have that problem.
I was disappointed when they were unceremoniously ripped out, but yes, websockets are better.
But there is more: (https://www.google.nl/search?q=fast+fourier+transform+use+ca...)
It must be cool for sure.
Nothing in Chrome, but an animated (slowly) image in Firefox
For web cams, where you might not want true "live" video but prefer higher resolution still frames, it sounds like a reasonable choice.
The MJPEG format doesn't really exist anymore: it was designed in the '90s to account for interlaced video content, but that's a rare breed nowadays. For progressive video, Photo-JPEG is equivalent.
Many popular intraframe video codecs are basically the JPEG algorithm with some modifications for specific pixel formats and some custom metadata. These include Apple ProRes, Avid DNxHD and the stalwart DV format (as in MiniDV tapes).
JPEG optimization using open source tools is an area that really needs focus.
There are a number of lossless JPEG optimization tools, but most are focused on stripping non-graphical data out of the file, or converting the image to a progressive JPEG (since progressive JPEG's have rearrange pixel data you can sometimes get better compression since there may be more redundancy in the rearranged data). Short of exceptional cases where you can remove massive amount of metadata (Adobe products regular stick embedded thumbnails and the entire "undo" history for an image) lossless optimization usually only reduces file size by 5-15%.
Lossy JPEG optimization has much more potential. Unfortunately, beyond proprietary encoders, the most common lossy JPEG optimization exclusively is to reduce the JPEG quality. This always felt like killing flies with a tank, so advances in this area would be awesome.
I've written extensively about Lossy optimization for JPEGs and PNG, and spoke about it at the Velocity conference. A post and my slides are available[2].
[1] - http://httparchive.org/trends.php
[2] - http://zoompf.com/blog/2013/05/achieving-better-image-optimi...
http://cbloomrants.blogspot.com/2012/04/04-09-12-old-image-c...
But it's useful to point out that a lot of new overly complex formats can't beat something as simple as that.
There are a lot of post-processing techniques to try and mitigate this, but in my experience they tend to do about as much damage as they fix. The proper solution is to overlap the blocks using one of the myriad techniques DCT-based audio codecs use.
It is bizarre to me that for all of the attempts to beat JPEG, nobody seems to have tried simply overlapping the blocks by 2 pixels. You'd have an implementation only marginally more complex than JPEG (in fact, you can even implement it on top of an existing JPEG encoder/decoder) with a slowdown of only 25%.
But a bigger problem is that no one is really interested in designing a new still image codec that's better than JPEG, since JPEG can't be unseated. So video codecs are where the practical development goes. And avoiding in-loop deblocking filters there means OBMC, which is extremely computationally intensive.
You might be able to shave a percent or so off the download size of compressed assets.
[1] https://news.ycombinator.com/item?id=5316595
(For context: libpng is a "purposefully-minimal reference implementation" that avoids features such as, e.g., Animated PNG decoding. And yet libpng is the library used by Firefox, Chrome, etc., because it's the one implementation with a big standards body behind it. Yet, if Mozilla just forked libpng, their version would instantly have way more developer-eyes on it than the source...)
And if you want much much smaller PNGs, then try http://pngquant.org or http://pngmini.com/lossypng.html
We've been using the desktop version. Would love to use something similar on a server, but jpegmini is overpriced for our scenario (I'll not have a dedicated AWS instance running for compressing images every second day or so). Will definitely check out this project :)
In fact, on a JPEG-heavy site that I was testing with FF 26, there was such a degradation in terms of responsiveness that transitions would stutter whenever a new image was decoded in the background (while preloading).
It made the effort to save 2-4% in size wasted with a worse user experience.
The problem is that even if the bug would be fixed in recent FF versions, libjpeg is basically used in all other browsers as well.
All jpegoptim does is rewrite JPEG with optimized Huffman tables, so it shouldn't have any impact on decoding performance. In the process it also changes progressive to baseline, which is even slightly faster to decode.
If you can reproduce the problem with libjpeg-turbo (which is the library that browsers use) you should definitely file a bug.
Maybe images could be encoded with reference to a common database we share that has the most repetitive data. So perhaps 10mb, 50mb or 100mb of common bits that the compression algorithm could reference. You would build this dictionary by analyzing many many images. Same type of approach could work for video.
JPEG is designed to compress any image. But imagine a new algorithm, gJPEG, which is only designed to compress photographs of grass. And furthermore, you get 100MB of raw buffer space in the executable to store some precomputed data that would be useful to gJPEG doing its work. It's quite possible you could significantly improve on the general performance by factoring out some data that's common to typical grass photographs, so that data could be stored once-and-for-all in the decoder and then omitted from each of your (presumably) billions of individual grass photographs. On the other hand, it's pretty tricky to make it work, so you might not be able to do such a thing effectively.
In information theory, my understanding (which is quite limited) is that when we talk of bits transmitted we can think of it as "uniquely identifying from within the set of total possible messages". So, if you shipped the entire 10,000 image catalog "transmitting" an image from within that catalog would take you a mere, uh, let me count my fingers, 13 bits.
We could go one step further and find some way of hashing all the data together to remove any redundancies and so forth - but the problem alas is about defining arbitrary images :).
What you described tho is kind of what happens with all compression algorithms except on the "micro"/individual image level. You may already be aware but check out Huffman coding: http://en.wikipedia.org/wiki/Huffman_coding for a simple intro.
How do you read my question and interpret simply it as "lets send all of the images in full and then give their index and call it compression?"?? What I suggest is that we take a standard encoding technique like Huffman, or some modification, but rather than creating a table based on data in an individual image, build this code table by analyzing many, many images.
I have read the Wikipedia article on Huffman coding before. However, the details are not really important in regards to my point.
What I am suggesting is that rather than looking at just the bits in individual images and using them to construct a Huffman table or some other kind of reference, look at the bits on many, many images and create a larger reference table. And then of course you may need a local table for things in the image that don't quite correspond to the larger table.
Earlier compression techniques were much more constrained in terms of processing power, RAM, network connectivity etc. and so distributing and using a large table for compression was not practical. I am suggesting that someone who has knowledge of compression engineer a system where 10MB, 50MB, or 100MB of RAM is used and a large common bits file is transmitted, rather than starting with the idea that almost all of the data or all of the data has to be contained in one file. I am not suggesting that an existing compression algorithm could be translated directly into this general concept. I am suggesting an engineering effort starting with different constraints and trade-offs.
Like this: imagine I've got an array of shorts representing audio data. If I've got two files with similar segments, I've saved one short:
[1, 2, 3, 2, 1, 2, 5] [7, 2, 3, 6, 8, 9, 3]
So I can say that [2, 3] is represented by a new value (z), and can shave a short off of both streams. Then what happens if a new stream comes along with no similarities:
[8, 2, 7, 1, 7, 3, 7]
...you still have to send each value.
Maybe I've just demonstrated that I don't know anything about compression, but I would be interested in working with you on this.
Lots of different people on this forum, no offense was intended, and like all nerds I get excited when I get to share knowledge.
>How do you read my question and interpret simply it as "lets send all of the images in full and then give their index and call it compression?"??
Because it struck me as analogous, and yeah I'd call that compression - the message length for one is immensely improved.
Okay, so here's my admittedly piss poor understanding of most compression: you either find more intelligent ways to strip bits from the source in ways the consumer won't mind, or you find more intelligent ways to build reference tables given your problem domain.
I'm sure you know this, but the first one is why jpeg/various mpegs are successful: they have complex quantization models that eliminate gradients we won't notice or frequencies we can't hear.
The way you achieve better results is through building better models for how information in your problem domain is related. If we're compressing text and we know the language we can start referencing letter frequencies and index along that and so on.
The way this works in video to my knowledge is, amongst many other complicated things, they take NxN blocks of images and store only the deltas between Y numbers of frames.
So - perhaps you "image reference blob" could build a reference table for all 16x16 px blocks and transmit only the indices for them, and we're back at my original comment. But to my (again, please correct me) understanding those are kind of the only alternatives? Encoding and decoding is an interesting topic.
I very infrequently have to think in binary and I had perhaps too much fun counting to 10,000 on my fingers; my undergraduate was a long time ago and my knowledge on the topic sparse, so I'm interested in hearing more about it.
Edit: A good reason given in the reply by joshmoz below.
The beauty of open source is that if the maintainers of libjpeg-turbo want to incorporate it, they can, and I'm sure the Mozilla devs would be more than happy to help.
You'll never get to play alongside big boys like Apple and Facebook with this 'talk to upstream' attitude of yours. That's just not how the game is played.
As long as ISPs/telcos don't go back to the days of AOL network wide compression to reduce bandwidth beyond low quality I am for this at service level like facebook/dropbox uploads. I hope this inspires more in this area. Games also get better with better textures in less space.
Still to this day, I am amazed at the small file sizes macromedia (adobe now) was able to obtain with flash/swf/asf even high quality PNGs would compress. So yes we all have lots of bandwidth now but crunching to the point of representing the same thing is a good thing. With cable company caps and other bandwidth false supply shortage that focus might resurge a bit.
There is some good news in that OpenJPEG (http://www.openjpeg.org/) has been making significant progress in recent years and is now on track to become an official reference implementation:
https://groups.google.com/d/msg/openjpeg/OMc40gUsBIw/UM1ggXk...
Hopefully this will also translate into continued performance work and robustness testing, which would mean potential hope for a browser other than Safari to add native support:
Software encode/decode of JP2k is the hard part. That's why there's little adoption of J2k outside of hardware solutions.
[1] http://en.wikipedia.org/wiki/Digital_Cinema_Initiatives#Imag...
JP2K is still the leader in visual quality per byte. All DCT-based compression systems (jpeg, mpeg, dv, etc) are prone to "mosquito noise" artifacts which are extremely annoying in moving pictures since the noise moves around and looks like mosquitos flying about. The usual workaround for mosquitos is to blur the picture a bit to make it easier to compress; but that softens the edges of object onscreen. Wavelet systems like jp2 suffer from different, less annoying, artifacts.
gzip'ing 4:2:2 is a terrible idea :-). Keep in mind that Jp2K does have a lossless mode which you can use if you really don't want to lose any quality.
The advantage of the wavelet format is that they can implement progressive resolution decoding, so a decoder only needs to read half of the data from a 4K source to decode a full quality 2K image.
But, as always, when you think an engineering decision is insane, you're probably missing the context in which the decision was made.
This is Mozilla's roundabout way of saying that they want to put off starting on WebP or JPEG 2000 as long as possible.
I haven't heard that brought up as an objection to the format before, though. If it were really a fundamental stumbling block, there are likely ways to adapt the format around it.
[1] https://developers.google.com/speed/webp/faq#what_is_the_max...
Though I'm sure they have their reasons to stay away from an across-the-board superior format. I just wish I knew what they were. Sigh.
That's not hard: they put out a report months ago concluding that the claimed advantages of WebP were in reality slim to nonexistent, and that it could likely be matched and even outperformed by improved JPEG encoders...which is exactly what they're building now.
According to Google it's about 30% smaller than JPEG: https://developers.google.com/speed/webp/docs/webp_study
It's a massive effort to switch entire industry to a new format for a 30% gain. With a drop-in replacement of libjpeg you can get 10% gain right now.
Also WebP is based on 2006 VP8, which is obsolete now. Chrome ships with VP9 that's further 26% smaller on keyframes.
Developing a still image format from Daala is possible but not as trivial as it looks (the same can be said for the WebP work from WebM).
The speed modern web concerns me. The standards are not moving forward. We still use HTML, CSS, Javascript, Jpeg, Gif, and PNG. Gif especially is a format where we could see similar sized/quality moving images at 1/8th the file size if we supported algorithms similar to those found in modern video.
In all of these cases, they aren't "tried and true" so much as "we've had so many problems with each that we've got a huge suite of half-hacked solutions to pretty much everything you could want to do". We haven't moved forward because we can't. WebP is a good example of a superior format that never stood a chance because front-end web technology is not flexible.
GIF, sure. But, IMO, nobody should really be using GIFs anymore. We've had <video> and APNG for a while if you want animation.
What would be great if there was some way for an animated gif's frame delays to opt-in to being interpreted correctly by browser- That is, a 0-delay really would display with no delay, and so optimisation strategies involving the splitting of image data across multiple frames could be done- and when read in by a browser, all frames would be overlaid instantly, module loading time.
What other things can be done to further optimise animated gif encoding?
one format can support both lossy and lossless images and use better compression between frames to achieve 10x better results for animations than animated pngs without visible loss of quality
I spent 2 seconds in GIMP to make the following GIF: http://i.cubeupload.com/kbrcfQ.gif Please show me the video format that can replace it.
Getting better encoding by changing the quantization method can't be purely a function of file size, traditionally PSNR measurements as well as visual quality come into play.
Good to see some work in the area, I will need to check out what is new and novel.
That said, a company I worked for many moons ago came up with a method where by reorganization of coefficients post-quantization, you could easily get about 20% improvement in encoding efficiency, but the result was not JPEG compatible.
There is a lot that can be played with.
They are shell scripts running many different optimizers
e.g. the hardware decoder in the Raspberry Pi http://forum.stmlabs.com/showthread.php?tid=12102
You can test it out by selecting the "lossless" option and uploading a jpeg. Enjoy!
This is just a start, we wanted to have something people could use on day one. Further developments to come.
https://blog.mozilla.org/research/2013/10/17/studying-lossy-... was rather flawed and you never acknowledged this.
(see https://news.ycombinator.com/item?id=6581827 - only one of the four sets of test results might have been valid)
By the way, I think this is an awesome effort by Josh and the others working on this. I work mainly in security and networking, and I'm excited about anything we can do to make end-to-end communication more efficient because these kinds of improvements help enable more use of HTTPS and help reduce (hopefully eliminate) the need for content transforming proxies.