Data Compression Nerds Hate This One Trick [video]
media.ccc.de
media.ccc.de
The original impetus that led to MNG was animated GIF. While GIF was also designed as extensible, GIF practically only had a handful number of extensions and wasn't that complex. MNG in comparison was essentially a binary version of SVG as I have described, except that it had no vector graphics and can more accurately be described as a binary slideshow format [1]. No one asked for this! The PNG development group forgot its goal and made a much bloated specification. As a result libmng, while working, was a quite big library to be incorporated into Gecko and ultimately rejected in favor of APNG which only required a small patch to libpng. It is amusing that the original PNG specification had chunks corresponding to GIF extensions including animation one; those chunks might have been widespread in the alternate universe!
If the group's goal was indeed a slideshow format, it should have been two separate formats, one for multi-part images and one for animations and object transformations. The former is necessary because one aspect of GIF was that a difference from the previous frame can be encoded, so this "Delta-PNG" format should have been made into the mainline PNG specification as an option. (These additional images could have been used as previews, for example.) And anything else should be a simple textual format that refers to external files. MNG instead had chunks for JPEG images---came with its standalone sister format called JNG---, which is absurd when you think about it---why should there be JNG when JFIF and Exif already exist? The modern SVG happens to be perfectly usable as this second format, and it is fine being textual.
[1] If you can't believe what I've said, see the table of contents of the actual specification: http://www.libpng.org/pub/mng/spec/
See this example[1] for illustration.
They specifically called out that it doesn't support vector graphics, so the latter shouldn't be a surprise.
And if you want to check if it's like SVG but worse, you need to compare that spec with the SVG spec, not look for the word SVG.
Of course it turned out that ID had decided they wanted a longer bleep-boop every time you spun the wheel, and the real-time screen position update was dependent on the sound completing. Click-ticks were completely unacceptable, but adding a scroll buffer and detecting it's state while filling an audio buffer with tiny fractional bleeps allowed reasonable latency scrolling. Of course buffer cleanup and beep-slice management was the most annoying. Sadly, the phone shipped.
Over the years I saw tons of software in windows that made mistakes like this. Some stupid animations or sounds that were synced with the actual logic of the application holding everything up.
Years ago there was some app our team installed that just took forever to install around 500MB of data, like 30 minutes. When we monitored it there was nothing going on most of the time. We ended up just batching out all the files, registry keys, and dll registrations to install in less than 5 minutes.
As that data compression nerd, QOI is both a refresher and an annoyance. It serves as a good example that input modelling is very important, so much that QOI almost beats PNG in spite of very suboptimal coding. But QOI also represents a missed opportunity because, well, we can do much better if we are willing to use something like QOI instead of standard formats! Weirdly enough people seem to value only the simplicity when it comes to QOI, which is not the main takeaway I believe. Maybe QOIR [1] might result in a better balanced format in the future though...
I can't imagine people will start storing their family pictures in a new format they've never heard of which is not supported by any software they use for "just" 20% better compression. Do they even want lossless compression in the first place (if you don't ask them directly and call it that)?
For example, was it worth it in the end? Did they announce anything? Did they switch to another method or give up on the idea, or do we not know?
> While we did ensure that the reported vulnerabilities don’t affect our internal use of Lepton, we unfortunately don’t have the capacity to properly fix these and future issues in this public repo.
As far as I know this is indeed the last known public mention of its use, but given that Lepton was already in use and dropping it would substantially increase its traffic, it is reasonable to assume that its use somehow continues to this day.
https://github.com/Hakan-Abbas/HALIC-High-Availability-Lossl...
The apparent death of HALIC was indeed unfortunate, I've heard that its decoder was less than 1K lines of code so it must have had a carefully designed and optimized context model in its heart. But there are so many other formats that are F/OSS but weren't highlighted enough anyway...
Fwiw, all (lossless) compression algorithms will increase the size of some inputs.
They rarely meaningfully increase the size though. Typical compression algorithms, when faced with hard to compress sections, can add a handful of bits to wrap them up. Total overhead a fraction of a percent.
When QOI encounters noisy pixels, it has to spend an extra byte per pixel, inflating the file by 33%.
An interesting and somewhat inspiring bit of trivia from the video: the creator barely understands modern image compression techniques (from their own words), but this hasn't stopped them from coming up with that impressive result.
After watching this video, my first thought was whether recent results from columnar compression (e.g. https://docs.vortex.dev/references#id1) applied "naively" like QOI would have good results.
I started with a 1.79MiB sprite file for a 2D game I've been hacking on, and here are the results:
PNG: 1.79 MiB
QOI: 2.18 MiB
BtrBlocks: 3.69 MiB
So, there's magic to being Quite OK that is more than just applying compression techniques than elsewhere :)
> How one guy in his bedroom (kind of) beat all of PNG's combined multi-decade effort in one year, and why that's strange.
PNG is a 30 year old format and hasn't changed much over the years (as far as I am aware). PNG at it's core is still just some basic filters with DEFLATE on top. The decades of work would mostly be spend on optimizing the encoding speed and finding better heuristics for selecting filters. There is a lot more impressive codecs nowadays.
Apart from that I am still a big fan of QOI. It has amazing results for how simple the format is.
What I want to see is the total IO + CPU time across libraries for my specific IO and CPU constraints.
Sure, it makes benchmarks more involved to display, as scalars are not enough anymore, you need multiple curves, but that's meaningful at least.
To illustrate, if I have very fast IO, then I probably don't care of the compression ratio, it will be faster to download the raw payload and have 0 decompression cost.
On the other end of the spectrum, if I have very slow IO, I would gladly have a much slower decompression algorithm but higher compression ratio for a faster overall timing.
This is especially important because cloud storage for instance are rather cheap, but slow. Caches/CDNs are very fast, but storage is expensive. Etc
This knob is typically called as a (compression) level, even though it isn't advertised as such because it is hard to translate level into your metric. Some libraries like zstd support IO-adaptive operations as a simpler alternative too.
The benchmark doesn't know what your CPU performance and bandwidth are (or will be on some future device). You should be able to multiply these costs out yourself.
CPU speed is getting faster at a much higher rate than IO. It makes sense, one is infrastructure, the other is the silicon in your pocket.
That's why you're getting downvoted, and why algos always push more and more to the CPU. Statistically, the correct posture is to expect the CPU to consistently get faster. I mean, if you make an image format now, you want it to work with CPUs in 30 years. Will IO speed gains overtake CPU improvements in 30 years? Most certainly not.