Hacker News Reader

Comment by lifthrasiir | Hacker News Reader

Slides: https://filmroellchen.eu/talks/QOI%20EH25/#/0/1

As that data compression nerd, QOI is both a refresher and an annoyance. It serves as a good example that input modelling is very important, so much that QOI almost beats PNG in spite of very suboptimal coding. But QOI also represents a missed opportunity because, well, we can do much better if we are willing to use something like QOI instead of standard formats! Weirdly enough people seem to value only the simplicity when it comes to QOI, which is not the main takeaway I believe. Maybe QOIR [1] might result in a better balanced format in the future though...

[1] https://github.com/nigeltao/qoir/

View on HN

Let's say you had a lossless image format that's 20% smaller (on average for pictures people send over networks) than PNG. Let's say it takes 10% more computing power than PNG. Do you stand to make money? What would it be used for?

I can't imagine people will start storing their family pictures in a new format they've never heard of which is not supported by any software they use for "just" 20% better compression. Do they even want lossless compression in the first place (if you don't ask them directly and call it that)?

That's the portability bit the presenter mentions, and is a very important concern in practice. But how about recompression? For example many PNG files are suboptimally compressed partly because PNG is an old format and also because many softwares have been too dumb to produce a well-optimized PNG. In that case we may benefit from a transparent recompression, which may be done either by using a better library like libdeflate or by internally using a separate format that can be quickly transformed from and back to PNG. In fact Dropbox did so for JPEG files [1]. When I'm saying "so much better" I was thinking about such opportunities that benefit end users.

[1] https://github.com/dropbox/lepton

Dropbox apparently abandoned the project. Do you know what their takeaways were from trying to improve the JPEG storage?

For example, was it worth it in the end? Did they announce anything? Did they switch to another method or give up on the idea, or do we not know?

Dropbox apparently still uses Lepton or any successor internally, but the open source version is abandoned because it posed larger maintenance burden than internal projects.

Has Dropbox said this anywhere? Or are you assuming it based on something like "this kind of project is very easy to maintain internally so there's no reason why they would have stopped using it"?

At least at the time of abandonment, Dropbox did say (emphasis mine):

> While we did ensure that the reported vulnerabilities don’t affect our internal use of Lepton, we unfortunately don’t have the capacity to properly fix these and future issues in this public repo.

As far as I know this is indeed the last known public mention of its use, but given that Lepton was already in use and dropping it would substantially increase its traffic, it is reasonable to assume that its use somehow continues to this day.

That's pretty strange. If they're using it in a meaningful way, then they're applying it to arbitrary files, so they'd generally need fixes for almost all the bugs. So what resources would be lacking so badly that they give up on releasing?

It could just be a lame excuse. Open sourcing with zero support would be better than not doing it. But I imagine that this seems valuable enough to keep secret, or at least not to pay money to polish up for external use.

QOI is just a simple filter. It cannot do full compression. In fact, in certain cases it can increase the size instead of compressing. It is unnecessarily overrated, of course, mostly because it is open source. The rest is irrelevant. There is another codec that is as fast as QOI (or even faster and multi-core) but with a much higher compression ratio. HALIC (High Availability Lossless Image Compression). But because it's closed source, it definitely didn't get the attention and respect it should have gotten. And that's why I think it stopped developing.

https://github.com/Hakan-Abbas/HALIC-High-Availability-Lossl...

>>> In fact, in certain cases it can increase the size instead of compressing.

Fwiw, all (lossless) compression algorithms will increase the size of some inputs.

> Fwiw, all (lossless) compression algorithms will increase the size of some inputs.

They rarely meaningfully increase the size though. Typical compression algorithms, when faced with hard to compress sections, can add a handful of bits to wrap them up. Total overhead a fraction of a percent.

When QOI encounters noisy pixels, it has to spend an extra byte per pixel, inflating the file by 33%.

one pretty simple qoi improvement would be a tag to say to store the next n pixels uncompressed. I think there's room in the tags for adding it

I wanted to use HALIC but ended up using SQZ instead. QOI is overrated but QOIR and my own QOIX are a bit more interesting, albeit ultimately stupid.

A simple filter may still do wonder if it is carefully chosen ;-)

The apparent death of HALIC was indeed unfortunate, I've heard that its decoder was less than 1K lines of code so it must have had a carefully designed and optimized context model in its heart. But there are so many other formats that are F/OSS but weren't highlighted enough anyway...

For a format to proliferate as a standard, it has to be open source. Nobody cares about 5% smaller images if not everyone can view them.

Historically speaking, that’s not true; WMV and FLV were extremely popular for over a decade in the early 2000s. Closed source formats can, and have, become de facto standards if they solve a pressing need that open standards don’t (or provide a stepwise increase in functionality or performance) and if there’s sufficient market demand.

Reply on news.ycombinator.com