Hacker News Reader

Comment by hirako2000 | Hacker News Reader

hirako2000

Certain company that commented on the thread already turned their image processing pipelines to use jpeg xl instead, given the significant bandwidth savings. Now all they got is the significant hole in budgets since it may never pay off.

We can't know how many are using jpeg xl since there is no available metrics to find out.

View on HN

We can easily scrap the internet. There should be a statistic about it.

But for companies they should have two formats anyway: the original file and the optimized file.

Changing the optimized file should be an existing batch process

>they should have two formats anyway

Good thing JXL has the best progressive support of any image format available! [0, 1] Something which you can only find partially with incremental decoding in WebP and doesn't really exist in AVIF (just one of the many limitations of being bolted onto a video codec). You might often reconsider having both versions of a file with this capability. (which wasn't exploited much or at all with "dumber" and heavier progressive JPEGs)

0. https://opensource.googleblog.com/2021/09/using-saliency-in-...

1. https://jpegxl.io/articles/faq/#doesjxlsupportprogressivedec...?

With the high cost of cloud storage I see it as a pretty big burden to host multiple copies of an image, particularly if you feel compelled to have multiple optimized images such as JPEG, WEBP, AVIF, ... That's why I didn't start using WEBP for images until support was universal enough that I could give up JPEG.

(For a large image collection you have some images that are heavily requested and most of the cost is network transfer, but you also have many images that are infrequently requested and for those the cost of storage in the dominant factor. I launched a large image collection in the late 2000's and many sites that did the same thing at the same time were ultimately crushed by running costs, Pinterest and Instagram were survivors, overall the economics of video collections turned up to be better than images.)

I guess technically you could store original and encode to <whatever format is popular> on-demand then just cache the most popular ones

Yes, we do that. Well, the CDN does the caching -- but we still get a 99.99% hit rate.

Please forgive the case study format, but I'm quite proud of what we did and this is the easiest way to back up my assertion with data: https://aws.amazon.com/solutions/case-studies/skyscanner-clo...

We're currently generating JPG, WebP, and PNG, depending on the source and the target. The problem with doing this on-demand is that formats that are great for delivery but slow to encode aren't useful. I did try to add AVIF, but the latency on the first request was too high for it to be practical.

I guess "preheating" cache with preemptively encoding and putting new content in cache before it is shown to client could work?

But still, anything AV1 based just needs a lot performance to encode that just doesn't feel worth it at the moment. Our system to encode videos was tooled for that but developers eventually decided it's too slow/cpu hungry to bother. It was some time ago tho, encoders did get better...

In internet-scale systems complex delivery systems involving transformation and caching are not necessarily a win for performance insofar as once you give up a millisecond of latency you cannot get it back. You very much can get a win out of a CDN but somebody has to do aggressive tuning of absolutely everything.

It's not for performance but for not having to permanently store terabytes of differently encoded data.

You could possibly also pre-populate cache - all new media files get sent to encoder and cached preemptively, and if they are popular enough they just stay there till they are not.

Reply on news.ycombinator.com