H.264 is Magic (2016)

Yeah, in this era of mobile device dominance i think hardware decoding is key. Suspect Apple will add it then suddenly the Android SoC folks will suddenly consider it a priority and not before.

The Broadcasting industry has definitely sided with VVC. So 8K broadcast will likely be using VVC along with possibly LCEVC [1] as announced by the Brazilian SBTVD standard.

The AV1 question is harder to answer. One could argue VP9 is good enough and a few more years before it reach 100% VP9 hardware decoding on Smartphone. Of course Google could force the usage of AV1 in the future if they slowly phaseout VP9 video and only support AV1 on Youtube, while using H.264 as baseline. Qualcomm should have AV1 decoding on Snapdragon by late 2022 / 2023. Mediatek are starting to roll AV1 decoding as well. No timeline on encoding and generally speaking I dont expect any mobile hardware vendor to use AV1 encoding in the near future.

Even to this day I still do not believe Apple will support AV1 decode, at least not until Google forced the AV1 usage. That is a very contrarian view not just on HN but generally on the internet. But recent news with MPEGLA and AccessAdvance might have change things bit.

[1] vhttps://en.wikipedia.org/wiki/LCEVC

What I think the AV1 initiative neglects as a way of spreading their tech would be giving away a great open implementation of an encoder. x265 and x264 spread their respective codecs all over the internet through pirated content, which forced adoption. The fact that you can play .mkv files almost anywhere now is a testament to that.

wow, this is quite interesting. Thanks!

Thanks a lot for the link, I've been struggling to find good technical explanations/ressources on the subject

"Your browser does not support JPEG XL"; I'm sure it's cool technology but whoever is behind it needs to hire developers and advocates to push code to Webkit / Blink to add support for it, else it won't get mainstream adoption.

Lossless JPEG transcoding is the killer feature moving forward from existing images. I can't wait for jxl's broad adoption.

As a mobile developer I generally see WEBP getting served to Android devices and HEIC to Apple devices. Is there any advantage to JPEG XL over those?

If our app supported older iOS devices, maybe JPEG would be needed as a fallback, but it seems like JPEG XL wouldn't be compatible with old devices anyway, right?

Did they avoid the prohibitive licensing that left JPEG 2000 dead in the water?

JPEG similarly has a progressive mode.

The problem with these formats is that they don’t offer enough of an advantage to overcome the disadvantage of have less than the enormous install base of JPEG. Look how much better PNG is than GIF and realize it only succeeded in replacing it (for static images) because of the Unisys patent suits, not because it’s so much better.

As a professional photographer I can’t believe we’ve been stuck with jpeg so long - 8 bits simply aren’t enough and lossy saving is a bummer.

I’ll need both Capture One and Lightroom to support whatever gets decided on though, and that could take a while.

Whatever happened to FLIF?

>For images, there's a finally successor to jpeg

We've heard this a few times... JPEG2000, WebP, HEIF, AVIF. I bet JPEG XL will finally be it!

What's fascinating to me as a web dev is that (unless I'm mistaken), if I store all my images as JPEG-XL, I won't need to create thumbnails or resized images anymore... I think I can literally just hold onto the originals and just truncate the image binary stream at whatever resolution is required.

I'm not sure how this would interact with "right-click to save image" though... what resolution would it be at that point?

Is this too late? iPhones are shooting in HEIF now, which means there's already millions of HEIF images in the world. Surely that is the de-facto next generation image standard.

I often wonder what we could do with a truly patent free codec once all the H.264 patent expired.

What does it mean for developers that it is patented?

Last I checked[0] I think there were some remaining H.263 patents in the US, but they only covered some error correction or optional annex features.

Interestingly enough, there's some non-standard-essential patents still floating around out there. For example, Mediatek patented converting Sorenson Spark bitstreams for decoding with standards-compliant H.263 hardware decoders[1], and that patent is still live until later in this decade.

[0] I wrote a Sorenson-flavored H.263 decoder in Rust for integration into Ruffle.

[1] The invention is literally just truncation-with-saturation wrapped in a bunch of patent language. If you have the money to front a legal fight against them, feel free to get the patent invalidated. But I don't think it matters.

Can you provide a reference where one could get started with this?

the frequency domain of the frequency domain is just a flipped version of the original image, so that would not work. the idea of frequency-domain based compression is that natural images have little high-frequency information.

With only a slight compression, the image would go shit or totally unrecognizable.

A quick album (there are captions):

https://imgur.com/a/7yHPzRH

This wouldn't work well because in the frequency domain representation, different "pixels" have very different importance for the overall appearance of the image: The pixels at the center of the frequency domain representation represent low frequencies, so compressing them will drastically alter the appearance of the image. On the other hand, the corners/edges of the frequency domain representation represent high frequencies, i.e. image details that can be removed without causing the image to change much. That's the crucial benefit of the Fourier transform for compression: it decomposes the image into important bits (low frequencies) and relatively less portant bits (high frequencies). Applying compression that doesn't take that structure into account won't work well.

You can try with Imagemagick (https://legacy.imagemagick.org/Usage/fourier/#im_fft) :

    for quality in 50 75 90 95 99 ; do
      # FFT
      convert source.png -fft +depth +adjoin source_fft_%d.png
      
      # compress FFT magnitude and phase
      convert source_fft_0.png -quality $quality source_fft_magnitude.jpg
      convert source_fft_1.png -quality $quality     source_fft_phase.jpg
    
      # IFFT
      convert source_fft_magnitude.jpg source_fft_phase.jpg -ift out.$quality.png
    done

Your idea is analogous to what’s done in MRI image reconstruction. “Images” are actually obtained in the frequency domain, but there are various undersampling strategies to accelerate the frequency domain data acquisition. These are then reconstructed into actual images with the inverse FFT.

Yes, I was expecting "a technical walkthrough" of how H.264 managed to improve on its predecessors. There’s nothing specific to H.264 in the article.

MPEG-2 and JPEG are very similar in philosophy as well.

JPEG in particular is still my favorite image encoding technique. Mostly because of how fast it is relative to the competition.

Even earlier with H.261 and MPEG1 which already used block coding motion no ?

(https://web.archive.org/web/2012/http://x264dev.multimedia.c...)

Tied as the leading implementation by 2006, pulled far ahead in SSIM in 2010, and all FOSS. Hat off to Glaser and Merritt!

An interesting Baader–Meinhof for me when I first heard: Glaser (as Dark Shikari) used to be active on HN and wrote the quote (oft misattributed to Descartes),

  Any community that gets its laughs by pretending to be idiots will eventually be flooded by actual idiots who mistakenly believe that they're in good company.

regarding 4chan in 2009. (https://news.ycombinator.com/item?id=1012082)

If you frequently read older threads, get ready to start noticing the username!

It would be better for you to serve .webp and .avif (or optionally .heif for apple devices) inside a <picture> tag with .jpg or .png as fallback, as these formats are designed for static images, but are based on I-frame compressors for vp9, av1 and hevc respectively.

It's curious because where do you draw the line between an algorithm that compresses information and an algorithm that just stores the inforamtion in the algorithm.

Right, you of course also need the decoder in order to reconstruct the data.

In this case, the "decoder" is the english language.

Well, if we're being pedantic here, raw RGB video as would be displayed on a RGB monitor does indeed take 3 bytes per pixel. Technically, YUV is either an optimization or a backwards compatibility measure, and hence the adjective "raw" does not apply to it.

If you actually read the rest of the article, the author covers this in the section about chroma subsampling.

YUV alone is still 3 bytes per pixel and can be converted back to rgb without any loss.

But YUV is almost always combined with color subsampling, which gets you to 1.5 or 2 bytes per pixel.

Most video I deal with is YUV (YCbCr) 422 at 30 bits per 2 pixels

Lectures from a course I took (CS 674 at UNR) can be found online here, along with class exercises and names of textbooks: https://www.cse.unr.edu/~bebis/CS474/

Would highly recommend it! We went over pretty much everything in this article, plus more :)

Timing is everything.

It's appeared on HN several years ago: https://news.ycombinator.com/item?id=12871403

PNG is a good baseline for image sizes before adding any kind of smart compression. Nothing wrong with that, and the comparison would not look much different against JPG (except it would probably look actually worse than the video).

> You make an H.264 encoded copy of that. Let's call it C1.

> You then make a new cut of your movie, M2, which is mostly the same footage as M1 except that you've shortened a few scenes and lengthened others.

Your new cut gets encoded from whatever you've chosen from frame-index numbers and timestamps in the various original raw yuv project files (or ProRes, etc, depending on what your camreas output).

The existence of the original cut should be irrelevent because what you're not doing is taking the compressed output product of C1 and re-cutting it into C2 unless for some catastrophic reason you have lost or deleted all of the original raw/uncompressed original camera files.

If you are generating new C2 from your original uncompressed (or lightly compressed) files, of course the encoder will be making new decisions on where to insert I-frames and P-frames based on the duration of each camera cut, changes you've made to CGI, other visual factors in the re-cutting.

> Suppose someone has an uncompressed source S. They compress it with a lossy compressor producing C and distribute C to you. You decompress C producing S'.

> You then compress S' with a lossy compressor (the same type that the original producer used--e.g.., if C is an MP3 you use an MP3 compressor) producing C'. I don't know about video, but for audio (at least back in days when MP3 was starting to get big) C' would be lower quality than C.

all of this is generally a bad idea unless you have no other option but to work with received files that are already compressed. say for example you're working on a documentary about something in a conflict zone and somebody sends you a highly compressed h264 file recorded on a smartphone. in this case there is no uncompressed original available.

you're going to want to first extract it into an uncompressed yuv raw file on disk so that you can work with it in your standard editor, and then whatever scenes you choose from within it will get re-encoded into your final output.

> If they just do something simple like make every Nth frame an I-frame then after the first different between M1 and M2 it is unlikely that C1 and C2 will have many I-frames in common, and therefore also not have many P-frames in common.

Modern encoders won't be using a fixed rate for I-frames unless you force it to. It will choose an I-frame when deemed optimal.

You're correct in that using a compressed source which goes through a video editor and then to another encoder will likely not pass through to the encoder which of the frames to encode were originally I-frames in the source. This is because video editors combine multiple sources together so there is no "single" source.

But if you're not actually editing and you're just "trimming" and "joining", this can be done with perfectly matched i-Frames and p-Frames, but probably not b-frames?

Even when using the commandline x264, you can specify which frame numbers you'd like encoded as I-frames.

> Are there any compressors that can figure out that they are dealing with something that already has undergone the "throw out imperceptible parts to make it more compressible" step done and just skip to the next stage, so they produce a C' that is a lossless representation of S'?

The compressor will try throwing away exactly the information that was thrown away the first time you compressed it. So basically it will leave the content as is, because there is nothing extra to throw away.

You can easily see this with an MP3 file at 128 kbps - the first time you compress it most of the very high frequency content will be thrown away - you can see this on a spectogram of the uncompressed file compared with one of the compressed file. But the if you compress it again, the second compression spectogram will look very similar to the first compression one, because there is not anything else that you can throw away.

But there is a complication - the audio file is typically stored in the time domain (PCM), but the compressor operates in the frequency domain (FFT), and there will be a conversion between these two domains that you can't avoid. This conversion unfortunately will lose a bit of information and degrade quality a bit.

For the first question, I ended up using this trick for a video editing app on Android phones about 10 years ago in order to cut video together on low end, out of date (Android 2.3) phones. They couldn’t handle video compression in a reasonable time and we didn’t want to upload/download to process on a server.

The point of the app was to sync the cuts to music cues, so each clip had a defined length. I ended up doing it all through file manipulation. You can cut into a video file starting at any arbitrary I-frame then trim it to the desired length. I would cut the input videos down to size then concatenate the files, replacing the audio with the new soundtrack at the end.

It worked great, only took a few seconds to create the final edit. Of course you couldn’t overlay text or filter video, but I still think it was a valid solution.

With the requirement of starting each clip on an I-frame, there was some imprecision in where your cut would actually start—an arteur might have a problem with their masterpiece being butchered that way, but it would certainly work well for some special cases like efficient distribution or being able to show a diff that a video was unaltered outside of timing cuts.

> If they look for scene changes and make new I-frames on scene changes, then we might expect that at least for the scenes that start identically in M1 and M2 they will get identical I-frames and P-frames up to their first edit if any.

> Question: are there any encoders that when encoding M2 to produce C2 can be given M1 and C1 as references using them to adjust I-frame spacing so as make as many C2 I-frames as possible match C1 I-frames?

I suspect if you were to encode both files with x264 with identical settings, in crf mode, with vbv disabled and keyint=0 (unlimited keyframe length), its scene detection should place I-frames in the same places (that is, on scene cuts, not on the timeline). Maybe some scenecut tuning would be necessary.

> That would allow C2 to be stored efficiently as a binary diff from C1. This could be handy if C1 and C2 needed to be checked into a version control system, or you needed to distribute C2 over a low bandwidth or expensive link to someone who already had C1.

I'm not aware of any automated way to do that, but you can do a similar thing manually using mkv's ordered chapters. You first compress the original cut, then you compress any additional scenes and insert them where you need. For example, you can make a mkv file for a theatrical cut of a movie, and then make a separate file for a director's cut that is only as big as the additional scenes are, since it uses the theatrical cut file for the common scenes.

Isn’t that exactly what HEIC is?

Bro. Why are you promoting so much an app that barely does anything (opens a series of images from a hosting), as 'pornographic application that change your life'?

They did the same for AV1, image size is 50% of HEIC (25% of JPEG) due to increases in coding efficiency.

Wow that comparison is really cool. Looking at BPG vs original, you can see some loss of fine detail as the file size cranks down, but you can get an excellent reproduction with an order of magnitude less storage. Seems like BPG is smoothing the image a bit. For example, the space shuttle image has film grain in the sky, but BPG smooths this away.

Significantly better than JPEG at small file sizes... BPG eliminates the obvious color banding and does a much better job preserving detail.

What's the patent status? As far as I know, H265 is patent encumbered(and not actually supported on a large chunk of Android devices for the same reason) , so including this somewhere might invite legal trouble.

I cannot get over how much of a CompSci God Fabrice is. What a person!

That's amazing, thanks for sharing. Going in rather uninformed, I expected it to 'look better' by typical smoothing tricks and whatnot...but no. It has -tons- more detail...looking at little things like wrinkles in the socks and shirt and whatnot. Impressive.

This was fascinating. I changed to JPEG large vs BPG small and they are almost indistinguishable. JPEG small vs BPG small wasn't even close (JPEG being much worst).

Why would anyone use such exotic unsupported format when there is WebP, AVIF or JPEG XL, which are all much more supported and better? It's shame they are not natively supported by most of the camera Android apps.

It does look really good for detailed areas, but when you look at the abandoned building pic BPG can never quite get the smooth sky gradient right, even at Large size.

> You wouldn't say HHHHHHHHH.

You wouldn’t and shouldn’t. There are only nine “H” in there!

Ah ASCII art. Long rows of identical characters. The only thing that actually allowed a dial up modems compression claims to actually live up to the marketing hype....

Doesn't that description basically presuppose that you already understand compression? "HHHHHHHHHH" is 10 characters, but "10 tosses, all heads" is 21. Already understanding compression, I of course know that the second part could be efficient if encoded properly, but someone actually needing the explanation probably wouldn't know that.

It's not really wrong, and not really an "oversimplification" as the parent article calls it. It's just a bad example, and it seems confused about the audience it's trying to reach. Either they already understand compression, and it's superfluous, or they don't and it's just confusing.

I think you could find this interesting: https://www.youtube.com/watch?v=aF1Yw_wu2cM