HEIF – High Efficiency Image File Format
nokiatech.github.io
nokiatech.github.io
[1] https://github.com/nokiatech/heif/blob/master/LICENSE.TXT
The image encoding layer of HEIF is the same HEVC, but the container is MPEG-4 Part 12 (the Quicktime-descendant ISO Base Media Format; the core behind .mp4, .3gp, etc.), which theoretically gives it wide support. In this structure, HEIF supports image sequences, which will help it compete with animated GIF, GIFV (which is really just a short MPEG-4 Part 14 file containing usually an H.264 video track), animated WebP (who uses these?), and WebM.
This format doesn't really blaze new ground (EDIT: it does in the sense that it defines a container format to express image-y constructs like still images and sequences of images in an ISO media container, but see my other comment that asks how this is similar but different to video [3]), but if this repackaging and the resulting code donation and political clout helps it gain traction, we still would gain a lot.
The problem, of course, is always with backwards-compatibility. WebP was aggressively promoted by Google the same way Microsoft used to promote its quirky Windows Media formats back in the early 2000s, but the WebP and non-WebP camps are still largely separate. This is unfortunate, because WebP in my opinion really isn't very good, and a backwards-compatible compressor on top of JPEG, such as Dropbox's Lepton [2] achieves similar results. Any HEVC-based image compressor is necessarily incompatible with JPEG; so broad buy-in would be required from makers of software and hardware products, from operating systems, file managers, image viewers, image editors, cameras, and the like, to really enjoy its improved compression rates, vs. being just another incompatible format that only functions inside controlled ecosystems.
The video codec AV1 currently in development was supposed to be a grand alliance of disparate companies to agree on a common format for the future, and rectify the WebM vs. non-WebM split, but continuing to promulgate sophisticated formats based on work outside of this scope (such as MPEG- or ITU-derived custom formats) just muddies this further.
[1] https://bellard.org/bpg/ [2] https://news.ycombinator.com/item?id=13230805 [3] https://news.ycombinator.com/item?id=14490734
What's really different between this and the existing usages of the MP4 container for images, though? Is it just the packaging as an "image" format, or specification of more image-y metadata?
You are right -- the main difference is that BMFF was originally designed for time-based media (video, audio) but isn't in itself well suited towards media that isn't time based, like images and their associated metadata (e.g. Exif).
HEIF extends the ISO BMFF so that you can have untimed media -- one or more photos, a collection -- or even mixed media comprising both untimed and timed media. Apple's live photo is a good example of mixed media, comprising a photo and a video.
But the HEIF container format goes so much further. You could have image sequences -- for example a higher-quality version of Animated GIF (using I-frames only, or P/B for more compression) with looping -- tiling, auxiliary images like depth maps and alpha channels, or even stereo images with a left and right channel.
Though originally HEIF came out of the HEVC standards track, hence the words "High Efficiency" in the name, it was later extended to include other codecs like JPEG, AVC. There's no reason it couldn't be extended to include VP9, PNG, or any other codec.
Think of HEIF as a versatile, extensible, standardized container format. The media coding scheme is separable. This is a big deal for the future of image/media coding because we're no-longer locked into "yet another format" that's tied to the codec. With the ISO BMFF box model, HEIF can grow to adapt new constructs and codecs for some generations to come.
There were similar approaches in the past and they failed. Electronic Art's IFF, or later TIFF, were also designed as extensible containers. The vast majority of the software handling these formats handles only the most popular codecs, the fringe one pushing it forward will get ignored.
If we can get everyone to agree on a format that handles layers, exposure bracketing, transparency, animation, high bit depth, different color models, both lossy and lossless compression, every common type of metadata, etc., that would be great.
TIFF is not useful if you want to use a state-of-the-art lossy codec and get hardware-accelerated decoding on an arbitrary mobile device, or if you want to display an animated image with transparency on a web page.
That's why it's best to accept whatever format is already out there and author to it - the only thing that matters is that it actually works when it's out there.
Actually I'm surprised there's still no good way to do animations with transparency, but maybe nobody wants to use it?
It's also extensible, in that you can define new item types and box structures to describe any new imaging construct you like. For example, you could host images with alpha, depth maps, stereo left/right channels, a HDR image with each of the raw brackets and the final fused image, etc.
The format relies upon the ISO Base Media File Format file-type branding to distinguish the infinite possibilities into a clear set of capabilities required for playback; in the same way MPEG4 does for video codec profile and levels.
In the past, I articulated, on the topic of 'Ask HN: Software for writing a diary that will be around in 20 years from now' [2]:
I'd be wary of the archival potential of formats that are solely used on the Web with little usage on tangible physical hardware by major commercial publishers -- the Web of today moves very fast and technologies come and go. Google is pushing WebP, WebM, but work is already under way on a big consensus format called AV1. When AV1 comes out, new VP8/VP9 content will likely no longer be produced. Browsers periodically prune older features, 20 years is almost as long as the web has been around, and given enough time support for the format may only be available in software that make format coverage an explicit goal (ffmpeg, libav, VLC). Opus is being made a mandatory audio codec for WebRTC, teleconferencing is usually ephemeral -- will there be lots of .opus files sitting on disks in the future? Too early to tell, not worth gambling on.
The context of this was choosing formats for the express purpose of long-term archival, but our incidental usage of formats today will shape the sort of files that will be naturally around in our future.
[1] https://news.ycombinator.com/item?id=5589206 [2] https://news.ycombinator.com/item?id=12979854#12980359
We had similar problems in the past with physical media, and still do to some extent, but in the purely digital domain this problem is somewhat more tractable (content negotiation helps facilitate these transitions for those willing to work that into build pipelines and maintain those over time, for example.)
There is just less expectations to be able to interoperate via "dumb files" in the "app ecosystem" where the apps run in their own isolated sandboxes and import/export are explicit actions.
Not saying that this is the best thing, but the situation is different enough from the time when WebP/WMP were introduced that HEIF actually might have a fighting chance. The fact that HEIF appears to be far bigger improvement over JPEG than previous efforts of course also helps.
Apple is just blatantly not interested in cooperating with the rest of the market. Mozilla, Google and Microsoft are all part of AoM, along with a bunch of hardware companies and distributors yet Apple is absent and apparently pursuing HEVC instead.
This isn't new either, Apple pushed HLS when others were pushing DASH and MPEG-TS when others were pushing fragmented MP4.
They seem really determined to go their own way and ignore everyone else for some reason.
And many more companies support it over AoM: http://www.mpegla.com/main/programs/HEVC/Pages/Licensors.asp... http://www.mpegla.com/main/programs/HEVC/Pages/Licensees.asp... http://www.hevcadvance.com/pdfnew/LicensorList.pdf http://www.hevcadvance.com/pdfnew/LicenseeList.pdf
Including the likes of: BBC, Samsung, Dolby, GE, MediaTek, Philips, Mitsubishi, Warner Bros, Sony etc. And as for hardware vendors well AMD, Intel, Nvidia etc all support H.265.
I didn't say it was, I said Apple was pursuing it.
> And many more companies support it over AoM
You can't claim that yet as AV1 isn't yet finalized.
Do more companies support it? Yes.
Failure to ship is also a competitive disadvantage - AV1 may be great (we don't know) but if this one gets traction first that may not matter.
You need to think about HEIF as the container and disconnected from the codec used to compress the media content. In that sense, HEIF is a massive trailblazer. It gives us an ISO standard to describe image and mixed media consistent with existing methods (i.e. MP4) while allowing for new constructs (i.e. useful features of the future in addition to tiling, depth map, stereo, etc.) and new codecs as they are developed and adopted. HEIF is a universal forward-looking format.
Until now, almost all of the major imaging formats were tied to the codec and not terribly extensible to new use cases. While BPG is a great format in the short term (it gave us HEVC coded images around 2.5 years ago), it isn't an ideal choice for the long term when viewed as above.
BTW: Did you know that you can embed LPCM/μ-Law PCM/ADPCM audio data in JPEG/Exif?
TIFF is a great format but its ability for extensibility is limited. You can't readily contain in a video track in addition to a photo, or a burst sequence that utilizes intra-frame prediction.
> BTW: Did you know that you can embed LPCM/μ-Law PCM/ADPCM audio data in JPEG/Exif?
Yes; I once owned point-and-shoot cameras that did this. The audio was pretty poor quality because they didn't employ compression and wanted to keep the file sizes small.
However you're limited to the 64k maximum JPEG marker segment size for non-image metadata; or ugly hacks like chaining segments (e.g. ICC profiles). Exif is strictly limited to being contained within 64k. How big is your audio track again? ;-)
JPEG has had its time as a wildly successful format but has also held back the imaging world from adopting a standardized way to cater for new applications; burst above, mixed media (photo + video + audio), depth maps, stereo images, alpha (that's not a hack), lossless, etc., etc. HEIF has all the key ingredients, including extensibility, to support these modern applications and grow as the universal container format for decades to come.
However, the HEIF wrapper for H.265 strikes me as quite complex. It brings baggage of ISO wrapper formats and tries to support everything ever crammed into any image-ish format. That, in addition to patent licensing, may be another difficulty in widespread support for the format. It will be hard to implement robust, secure and fully-featured players.
Besides the coding efficiency, JPEG really hit the big time because of the readily available libjpeg cross-platform implementation. In this case, HEIF can leverage existing implementations of the ISO Base Media File Format box model; it builds heavily on that standard.
Nokia is doing the right thing by releasing their format handling library, though perhaps they could loosen up their license to include commercial use. ;-) Hopefully there are some developers amongst us inspired enough to start a new open project that, like libjpeg, brings HEIF to the masses.
What's so bad about WebP? It's not the best thing in the universe but it beats JPEG and PNG and has a wide support base. With a shim it has native support in most browsers anyways since WebP is a just a single frame of video. And as a bonus you don't have to worry about someone suing you for royalties down the road.
WebP is based on VP8, which is now obsolete. It was meant to be a H.264 competitor, a generation behind H.265 and VP9/VP10. WebP compression efficiency is much closer to JPEG than H.265.
This format beats WebP by a margin wider than WebP was smaller than JPEG.
* Lack of clear spec ("The code is the spec")
* Lack of versioning of spec (at least 3 major versions with different capabilities/features of which not all degrade nicely)
* Forced 4:2:0 chroma subsampling
* 8bit-channel only (no support for 10bit+ aka wide-gamut images)
"FLIF does not have a lossy mode, but interlaced files can be decoded progressively so we can simply use something like dd if=lossless.flif of=lossy.flif bs=1024 count=5"
You can see this in the examples where BPG does well in stream loading: https://nokiatech.github.io/heif/technical.html
HEIF would probably be similar to BPG in performance.
Still FLIF looks pretty darn interesting.
I don't know what you mean by well. As far as I'm concerned lossy FLIF beats JPEG easily. Here is a comparision:
Here is the correct link: http://flif.info/example.html
And yes "well" is relative and what I meant by that is that its not readily easy for a normal user to do it.
BTW I think Flif is damn good particularly progressive decoding. Its unclear how good HEIF progressive decoding is.
https://en.wikipedia.org/wiki/High_Efficiency_Video_Coding#P...
If you choose to implement this format, I hope you enjoy negotiating licenses with at least four different entities so that you can display pictures.
Worse, there is a financial incentive to implement only a subset of the decoders - HEVC is more expensive than AVC which is more expensive than JPEG. Even things like 4:4:4 color are more expensive than 4:2:0 color for HEVC.
It's a standard format, it's faster than the competition, it has just as good (or better) quality vs compression metrics...
... all it needs is support and it'll take off. Apple is a good start for that.
And either way VP9 and AV1 are simply inferior codecs to H.265 in almost every way. And as for royalty free well that's because MPEG-LA has't ever sued but they did give a royalty free license to Google for patents infringed by VP9. And since Google accepted it does indicate that AV1 is likely to also be patent encumbered.
We would all like a royalty free codec but it isn't happening anytime soon. And so failing that I would prefer to go with the codec with the best support which by far is H.265.
Source?
AV1 is already beating HEVC in compression and it's not even finalized yet: https://youtu.be/lzPaldsmJbk?t=2416
Also the link you posted is using x265 which as the article suggests was ranked 4th out the 6 available encoders: http://compression.ru/video/codec_comparison/hevc_2016/MSU_H...
And whilst AV1 and H.265 are pretty close in PSNR where it is inferior is in vendor support. There are TVs for example available today that natively support H.265 whereas AV1 is still in development. Not to mention all of the terrestrial broadcast providers all using H.265.
The other encoders there are commercial or not publicly available as far as I can tell, so it's hard to use them as a comparison. Plus according to that PDF, the best encoder is 0.85x x265, which is about the same as AV1 in its present state in the video I linked. And again, AV1 isn't even finalized yet, so I actually find it somewhat impressive that it can already compete against these highly tuned encoders with years of development put into them.
> where it is inferior is in vendor support
Of course there's no vendor support now, the codec isn't even finalized yet. However the codec is backed by Microsoft, Google, Mozilla, Qualcomm, Intel, AMD, ARM, Broadcom, NVIDIA, Adobe, Netflix and the BBC, i.e. all but 1 of the major browser and OS vendors, every major mobile, server and desktop processor company and the 2 most widely used video streaming companies.
Also without Apple it's over. YouTube and Netflix will be forced to support H.265 or give up on iOS/OSX users (350 million users).
AV1 isn't finalized yet. Of course HEVC is being shipped in more stuff, it's being shipped at all. I'm not debating this, there's no debate. At present HEVC has more support than a codec that isn't yet finished. H.264 had more support when VP9 was being developed too, that doesn't make VP9 inferior to H.264.
> broadcast standard for terrestrial TV which is still hugely popular in many countries
Broadcast TV is a different world to the web and has very different concerns.
> Also without Apple it's over. YouTube and Netflix will be forced to support H.265 or give up on iOS/OSX users (350 million users).
This is demonstrably not true as YouTube and Netflix both currently and successfully utilize VP9, which Apple doesn't support. Apple users just get the inferior H.264 codec.
The question is really whether the browser and other OS vendors are going to start supporting HEVC and given the current patent situation, that seems impossible, so we're going to end up in a fragmented world where for web streaming, most users get AV1 while Apple users and anyone else who doesn't support AV1 gets H.264.
You keep saying that as if it solves anything.
Released > Unreleased.
Is that a criticism of the format's potential? No! But it may as well not exist until it is finalized... so it can't be a competitor to a format that has been.
It will have good vendor support when it exists.
This was your claim.
Let me sum up the entirety of my position so we're on the same page:
- In terms of compression, AV1 is competitive with or beats HEVC, despite not being finalized.
- AV1 has support from everyone but of import save Apple when it comes to the web.
- Apple's support is not necessary for a codec to be successful (see VP9).
- AV1 is not finalized. There's more work to do. Particularly, I don't expect the present encoder/decoder implementations to be competitive with the HEVC encoders when it comes to CPU efficiency.
- HEVC has solid adoption in some areas (the more "traditional" areas like video cameras, TVs, broadcast) but has zero adoption when it comes to the web and this is extremely unlikely to change due to the patent mess.
- As members of the AoM, it's near certain that Microsoft, Google and Mozilla will add AV1 support to their browsers, which, depending on the particular source you look at, cover roughly 70% of the total browser market.
- Due to AoM support and HEVC not being an option, AV1 will be the primary successor to H.264/VP9 on the web.
- Due to its likely widespread availability from streaming services on the web and support from hardware vendors, AV1 will be prolific in other kinds of hardware.
- Due to the support of Apple and the broadcast industry, HEVC will likely not go away anytime soon.
- I agree that HEVC is not currently a viable alternative to HEVC, it not being finalized yet and all, however despite that, there are already aspects in which it is already superior, namely compression and likelihood of being supported on the web.
AV1 it's not same thing as VP9
> "If you [...] order or agree to the institution of patent litigation [...] against any entity [...] alleging that any of these implementations of WebM or any code incorporated within any of these implementations of WebM constitutes direct or contributory patent infringement, [...] then any patent rights granted to you under this License for these implementations of WebM shall terminate as of the date such litigation is filed."
Image formats are already some of the biggest attack surface of modern systems, "executable" image formats (hello PDF) have an absolutely ghastly track record there.
And not being able to open an image if you don't have an internet connection sounds dreadful.
Caching the decoder smells like having already downloaded the image library. Mostly just different in that "first run time" is now always the same thing as "install time".
Re: requiring an Internet connection—you still do require an Internet connection to display an image, if it's in a format whose decoder you don't currently have installed; you just currently have to manually 1. figure out what the format even is, 2. select a library package for a decoder for that format, and 3. install that package.
Presumably such libraries would be able to register decoder-URLs they are (non-reference) implementations of—like they register MIME types today—so your system would be able to display standard formats no problem; it'd just be the weird rare or new ones that would trigger the zero-install process for a decoder.
And that's accepting that WASM is safe, which I do not axiomatically accept. History suggests that I am very safe in claiming that most implementations will end up with some catastrophic-level security vulnerabilities in them before all's said and done.
Okay, but as I said before, for specific cases we can still do things the old way. I.e., upon decoding we detect that the url points to a known format, and we run the fast+safe decoder. For unknown formats, we download the slow decoder and use that. This way, we have more than what we would otherwise have (flexibility). And in the future, WASM will be faster anyway. I only see benefits. Of course, the sandbox should be formally proved correct first.
The computational power you need for image decoding is extremely narrow and easy to make safe. You need some mathematical operations and some loops. You don't need any APIs or data structures. Mask off all the pointers and you can have have a provably safe interpeter/compiler that runs pretty fast.
Wikipedia also does this to enable video playback on Edge and Safari [2].
The idea to do this within the format itself is quite old. The very early versions of what became Vorbis had a very similar concept, with programmable passes. Matroska once had a field for a link to download the VfW decoder dll (the horror!). It's never really caught on, due to reasons already listed by sibling comments.
[1] https://arewecompressedyet.com/analyzer/?maxFrames=4&decoder...
The HEIF examples on the Nokia HEIF site do something similar; they implement a HEVC decoder (and do the HEIF container processing) all in JavaScript using http://www.libde265.org. The images you see are decoded into a HTML5 canvas.
Compared to this, HEIF is a different way of packaging HEVC frames; namely into ISOBMFF (MPEG-4 Part 12), the most basic level of MPEG container. This comes with benefits [2] that come along with using that particular container, but also drawbacks, because with some of the advanced features of HEIF you're essentially describing a sequence of frames -- which is almost like video -- but you're doing it in a way incompatible with the way you'd describe video. I'd be curious if the two "representations" are losslessly convertible, for example.
[1] https://bellard.org/bpg/ [2] https://nokiatech.github.io/heif/technical.html
They are: you could just extract the HEVC NAL units, and re-write them into a MP4 or QuickTime container, making sure to properly place the codec configuration box, etc.
HEIF also goes beyond a sequence of frames, in that it can describe alpha planes, depth maps, tiling, etc. In that case there might not be an analog with a standard video. If you really wanted to decompose a HEIF container, you might choose to extract the raw media into elementary streams (for HEVC or AVC; or if you're using the JPEG codec, just plain JPEG files) adjacent to any metadata like Exif, etc. This is essentially what Nokia's conformance files are [1].
[1] https://github.com/nokiatech/heif_conformance/tree/master/bi...
It's more like HEIF is for HEVC what WebP is for AV1.
Could have been branded better if I'm honest. I know engineers generally don't care about stuff like this but some of us will have to say this term many times a day if takes off.
This is an (significant) improvement over JPEG.
Why is that? Every edit you make to that photo means that the photo will degrade in quality. I always thought JPEG artifacts and compression lossage was a bad thing for photos, not a good thing?
Sure JPEG artefacts are bad, but that can be mitigated (stop compressing so much). Your photos being an order of magnitude larger is worse, and inherent to the way PNG works.
https://www.youtube.com/watch?v=IheZzcYUV9w https://www.youtube.com/watch?v=w7vXJbLhTyI