What's Going on with HEIF and Mac OS 10.13 | Hacker News Reader

What's Going on with HEIF and Mac OS 10.13

shapeof.com

85 pointsmpweiher62 comments

"Apple says it plans to add HEIF encoding for Macs in a future release" -- https://arstechnica.com/gadgets/2017/09/macos-10-13-high-sie...

It seems a bit crazy that for a new image format, developers need to rely on an operating system shipping an API, instead of depending on an (open-source?) library.

Presumably both are options, but the OS library has hardware acceleration.

It looks like FFmpeg can do it, albeit in a somewhat roundabout way.

http://jpgtoheif.com

Patents.

One gets you guaranteed immunity if you sign up for three different patent pools and pay a massive arm and a leg. The other claims to be I encumbered but the reality isn’t so clear.. but you’d probably have Google’s lawyers to help you out if it came to that.

They can do it themselves if they want to. Apple is including support with High Sierra for everybody but anyone can add support to any formats on their own apps without involving Apple at all. Many apps, like VLC and HandBrake for example, have been supporting H.265 for a while.

It looks like this is also true for mpv. Honestly, having to deal with codecs is a waste of time from the user's perspective. I seriously love how players like mpv and VLC can play pretty much every format under the sun without additional configuration.

Without any GPU support sadly.

The format is incredibly complex kitchen-sink of features and legacy ISO specs. There is an open-source library for it, but it's written in C++, and issues filed about it make it look like it's insecure (i.e. a large complex C++ codebase written by less than an expert) and not even entirely compatible with Apple's implementation.

So I think it's fine to leave HEIF as basically an internal format of Apple's camera app, and for everything else export to formats more suitable for interchange (e.g. BPG which uses same the compression scheme, but is a simple lightweight container).

I dunno, I think the container format sounds pretty reasonable, at least at a high level (I haven’t tried to implement it or anything). Many of those features [choice of lossy or lossless compression; extensible embedded metadata; multiple images per file; animation; transparency; embedded thumbnail; non-destructive indication of preferred crop/rotation; choice of color spaces and bit depth; ...] are extremely useful, but currently are only supported (in conjunction) by completely proprietary app-specific formats like PSD or the like.

I can imagine at least a dozen use cases for this format that current formats don’t serve at all.

See this post implementing it in ffmpeg: https://ffmpeg.org/pipermail/ffmpeg-devel/2017-August/215003...

I guess the main possibilities are (1) use an existing openly specified format, for which there is already at least some published working code, even if it isn’t perfect, or (2) invent a brand new format, and hope you don’t screw it up somehow by overlooking something important. I’m not entirely convinced #2 would have been any better (does Apple have a good reputation for inventing data serialization formats?), and inre #1 I’m not sure if there’s any extant choice better than this new format with support for all of the uses Apple cares about.

Considering that all the widely supported formats throughout the industry have various severe limitations so that it’s impossible to cover the typical use cases (even for just web images, say) using less than 3 or 4 completely different formats, it’s not clear that anyone else has figured out how to really solve this one either.

Too bad that the world couldn’t find someone with good taste, extensive experience, and a forward-looking vision to make a more future-compatible image format about 15–20 years ago.

In this particular case, (1) and (2) were pretty much the same, as the format that became HEIF was drafted up in 2013 and finished in 2015, by Nokia, as a contribution to the proceedings of the joint MPEG/ITU-T JCT-VC working group working on the standard MPEG-H; it was accepted.

Previously, with Apple 'live photos', having two separate files sitting on the disk was good enough.

HEIF is very new, but it's now blessed by the premier standardization organizations in this space as a component of the next wave of media coding. Aside from the the innate value of standards in which industry players agree on common formats and semantics, Apple's move needs to be seen in context of the rival 'Alliance for Open Media' [1], which includes many big names working on future open media codecs outside of the usual MPEG / ITU-T / contributors' patent pool combo. Apple's adoption of an MPEG standard is also signalling its commitment to that particular brand of standardization.

[1] http://aomedia.org/about-us/

It was over engineered from a Image format perspective, but it is very likely Apple has a lot of things in mind and things like Live Photos or whatever they are making.

Webp supports most of those features actually, iirc. I mean HEIF is to HEVC as WEBP is to VP8, right?

Not that webp is as incredible as Google claimed, but that was always a given with the inferior codec it’s based upon.

How does WebP compare?

Keep in mind WebP is essentially a VP8 I-frame in a RIFF container [1]. VP8 is the predecessor video codec to VP9; VP9 being the codec targeted by its makers to be in competition with HEVC which is commonly deployed inside HEIF -- whether they succeeded in this promise is an exercise left to the reader, possibly by following sources such as the ones compiled on Wikipedia [2].

But not all compression techniques that are applicable to video are relevant with still images. WebP's performance has been evaluated and some have found it lacking [3], while some have found it good [4].

Discounting the fact that the HEIF container has some useful features on its own, the picture inside is often an HEVC I-frame, which is the same idea as Fabrice Bellard's BPG. Therefore, comparisons between VP8 and BPG may also be indicative of typical HEIF-as-it-will-be-typically-deployed performance.

[1] https://developers.google.com/speed/webp/docs/riff_container [2] https://en.wikipedia.org/wiki/VP9 [3] https://en.wikipedia.org/wiki/WebP#Criticism [4] https://blog.cloudflare.com/a-very-webp-new-year-from-cloudf... [5] https://bellard.org/bpg/

Personally I'm not impressed with the quality of the lossy encoding, it's not enough of a improvement on JPEG to make it worthwhile, for lossless however it's the best format I know of right now, compresses MUCH better than PNG, and both compression and decompression is MUCH faster.

FLIF compresses better than webp, but it also compresses/decompresses more slowly, and the difference in size is quite small so overall for lossless compression, WebP is my choice.

Oh, thank you!! Music to my ears! I designed the WebP lossless format. Our most recent effort PIK tries to achieve the same in the lossy world that WebP lossless has done in the lossless coding -- a practical, efficient, fast and simple compromise.

You're welcome, and huge thanks for WebP.

PIK had flown under my radar, looks really interesting. Any estimate of when a first release will be out ?

Well, that's a two-pronged issue: one is that it is a patent minefield unlike ever before seen, the other is that it is incredibly slow in software at any quality level.

Also, it is legally terrifying to build anything on HEVC, I'm frankly surprised Apple is even trying. The basic license fees are easily ten times as much for HEVC as they were for H.264, and there are known applicable patents which aren't even in the pools (both pools! there are two!). There are also potentially license fees that apply to individual performances/transfers of content, so if you're Netflix or something you'll end up paying for every unit of content shipped.

For the lot of these reasons, the Alliance for Open Media (including the Daala team (Xiph and Mozilla), The Thor team (Cisco), Microsoft, Nvidia, AMD, Intel, Mediatek, Netflix, Google, and a good couple dozen other major players) has formed and will probably be freezing a standard bitstream format this year on a format that outperforms HEVC for a given bitrate, a given CPU time target, and on energy consumption (and implementation complexity) in hardware.

>both pools! there are two!

There's actually three now, but the newest one hasn't announced their pricing yet: http://velosmedia.com/

And don't forget all the people not in pools! It's like a support program for lawyers...

Yeah, while I can understand Apple picking up HEVC, this image format makes very little sense to me.

Outside of Apple I can't see this have any uptake due to being patent/royalty laden, which means that images you encode using this format need to be reencoded again (and degraded in case of lossy, which is the most common type) in order to be shown on the web or on other operating systems/devices.

Meanwhile, every Apple user’s photo library will be half the size compared to JPEG. For many users photos are the single largest category in their iOS device storage. More importantly for Apple, they (want to) store everyone’s photos in their cloud, so they need half the storage and bandwidth to pull that off.

For sending to other people, most photos are going to be downsized anyway so a transcoding isn’t the end of the world.

Apple doesn't like downsizing messages. iMessage sends full resolution, multi-MB pictures.

That's a setting - though I don't know the default.

Irrelevant because we are talking about compression formats.

It is very likely in the next 12 - 24 months time, every Smartphone shipped will have paid the HEVC patents. So while the PC industry may be a little different, all mobile phone will have HEVC support soon.

Now whether the software for HELF support will catch up is another question.

Bold (nay, courageous!) move to launch a JPEG replacement and not follow up with desktop support.

What happens if you edit a HEIF in Photos.app, and perhaps try to share a flattened version via imessage, email or upload to wordpress? Do you get a transcoded JPEG out? (What's the quality loss like for that case?)

Without industry buy-in and high quality open source support, I'm struggling to see how this will ever expand beyond archiving iPhone camera rolls...?

To avoid compatibility issues every time an image leaves the device it does so as a JPEG. Even if nobody else supports the new format, in their mind it's still a win because of the space saved on the device itself.

For some reason, discord sharing photos fails with HEIF images.. it uploads as 'unknown' with no extension and doesn't display.

Not sure if they somehow accidentally opted in to getting HEIF from the chooser or what.. but there you go.

> Bold (nay, courageous!) move to...

Something about this sequence of words reminds me of Kruppe (From the Malazan Book of the Fallen series).

> Without industry buy-in and high quality open source support, I'm struggling to see how this will ever expand beyond archiving iPhone camera rolls...?

It wouldn’t surprise me to see Microsoft support this, or perhaps camera makers. After all it’s 50% smaller than an equivalent JPEG, thats a nice feature.

There’s nothing stopping google from supporting it other than the fact that they seem to like to force their own standards on everyone else (locking 1080p+ YouTube behind VP9).

Even if it’s only available in the Mac/iPhone ecosystem… that’s a HUGE ecosystem. Anytime something is being sent over iMessage you can be pretty sure that the receiving device can decode that image format so you don’t have to send a JPEG or an H264 file.

The ability to effectively double the capacity of iPhones when it comes to taking pictures and/or video is a really big deal. Combined with the fact they finally got rid of the stupid 16 GB tier and even the cheapest iPhone can now hold a LOT more pictures.

or perhaps camera makers

It would be nice to see camera maker support, but camera makers have been more than a bit retrograde in the software department for a long time: https://jakeseliger.com/2017/09/22/if-i-were-a-camera-compan...

Camera software is terrible to the point of being defective (the following is true for all or almost all professional digital cameras):

1) histograms for processed jpeg using a built in preset instead of the actual raw (want to know if you've really clipped the highlights? Can't)

2) no automated ETTR even on mirrorless cameras / while using live view, despite all the necessary information being available

3) no automated hyperfocal distance focusing, despite this being absolutely trivial to implement on any autofocus body

4) only 1d series canon cameras (which are used for action and sport photography, not landscape, so this feature is mostly useless) do multi-spot averaged metering, despite this being trivial and having been done on old Olympus film cameras

5) they're not programmable (i.e. focus to 5m, fire shutter after 4s, then...)

It's like they don't let anyone using the things with any skill get their hands on the firmware. Using Android would be a disaster of latency, boot times and bloat, they should just stop being terrible.

Sony camera firmwares already include an Android subsystem...

It also has a full web browser that is only used to download 'apps' like the remote control one (nice feature, but still), obviously with terrible input due to the camera being a camera. I was stunned when I saw it on my A7r.

The browser is part of Android, as are the apps. Input is slightly better on cameras that have a touch screen.

> What happens if you edit a HEIF in Photos.app, and perhaps try to share a flattened version via imessage, email or upload to wordpress? Do you get a transcoded JPEG out? (What's the quality loss like for that case?)

ArsTechnica discussed sharing from Photos.app in their High Sierra review:

> And when you send an image or movie via Mail or any Share extension, macOS assumes the worst of whatever device might be on the receiving end. At least for now, all of these files will automatically be converted when they’re sent.

https://arstechnica.com/gadgets/2017/09/macos-10-13-high-sie...

Developing a macOS app that targets HEVC/HEIF, I was wondering this myself.

Now get this--HEVC encoding is faster on a last-gen iPhone than it is on a top-of-the-line current MBP!

Because intel does not have HEVC encode hardware and A11 does?

No, they both do. And I'm talking about the A10 even. Still faster than an i7/Iris Plus 650. Of course, the graphics hw is killer in iPhones, so it's not such a surprise. But it makes you do a double-take.

That’s interesting; do you have a reference where I can read more about this?

You can always make an encoder faster than another encoder. What you have to ask is - how good is it for the speed? (This is called rate-distortion optimization.)

Better yet, how good is it for the bitrate and speed? (This is the really hard part.)

Nitpick, but RDO refers to trade-off between bitrate and distortion (quality), IIRC

Sometimes you're limited by time more than bits, but they're both rates.

I have noticed how slow HEVC encoding is on my High-Sierra 2017 MBP too. Would love to offload some of those tasks to iPad Pro. Would you know of any HEVC encoding apps?

I'm confused; I have an 8 and I have High Efficiency selected, but it appears I can take a photo and see it and open it on my work Win7 PC. It's a .jpg file format and seems to work just fine.

The mystery deepends with video: when I do a 4k60 video, MPC-HC is listing it as still using H264 but I thought 4k60 required H265:

Video: MPEG4 Video (H264) 3840x2160 60fps 90829kbps [V: Core Media Data Handler (h264 high L5.1, yuv420p, 3840x2160, 90829 kb/s)]

Audio: AAC 44100Hz mono 92kbps [A: Core Media Data Handler (aac lc, 44100 Hz, mono, 92 kb/s)]

What am I missing?

I believe it converts those files to compatible formats when you transfer them off the device. The exception seems to be when you have macOS 10.13 and use Photos, the images and videos stay in HEIF and HEVC.

In Settings / Photos, there's an option to keep the original hevc/heif when transferring to PC/Mac.

Perfect, found that now thank you! A bit weird to have it separate to the option to take in that format, though sort of understandable in a twisted logic since it's to do with the photos app not the camera app.

4K60 does not require H265, it's just that most H264 chips can't handle realtime 4K60 encoding.

https://www.github.com/google/pik could be an opensource alternative, more dense for photographs than AV1 or HEIF, less complex, and decodes 100x faster

https://encode.ru/threads/2814-Psychovisual-analysis-on-mode...

Without write libraries it's a dead product.

What surprises me is how fast HEIF files are on my iPhone 7 Plus (instantaneous browsing through photo library on iPhone. No different to JPEGs), yet how sluggish it is on my 2017 MBP 13". Select a .heic on your Mac and spacebar-preview it. Then select another one in the same folder and you'll see it takes a good few seconds (depending on size of image).

It was likely due to HEVC licensing. HEIF is just the container, so they still should’ve provided ways to generate .heif files using other codecs like JPEG.

They may have assumed that many people would associate HEIF with HEVC (what would technically be a .heic file), and wanted to avoid confusion. But we’re still confused!

So when AV1 happening? Not that mac would support it but was just wondering.

You can generate HEIF with FFMPEG right now, but that’s hardly a solution.

How does it compare to WebP?

It's at least a generation improvement. WebP is not very good; skal standardized it way early, before Google let anyone else review VP8.

They should have held off until VP9.

Everything about VP8 (and, by extension, webp) was such a bungled mess. You come out with a format that barely competes with something released 5 years earlier then release an image format based on it while its superior replacement is being developed and close to release?

AVC was released 10 years prior to HEVC; it had a good run and that encouraged adoption. Google barely got people to support their inferior (though mostly free) replacement before they obviated it not five years later, thus guaranteeing no one would ever bother with hardware encoding for an ever changing always outdated on arrival family of codecs.

A real shame.

Reply on news.ycombinator.com