iPhone 5 web teardown: How Apple compresses video using JPEG, JSON, and canvas | Hacker News Reader

iPhone 5 web teardown: How Apple compresses video using JPEG, JSON, and canvas

docs.google.com

321 pointsdbloom115 comments

Nice to see they're getting bit by their own decisions.

They've gone out of their way to ensure that you can't ever play an Audio/Video clip automatically on page load in iOS Safari. Every new iOS release for the first few years included a patch to kill any new workarounds that let you do so. (Curse you iOS 4.3 for taking away our simulated clicks.)

But now they have a use case of their own that needs it, so they invent the mother of all workarounds. And now the rest of us will start using it. And it will become the "standard" way to run video on iOS Safari.

Then they'll kill it off for iOS 7. Then they'll have to come up with an even crazier workaround so that they can render their own website a few months later.

Fascinating to watch this play out.

How do you know they feel they're getting 'bit'? Maybe they think this is the optimal solution for what they are trying to accomplish? Maybe they have an internal tool for generating the images they need to accomplish these things? Maybe it's "smooth as butter", as they would say, on their side?

There's a reason that we have video codecs like mpeg and h264 instead of using mjpeg for everything; the compression is much better. In that sense, a stream of jpegs is far from optimal, even if it's smooth as butter on a fast connection.

I wonder what they'll do when the much better h.265 comes out.

Yeah, so clearly they weren't trying to optimize for KB size.

>They've gone out of their way to ensure that you can't ever play an Audio/Video clip automatically on page load in iOS Safari

Which sounds very wise from a user perspective.

Imagine every crappy web advertiser auto playing videos on page load on Mobile Safari. Yuck.

I don't want videos to auto-play on my Laptop, why would I want them on my phone?

>Curse you iOS 4.3 for taking away our simulated clicks

Simulated clicks? I rest my case...

What do you do when that happens on your laptop? You close the page and move on. Why does it need to be any different on your mobile/tablet? I assume the battery cost of 2 seconds of video loading/playing once in a long time is negligible.

I usually do not use my laptop in a restaurant.

>What do you do when that happens on your laptop? You close the page and move on.

No, I proactively install apps such as "Click to flash" and the like to make sure this DOES NOT happen on my laptop.

>Why does it need to be any different on your mobile/tablet? I assume the battery cost of 2 seconds of video loading/playing once in a long time is negligible.

Because I could be anywhere surfing with my mobile/tablet, in the dentist's office or even in a meeting. I don't want pages to jump at me with video I have not requested. Ever.

And I don't want web marketers to assume anything about the battery life cost of their actions on MY devices.

I disagree with the person above you's "negligible" comment. From what I understand, there is a significant battery difference for Android games between ad-supported free and ad-free paid. And the difference is due to all the extra radio usage for advertisements.

Thing is, it's not a video. It's a glorified animated gif.

This has almost nothing to do with autoplay and the video tag and much more to do with a desire for a script-controllable animation.

(Most obviously: they don't start playing until the animation data is loaded, so that it doesn't do the herky-jerky frame-playing during load, like actual animated gifs.)

> This has almost nothing to do with autoplay and the video tag and much more to do with a desire for a script-controllable animation.

Yes it does. Apple has disallowed you to play a video on iOS without user action. So it would not be possible for the video to autoplay (even with the scroll trigger). HTML5 supports preload and has the canplaythrough event which allows you to avoid buffering. They went through this rigamarole because they have previously decided developers should not be able to do this.

They went through this rigamarole because it's an animation and animated gifs aren't great.

... And they made it an animation because of the limitations they built into iOS.

Would anyone deliver an actual video this way?

Say, something with more than a few dozen frames?

It would be unfeasible to do so. Assuming the displayed image is a reasonable size, say a dimension of 320x240, you would run out of pixel space in a jpeg due to dimension limitations after about 38 minutes @ 24fps. Granted, that's not using the manifest, just straight skipping around in the sprite.

To make this anywhere near possible on a large scale would be feasible though. The easiest way would to be create the manifest while encoding the video. Take the same compression scheme but save the encoding diffs for every frame.

Seems like a terrible use of time.

Apple's manifest supports spreading the video across several images (the spinning earphones use this).

Real-world video (filmed with a camcorder, not just a screencast) of any length probably wouldn't work well at all with this compression scheme, though. Apple's JS video compression basically only optimizes out parts of the image that don't change at all. If you're filming the "real world", pretty much every 8x8 block in each frame is going to have changes (it's basically guaranteed due to sensor noise, lightbulbs flickering, etc).

I'm not sure if I should be impressed by this. From a technical standpoint, it seems quite clever and unique, but it also seems like a lot of work just to avoid using a <video> tag.

As noted in the post, iOS devices will only do video tags fullscreen.

I just tried on my friend's iPod and the animation doesn't play there anyway (it just stays on the "lock" image). Maybe the iPod browser is different (my Android doesn't play the animation either) but I'd be surprised if the animation worked on an iPhone but not on an iPod...

It definitely works on iPhone (I've tried on my 4S and on Simulator).

Apple's JS disables the animation for iOS 4.x and below, so maybe that's why it wasn't playing on your friend's iPod.

> Apple's JS disables the animation for iOS 4.x and below, so maybe that's why it wasn't playing on your friend's iPod.

I see Apple is really embracing the idea of "the one web" here, introducing web-fragmentation even among their own devices.

Apple has long stopped impressing me. Back in the days when they were the clear underdogs, they were for web-standards and a unified web, because they saw how that was good, and that was what allowed them to become what they are now.

Now it seems they don't really have so much ideological issues with how web-pages should be built anymore. Funny thing that.

There's nothing non-standards compliant about what they've done, as far as I can tell. Disabling the animation seems like an attempt at ensuring graceful degradation.

Graceful degradation shouldn't be based on user agent sniffing. That defeats the purpose.

I would argue that any reasonable technique that allows users of older browsers to access your site or use your app relatively normally is within the bounds of what we would describe as "graceful degradation." Being non-automatic doesn't make it any less valid.

that is only true for iPhone and iPod touch, you can have inline video on the iPad.

And in iAds on the phone...

Seems like a lot of work to workaround a drawback in their own browser.

Yeah it's weird. My web developer side is impressed by the technical detail that go behind it. But my other web developer side asks why the f*ck should achieving a simple video effect on a web page should be this hard in 2012?

There is a whole pattern of similar things like this in web that is killing me a little bit every day.

Want to center a div on the page? No worries just wrap it in these 5 divs and apply these 20 css properties and you are good to go. Oh btw, IE doesn't like that, you need this.

Honestly I think we developers should stop putting up with crap like this, and accepting the burden of broken technologies using hacks, etc...

If we become less tolerant of crap, there will be less crap. As long as we accept it by using workarounds, etc... there will be more shitty things to come.

CSS's flexible box model will eliminate that issue once it has wider support. There is a solution in the pipeline, it's just a matter of time.

Not the best link, but here's some more recent info: http://css-tricks.com/old-flexbox-and-new-flexbox/

What's difficult about centering a box? Was that just sarcasm? A couple of approaches:

Margin Auto w/ a fixed width

Absolute 50% w/ negative margins

Text-align center parent w/ inline-block display on children

If it was sarcasm, disregard my stupidity.

I think it was a hyperbole, but there really is a lot of dumb shit in web design.

That's still non-intuitive. Just do a search about centering (or X problem in CSS) and see the thousands of threads and questions and corner cases, hacks and exceptions.

The whole thing feels like you are stretching something beyond what it was intended to be used for.

Things like element positioning should be so so simple that if you search about it, you get nothing back because no one had a problem to ask for solutions.

Although I must say I'm not mocking CSS in particular, most web technologies feel this way because they were designed many years ago for a web that was supposed to have simple 'documents' like an article, text and paragraphs and today each component of it is expected to do a lot more.

It's not getting all that much better either because each new version and improvement is building on top of the previous decisions, etc...

  > most web technologies feel this way

Most technologies you don't know feel this way. And the reason—you don't know them.

  > It's not getting all that much better either because each
  > new version and improvement is building on top of the
  > previous decisions

In this case: no.

If that wasn't intuitive enough, feel free to email me and I'd be glad to teach/walk you through the many unfortunate ways to go about this: me@ryanglover.net.

Right. Now center it vertically. Also don't make any assumptions (like fixed height, or display: block) about container and the box to be centered.

http://cssdeck.com/labs/aqdw0l21/0

Container has fixed dimensions in my example, but it does not matter in this case.

removing them matters...

If container height is given by percentage you have to wrap it in another div with display: table. And if container is supposed to behave like inline-block you have to wrap it in yet another one. If you need to sprinkle around some margins and paddings (esp. with percentage height) you have to choose boilerplate divs to apply them to carefully.

Btw. your approach is exactly same that we had 10 years ago but with CSS table behavior faking rather than table tag.

Yay. Progress.

I think I am waiting for simple vertical alignment since CSS1.

It's like variables (yeah I know, LESS / SASS ...).

>Margin Auto w/ a fixed width Absolute 50% w/ negative margins Text-align center parent w/ inline-block display on children

Yes, totally intuitive and logical solutions all of them.

If you have w3c Stockholm syndrome, that is.

E.g "Just absolute 50% it with a negative margin" (oh, yes, and you would have to know the width beforehand).

>If it was sarcasm, disregard my stupidity.

It's not the stupidity, is the willingness to put up with BS convoluted inelegant workarounds for lack of basic, bread and butter, features.

If you value elegant APIs, KISS and DSLs, you should run screaming in the opposite direction in all those three, horrible, horrible, techniques.

W3C first came up with their (broken) model, and then they kludged solutions to use cases with it, instead of starting from the use cases first. That's why we have had the tables before and the need for the flexbox and grid layout now.

It's not the stupidity, is the willingness to put up with BS convoluted inelegant workarounds for lack of basic, bread and butter, features.

You're right, there is a willingness to put up with it. Mainly because I understand how these solutions work, how to debug them, and make everything nice and peachy. Do I wish things were better? Of course! Who wouldn't?

Well, we're in the same boat then.

Centering a box horizontally is simple. The original comment was referring to centering an element _vertically_, which is a much greater PITA.

Android's XML layouts feel a lot like a non-braindead web. There are a lot of similarities really, but simple things like vertically centering an element are as easy as they should be.

Text scaling can be a PITA though.

Same with XAML/WPF on Windows platforms which is very refreshing if you've been building for the web for years.

Yeah I've dabbled just a bit with these and they seem like a nice way to build UIs, also making data binding a lot cleaner than on the web or in the various mobile SDKs.

Honestly I think Microsoft has the best developer toolkit in the business right now but they haven't yet had the product design prowess to put that technology in the consumer's hands in any significant numbers.

May I present you the flexbox model!

http://www.w3.org/TR/css3-flexbox/

http://www.html5rocks.com/en/tutorials/flexbox/quick/

If you need IE8+ support, there is also display:table and family.

Anybody knows, why can't we use GIF instead?

You probably could, but given that GIF files are limited to a 256-colour palette, it would probably look pretty horrible. Especially for such a large image.

Also you would end up having varying frame rates in different browsers.

> Apple's website needs to work on all major browsers

That doesn't seem true. The second-most-prominent links on the iPhone and iPad sites are to videos that only play in Quicktime. If they wanted the site to work everywhere, they'd use a format those browsers can natively play.

The videos linked to from the iPhone and iPad pages, then they're video tags these days. If you're talking about the movie trailers or whatnot, then Apple isn't trying to sell you Quicktime, but it's damn well trying to sell you an iPhone or iPad.

If you're talking about the movie trailers or whatnot, then Apple isn't trying to sell you Quicktime, but it's damn well trying to sell you an iPhone or iPad.

Using a technology you probably already have to own an iPad or an iPhone to have installed no less.

I think Apple's aim with these videos is for them to be a seamless, ambient way for the page to come alive (sorry if that sounds cheesy). Asking the user to install a plugin defeats the purpose here.

The keynote and Johnny Ive video are a little different because it's targeted towards people who are already Apple enthusiasts. They're much more likely to already have Safari or QuickTime installed (or at east be willing to install it).

Still, it does strike me as kind of odd that Apple is working so hard to make one video accessible while being pretty lazy with another one (they don't even bother to make it accessible to Chrome on Windows, which does have h.264 support, unless QuickTime is installed). Maybe those two parts of the site were prepared by different teams.

[EDIT]: Actually, I just tried again in Chome and IE 9 on Windows 7, and it works fine (without QuickTime installed). Maybe Apple fixed it?

Apple uses a bog standard <video>-tag with a h.264 encoded video as its source. Every browser that can play that video (Internet Explorer, Chrome, Safari) will be able to play that video. If your browser can’t (Firefox), then it won’t be able to play that video and Apple is so nice and offers a sensible fallback.

QuickTime is never required.

This is the first link on the iPhone page:

http://www.apple.com/apple-events/september-2012/

There is no <video> tag, there is a "Get Quicktime" button, no video plays in my Chrome browser, and here's a screenshot of the network tab showing it downloaded a .MOV file with content-type movie/quicktime, not h.264: http://i.imgur.com/h8dO8.png

That’s HTTP live streaming. Chrome doesn’t support that, so it gets the fallback. QuickTime is never required.

> QuickTime is never required.

How do you propose I watch this QuickTime video without QuickTime? Have you forgotten the context of this conversation, about the purported desire to support all major browsers?

It’s not a QuickTime video. That’s my main point. I disagree that Apple’s goal here is to make it work with all major browsers in all situations.

Sublime Text was already doing the same with the big animation of its home page: http://www.sublimetext.com/ (you can actually see three animations there, depending if you use Windows, Linux or Mac).

What I'm really curious about is how those images are being generated. Is there a tool already available for that?

It doesn't look like Sublime is using the same technique as Apple, but it does look really interesting. It looks like their system allows for better reuse of images, and it is a lot simpler too. It might be less optimal though for frames with a large number of small differences, since they just use plaintext encoding for that stuff.

Sublime's example does highlight a solid usecase for this technique that Apple's example doesn't really emphasize: <canvas> + JS video compression is still the only way to do this stuff losslessly, which is pretty important for what Sublime is doing.

http://www.sublimetext.com/forum/viewtopic.php?f=2&t=816...

http://news.ycombinator.com/item?id=4532146

This whole implementation could just have been someone's weekend hobby that made it to production. Remember how Apple got its intel based macs? http://qr.ae/8eDNG

Just want to mention it before we blurt out the obvious over-engineering argument. ;)

Speaking of Apple website updates, I am surprised that they have never offered a responsive design for their apple.com site where you buy iPhones and iPads. Every app they make is designed differently for the phone or the iPad, but not the website.

Since they have an Apple Store App that works better than the website, why would they do that?

You're 100% right, of course, it is in apple's best interest to have you use their app.

It's just that downloading a bulky app for something you'll use infrequently is a crappy way to to business.

The App does more than just let you buy product. You can reserve for pickup, use EasyPay, set appointments at the Genius bar, etc.

That said, perhaps they could integrate the two a bit better.

Because not everyone who wants to buy from the Apple store has a iWhatever...

It's ludicrously shortsighted, you'd actually think they'd want to make it easy for people with competitor products to buy from Apple not harder!

Well, the main reason is because apple.com has more than just a store. The Apple Store App doesn't have any info about unreleased stuff for instance.

Yeah it's surprising they don't have a mobile version of the site. (Side note, I don't see what responsive design has to do with it -- why does the technical implementation matter?) But then it occurred to me that maybe they leave it as the desktop version to emphasize how well iOS is able to view the "real" web.

They may not have a responsive site because they want to "show off" that the iPad and iPhone browsers are powerful enough to display fancy full-size sites, along with encouraging first-time users to pinch or double tap to zoom.

The navigation looks terrible on an iPhone. It's a remarkably bad design for mobile.

If you visit the site on iOS, it will prompt you to download the Apple Store app.

It's a shame webkit never implemented the Animated PNG extensions and that no browser supports MNG:

http://en.wikipedia.org/wiki/APNG http://caniuse.com/apng

This would probably some of what people want here.

Maybe they should have used flash instead. That would have given them a higher percentage of successful deliveries than this solution.

Wow, took like 200 comments for someone to mention Flash, the thing that's been doing this exact same effect in a 20kb file since the year 2000 without having to write your own codec. Shows how well the Apple anti-Flash campaign has brainwashed the crowd.

If someone is interested in creating animations in this way, I created a simple script which converts videos in PNG/JSON/canvas:

http://blog.webmatters.it/2012/09/canvas-rocks-video-sucks/

Very interesting. Looks like some oldschool video game developers got into Apple webdev team.

About the last paragraph, we (Adinpsz) tried the PNG compression technique for JS demomaking (actually it's even a self-loading PNG-HTML ;)). You can learn more here with the JsExe tool: http://creativejs.com/2012/06/jsexe-javascript-compressor/ And see it in action here: http://pouet.net/prod.php?which=59071 (http://adinpsz.org/online/fabrik/)

But HTTP compression should be better anyway for "real world" usecases?

Keep in mind that Apple outsources a lot of this front-end work to their agency, TBWA/Chiat/Day Media Arts Lab. As a services firm they have less incentive to use standards created by Apple, though I think they do really great work.

I was just about to comment asking whether Apple engineers actually did this. For sure this is some cool stuff, but I think it'd be a bummer to say "I work at Apple" and then in response to "oh cool, what do you work on?" say "I work on the iPhone -webpage-"

Why?

I worked at Blizzard. On the Blizzard web pages.

Sorry Dude. Apparently you're just not cool ;-)

I've always been impressed with Blizzard's web pages. It's the type of style I typically don't like, but it is done really well and appropriately. I often used it as an example in my classes to show how to do very graphical websites the right way.

So, good work! I think you're cool :)

The saddest job at Apple is working on the tries-very-hard-to-look-and-feel-like-iOS-but-doesn’t-really-manage iPod nano OS. The website – especially these pages – is pretty exciting in comparison.

I knew they did the TV commercials but this is the first I've heard that the Apple website is designed and built by anyone other than Apple's own marketing team. I find it really surprising and, to be honest, hard to believe. For one thing, there are often jobs listed in Apple marketing whose descriptions reference working on Apple.com.

Nice teardown. Knew they were using a <canvas> tag, but didn't dive deep enough see how they were using Base64 and decoding the frames. Thanks for writing this up.

Very interesting and kind of absurd. Why didn't they simply recreate the animation with moving sprites instead of taking the "video" approach?

Isn't this the same as Clipstream[1]? The company that developed it has filed for patent[2]. I wonder if Apple licensed the technology or came up with it themselves independently.

[1] http://news.ycombinator.com/item?id=4419345

[2] http://www.dsny.com/HTML5

After reading this article I still don't have a clear idea of how they did the animation. It seems pretty in-depth technically, but the overall execution is still a mystery.

For example, how does the _frames array of base64 data related to the two unlock_00X.jpg files? All it says is "The "unlock_manifest.json" file specifies how the updated parts are positioned."

The image data is read from the JPEG files in order as a series of 8x8 pixel blocks (in left-to-right then top-to-bottom order). Each instruction is basically "draw the next {{length}} blocks from the JPEG files to location {{location}} on the canvas".

Surprisingly, there is no mechanism to explicitly state which blocks to get from the JPEG. Instead, it always just uses the next unused blocks. (This means that blocks from the JPEG cannot be reused.)

Regarding the retina loupe effect demo this has been done a while back - using css and an image for the loupe edge. Picks up mouse and touch events. Even works on older browsers.

http://homepage.ntlworld.com/infinnerty/differences/attic/at...

This idea is also in use at http://www.sublimetext.com/ The demo is using this technique because streaming video would actually take more bandwidth in this case as only bits of the image are actually changing at one time/incremental changes. It works a treat!

Highly unlikely that a real video format would take more bandwidth if encoded properly. Video engineers have thought of this problem.

H.264 would emit "skip" macroblocks for everything except the area that was changing (probably near the cursor). The repetition of "skip" would be easily compressed.

I really like that Apple is willing to experiment and push the envelope on their corporate site. Very refreshing.

> You can see this in action on the Retina Macbook Pro "Features" page -- which loads about 5MB of JPEG images (using lots of separate HTTP requests) just for that 2 second effect.

I immediately thought of web1.0 sites of 1990-s, with lots of animated gifs floating around. Weird times.

I see a lot of comments describing this as new and unique. Hardly. Video compression by using image diffs and run-length encoding has been around since the 80s. In fact this is more-or-less the technique that Quicktime's first codec, "Cinepak", used.

OK, so this would be one of those instances where I support a headline rewrite for clarification. The iPhone connection is incidental and I expected something entirely different.

This was more along the lines of, "How Apple reinvented/over-engineered the animated GIF"

As a side note, publishing this on Google Docs is an interesting choice by the author - alleviates any need to worry about load from the piece becoming popular, but it almost makes it an anonymous article.

I was going to use Blogger, but multi-login bit me. I had already written the draft in Docs, so I just published that. (I also like how plain published Google Docs look)

I went ahead and added a really short byline though.

There could another simpler way. NoVnc + tightpng encoding. Create and record a high framerate VNC session to desired video. Tweak no vnc to receive data from local callbacks and replay the recorded data.

This is brilliant, maybe overpowered but brilliant. But the thing I cannot stand is the use of images for text. I would have preferred to se the use of @font-face with fallback to images for old browsers.

Man, I wish there was a library to help us normal folk do similar things. The ability to do small video on mobile would rock.

You know, that this isn't exactly a video... If you really need to play a small video like that... try using gifs. It's easier, and there is only two drawbacks : 1- The stumbling loading of frames. 2- You can't do the thing they do with the earphones, going back and forth in the "video". If you're fine with these two drawbacks... You probably don't need all that over-engineering.

Well, there is a really hackish way to do this, actually. You could preload the gif and then use something like http://slbkbs.org/jsgif/ to jump around the gif frames.

I doubt iOS is the reason they went through all this trouble. The performance even on the iPhone 4S is miserable.

Still none of the pages on apple.com are made for small mobile screens. Only exception is the iPhone manual.

Apple has a way of hiding new web tech in plain sight. Otherwise very good write-up. Thumbs up.

when you browse istore from an android device they added programmatic wiggle and jerk when you scroll the screenshots for an app. It's hilarious because android jerking wasn't that consistent. to make it authentic they needed to add randomness to it.

Great analysis, very interesting.

Obviously this is absurd overengineering.

And, despite the assumed total cross-browser compatibility claimed, I found that the video wouldn't play in any browser available to me, despite pinning the CPUs on my computer while just sitting there doing nothing.

this is so useless

In short, Apple don't want their demo videos to be easily downloaded and viewed?

Article claims the reason is so that the videos will play on iPhone. If Apple considers people who already own an iPhone most likely to view the page and buy the newer one, it makes sense that they'd go through a lot of work to show them animations they couldn't through the video tag.

Preventing download and viewing doesn't seem very important to me. Anyone could just use one of many screen recording apps to get good enough copies anyway if they wanted them for some reason, and who cares if they do have them.

Their more like very short animations than videos. Hardly worth downloading.

Reply on news.ycombinator.com