Understanding the iPhone 6 Plus Screen
oleb.net
oleb.net
Here's one from all the way back in October '13 that predicted the 1080p screen on the "5.7 inch" phone:
http://www.displaysearchblog.com/2013/10/in-2014-apple-will-...
I think they have made a conscious decision to bet on display scaling technology at high PPI's, so they have the flexibility to always buy the best displays they can get manufactured at the price point they need to hit.
In addition, it's not even clear yet that a sapphire screen on an iPhone (as distinct from a watch) would be an improvement, because sapphire apparently shatters more easily. Which is why Apple is reportedly working on a screen incorporating a layer of sapphire on top for more scratch resistance, since it's harder, bonded to a layer of glass below, for more durability.
But then you'd end up with a comically large UI at 6" with no space for any content because everything's so huge on the screen. The entire point of the larger phone is to have more content area.
If the entire point of this phone is to fit more on the screen it's doing a pretty bad job of it.
If anyone else did this they'd be pilloried for it being the total battery wasting font-destroying hack that it is, but Apple gets away with it.
This is true on the Macbook Retinas also - when I set mine to looks like 1680 or 1920, fonts get blurry due to this scaling, when if the software was better it'd be rendered at the native resolution and they would look as good as they do at the 2x resolution. (where the numbers work out that you can scale without a loss of fidelity)
Basically, I think they never got it working good enough and just fixed it to 1x and 2x.
Here's a screenshot from the utility they had for this. http://i.stack.imgur.com/0NToe.png
The underlying frameworks (e.g. CoreGraphics) are absolutely capable of rendering at arbitrary resolutions, and have been for a long time. It's not particularly hard: you interpose an affine transform which converts from local to device coordinates, apply it to all your geometry, and it falls out.
But "rendering at arbitrary resolutions" is not the same as "looks good at arbitrary resolutions." For example, a line 1 px wide at 1x will be 1.5 pixels wide at 1.5x. That means at least one edge must be aligned on a partial pixel, and there's the potential for weird antialiasing effects. There's also a behavior change: you cannot redraw half a pixel, so now "dirtying" a pixel requires redrawing more components at 1.5x than at 1x, which can cause performance problems and other bugs. And there's also a question of which pixels get the partial alignment: do you round in a direction? If so, which one? Or maybe you don't round and you have two pixels that are a quarter covered?
Which leads to the problem of centering. I wish to center a bitmap image within a button's border. The image is 101 logical points high, and the border is 200 logical points high. With a 2x scale factor, I can center absolutely, and still be aligned to device pixels. With a 1x, 1.25, 1.33, etc. scale factor, centering will align me on a partial pixel, which looks like crap. So I have to round. Which way? If the goal is "make it look good," then the answer is "whichever way looks good," which depends on the visual style of the bezel, i.e. whether the top or bottom has more visual weight. So now we need hinting.
And that's where things start to get really nasty. In order to make things look good at arbitrary resolutions, we want to round to device pixels. But the rounding direction is not a local question! Consider what happens if we have two visual elements abutting in logical coordinates, and they round in opposite directions: now there's a device pixel between them. That's very visible: you get a pixel crack! So you have to coordinate rounding.
WPF is a good example of a framework that attempted resolution independence and encountered this problem. Initially it has the "SnapsToDevicePixels" property, which triggers rounding behavior at draw time. But draw time is too late, because of the "abutting elements rounding in opposite directions" problem. So they introduced the "UseLayoutRounding" property, which does...something. And the guidance is basically "turn it on and see if it helps, if not, disable it." Great.
The web also has this problem in spades. Websites break in all sorts of fun ways when you zoom in or out. We tolerate this because, frankly, the bar is super-low for websites.
As I see it, the two options are:
1. Make everything vectors. You'll have to choose between weird antialiasing artifacts and potential pixel cracks; either way things will look bad. And you'll encounter bitmaps eventually, and have to deal with the necessities of resampling and pixel aligning at that point. 2. Scale only to integral sizes, and resample. You'll avoid antialiasing and pixel alignment issues, but pay a performance penalty, and things may look slightly blurry.
So which option is better? #1 has the potential for the highest-quality output, but at a significant price: developers must test their apps at more scale factors, and the failure mode is ugly drawing artifacts or outright bugs. #2 is more utilitarian: the output is not as nice, and you incur a perf penalty, but that's borne by the system instead of the apps, and the overall system is more consistent. #2 is also more forward looking: if you expect that pixel densities will continue to increase, then resampling artifacts will eventually be indiscernible, but a pixel crack will always be visible.
Apple took the practical and forward-looking approach to this problem. I can't fault them: to my knowledge, nobody has successfully implemented true resolution independence in a framework with wide adoption. If they have, I'd love to know how!
As far as I can tell, all the problems you describe boil down to "In order to make things look good at arbitrary resolutions, we want to round to device pixels."
As TFA shows, with iPhone 6+, you can't. Round to device pixels, that is. I wrote about this when the 6+ came out[1], it's good to see it confirmed empirically (awesome job by Ole, incidentally!).
So there no longer is a choice between your options 1 and 2, because #2 is out the window, no integral scaling available.
Of course, to make #1 work properly, what they need to do is remove all the places where snapping to device pixels is done, because otherwise you get pixel cracks, and those are noticeable.
The anti-aliasing effects are apparently theoretical at this point: yes you can create test cases to show them, but in practice the resolution is high enough that it no longer matters. From TFA:
"I am actually surprised how little of an issue the automatic downsampling is in practice. As I mentioned, I simply don’t notice anything of the effects I have illustrated here in real life."
Or as Jon Gruber put it: "Its 401 PPI display is the first display I’ve ever used on which, no matter how close I hold it to my eyes, I can’t perceive the pixels. "[2]
I do think it makes more sense to do the scaling directly in CoreGraphics, disabling all the features that make snapping to the pixel grid possible and letting the pixels fall where they may. It would certainly be less work overall and use less memory. I can only speculate as to why it wasn't implemented this way, my guess would be that getting all the pixel-snapping out is not that easy, and possibly it was a rush job (as suggested elsewhere) due to last minute unavailability of the higher resolution display.
In the future, I would expect either (a) an actual 3x display or (b) the display stack adapted to your (1) choice.
[1] http://blog.metaobject.com/2014/09/iphone-6-plus-and-end-of-...
IMHO this is what Apple should have done a long time ago, certainly on ios. Yes it has issues, but those issues are mostly faced by the OS vendor once, and have been successfully dealt with in the case of font rendering for example with antialiasing, hinting etc. You don't hear anyone decrying fonts being vector formats and asking to go back to pixel fonts nowadays, and resolutions are only going to increase.
The OS should be handling rendering, hinting, caching, raster representations based on vector data. As you point out this is a hard problem and Apple are uniquely qualified to handle it. Instead Apple have punted on this and developers are actively encouraged to produce raster representations of each asset which break for every new device which comes out (as you point out this doesn't just apply to fonts).
Controls like buttons or custom ones should be drawn in code, not included as raster assets, and the iconography for things like buttons, launch screens, branding or other app-unique assets could be stored as the original vector, as it should be. If you want to put a raster image on a button, well, sometimes that's not going to look good.
On top of the problems caused by resamping, what Apple have done instead of choosing the vector route is create an ever-growing headache for developers and themselves by focussing on raster assets and requiring pre-rendered bitmap reprentations of assets like icons, which usually start as vector anyway. How many different pre-rendered sizes of ios icon are we up to now, 20? Where does it end? This is not practical or forward looking, it's a hack which imposes real costs on developers, customers and Apple - there is a huge amount of work in creating and maintaining all the different launch screen, icon and other assets now required to be rendered at 3 or more different sizes for Apple devices, and this is bloating every single ios app with multiple megabytes of assets which could be generated on the fly from their vector representation, over the entire app store that alone is a huge waste of bandwidth for Apple and customers.
So there are lots of reasons to aim for real resolution independence, and from the point of view of developers, Apple chose a quick hack (which has devolved into a hacky mess over time), over a more difficult, forward-looking solution which anticipates that screen sizes and formats are going to multiply until it is impossible to cater to all of them with raster designs.
It’s not resolution independence, but it has the potential to make asset production much less painful. I don’t know if this approach also work for app icons or only for things like button and toolbar images.
Of course it'd be better if this was done at run time on the device, and bitmap representations cached as required (eventually they probably wouldn't be, even if initially they are).
http://oleb.net/blog/2014/08/replacing-launch-images-with-st...
I seem to recall Marco Arment took a bit of pride in having 100% procedurally generated graphics with overcast, so that he didn't have to create/ship any bitmapped images whatsoever.
The tools are there by Apple, but I don't think the hardware until fairly recently could really justify the performance hit - instead choosing file size over performance. These days though, in the majority of cases, the performance hit is negligible (especially when compared to dev/design time of generating @3x assets years/months later)
That's interesting, I might try that for launch assets, though it can't be used for app icons, I'd love to just drop in an svg and let the OS scale my icon.
Unfortunately the default path (including many default assets like icons and launch screens) is to produce tens of raster files generated from the original vectors and bundle them with the app which is wasteful, painful and prone to break with new hardware.
The pixels of the screen are now so small they have almost disappeared, so I think there's an argument we have reached the place where going mostly vector makes sense.
Clearly some assets will always be bitmaps (photos), but so many others are not well represented as bitmaps (icons, text etc) - content is fine as bitmaps but controls and chrome are usually better as vector.
Currently as a windows programmer I have not used raster images in years.
People often say "just use SVG/PDF/vectors for all the things" but look online and you'll find many good arguments against it, not least because icons and text at lower pixel sizes really needs to be "pixeled" manually to look good.
Since font technology is so old and so efficiently implemented, the obvious thing to do is to extend the same technology to rendering color fonts AND color vector artwork, such as icons. But Apple is going the wrong way on that too: they added emoji support to their fonts by embedding PNG images. Yuck. This article has links which explain more about the MS proposal, which is much better: https://en.wikipedia.org/wiki/OpenType#Color The glyphs can be built with layers, each of which has a color and a vector path. This is what we need. And then we need the Khoros Group, and then the GPU vendors, to focus on accelerating exactly the kinds of vector paths which are found in fonts. This will give us a better solution for rendering fonts and also a way of rendering resolution-independent icons and other types of application artwork at the same time. And the font format will constrain the icon designers to create icons that are efficient for the GPU to render: as few layers as possible, and without all the SVG features that make rendering so difficult.
In the meantime, it's a nice coincidence that the vogue of the moment is going towards monochrome icons. That means that the use of icon fonts (such as Font Awesome, and others) can go on for a few years even without adding color. But we can depend on there being a backlash eventually: we have color screens and are using them to show monochrome UIs? So the technology needs to be ready by the time the designers get back around to that way of thinking again.
Agreed it's a hard problem and that comment was certainly reductionist, but at the same time it's depressing to have this be the issue that makes fonts blurry - the hardware is fine but the software side is letting it down.
Of course extra space can absorbed into the screen size instead of its density. Android and iOS support this via their respective mechanisms, like autoresizing masks or autolayout on iOS, and I guess layout managers on Android.
If you were to make a new device with unusual screen metrics, you have a few options for app compatibility. You could report an unusual density, and risk ugly antialiasing or pixel cracks. Alternatively, you could report an unusual size, and get bad layouts, where lots of apps will have UI elements overly small, and excessive unused space. Or you could compromise: report sane (but inaccurate) sizes and densities, and resample. This maximizes compatibility, with some quality and performance cost.
(The idealized option is true resolution independence, where apps look great at any density and size. To my knowledge, nobody has achieved that.)
I figure Apple realized that the iPhone 6+ would be very heavily reviewed, and so wanted to ensure that existing apps would work really well on it. That's what their scaling approach enables.
But you wouldn't actually do that apart from in rare circumstances - instead, you'd create a View, size it to 100dip, and give it to a layout telling it to position it in say the center of the screen. When it's displayed, the layout manager will decide how big to actually draw it, taking the density into account.
Android doesn't have the issues with pixel cracks etc you are talking about, since the views are all stored in Layout Managers which operate at the native device resolution and calculate the final positioning. So they can see that this element would be 10.5px, display it at 10px, and move the next element up by 1px so everything works out.
For example, say we have 10 views, alternating black and white, each 10 dips wide, packed abutting into a container of width 100 dips. At 1.33x scale, each view wishes to have 13.3 device pixels. How do Layout Managers solve this?
1. Round each view down to 13 pixels, and repeatedly "move the next element up" by 1 px? Then you must account for .3 * 10 = 3 unclaimed pixels at the end of your container. That's going to be quite visible.
2. Round views based on their fractional pixel position, i.e. seven views down to 13 pixels, and three views up to 14? Now your stripes have different widths, which can be visible (especially if they have high-frequency content), and can also cause a "creeping" effect when they animate, if their edges are rounded independently.
3. Don't round at all, and position views on fractional pixel boundaries. This yields fuzzy edges.
Honest question - how would layout managers position views in this scenario?
So in that example there would indeed be 3 unclaimed pixels at the end of the container.
For things like hairline borders you typically just specify that they are however many pixels wide. It's perfectly valid to say width=100dip, height=1px. You can mix & match at will.
For the scenario where you just want to have a bunch of elements all fit on one row you usually just give them all the same weight and let the parent view divide up the space rather than saying parent=100, children each =10.
Wait, have you actually seen blurry fonts on the iPhone 6 Plus?
Do you know of any photograph which demonstrates them?
There are also much older implementations: Postscript has "only" 595 x 842 "logical" pixels for a whole A4 page but it's just a first step in working with any resolution of the printer.
And NeXT started with http://en.wikipedia.org/wiki/Display_PostScript and Apple has http://en.wikipedia.org/wiki/Quartz_2D
To competently argue the topic it's good to understand more than "this number is different than that number."
True, BUT you still have that problem on the 6+ with everything being scaled by 15%. In fact, the problem is vastly worse. Instead of the occasional line being 2px instead of 3px, you now have every line being blurry. And you can solve the 2px vs. 3px line thing quite easily by letting devs choose between specifying in pixels vs. points, which is what Android does. The problems you talk about are all problems that only exist if you exclusively work in points. They all go away when you can also use pixels.
And the density is not at all high enough to mask this. It's very visible as soon as you pick up the device, and the dividing lines in things like the calculator app are hilariously uneven and awful.
(1) Leave it to developers to make sure that their code correctly handles different DPIs. This results in every app having to produce its own solution for some very difficult problems, such as those ridiculous_fish mentions elsewhere in the thread, or else look terrible on some devices. (Yes, some of this can be encapsulated in libraries, but you'll virtually always do some custom drawing that requires manual handling.)
(2) Virtualize the pixels, so that depending on the DPI of the screen you're displaying on, a "pixel" does not necessarily correspond to a single device pixel.
Generally what happens is that you try #1, realize that many or most developers screwed it up, and shift to #2.
See CSS "pixels" for an example of this phenomenon.
(3) Expose both pixels and dips. Both are first-class units in Android. The 2D drawing API works exclusively in pixels, but the layout commonly works in DIPs instead.
Developers then do their layouts in dips but custom 2D drawing in pixels. Or really most anything in code is using pixels, dips are largely contained in the XML layout files.
Except we're talking about a phone that sits at or near the top of every battery life shoot out chart. And the pickiest screen-specific shootouts fail to mention any difference in font detail between iPhone models. Some hacks work.
EDIT: It also sounds like it was originally going to be the correct resolution, they just couldn't get the yields. If that's the case, perhaps they made it downscale at the last minute and used a 1080p panel?
I strongly assume that the lower resolution was indeed Plan B, and that it's the result of another twist in the saga of Apple's relationships with the mobile component suppliers.
On a not-really-unrelated note, the Samsung Galaxy Note 4's screen is apparently pretty stellar in all respects, not just resolution: http://www.displaymate.com/Galaxy_Note4_ShootOut_1.htm .
... You can easily connect your Macbook Retina to an external monitor of any resolution. OSX is not limited to your Macbook Retina resolution.
On the iPhone6+ Apple could easily draw everything in 2.6x resolution to match the display, but I guess they feel using 3x+downsampling simplifies the UI-design process. From a technical point of view it is an unnecessary hack.
I want the resolution on my Macbook to always be 2880x1800. But I want to be able to make things bigger or smaller, while still rendering at that resolution so things don't have to be resampled. (and in the case of the iPhone 6+, the default/only setting requires this resampling).
The problem is that when I set it to 'looks like 1680', rather than just rendering things natively at 2880x1800 but a little bit smaller, we have to get into all these scaling hacks (rendering at 3360 x 2100 and then scaling down to 2880x1800), which means you lose the accurate font rendering and clarity you got, everything has a slight blur to it because it's been scaled down.
What would be the downside? Larger UI elements and less available space on the screen? Could a @2x instead of @3x mode work or would that result in super tiny "bad hidpi" UI?
[0] http://www.idownloadblog.com/2014/11/07/littlebrother-iphone... [1] http://www.idownloadblog.com/2014/11/06/upscale-change-resol...
I wonder if this whole downsampling business is happening because the 6+ didn't get a "full resolution" screen in time and so they had to go with off-the-shelf 1080p screens? Related to the GT scandal maybe?
Hasn't iOS had tools for building resolution-independent apps ever since the iPhone 5 was released?