The Care and Feeding of the Android GPU
satine.org
satine.org
In this case, it's obvious that we've got server people working on mobile code. Future hardware will resolve these inefficiencies? UGH. That's definitely someone from the server world speaking.
Palm's WebOS put much of its core logic in JavaScript, which had a slower implementation (at the time) than Dalvik VM does. However, Palm carefully made sure to take advantage of the GPU to achieve many of the effects in the UI.
I think people that write articles about touch responsiveness or animation latency are suffering from the "benchmarking hello world" problem. Yeah, you could speed that stuff up. But 99% of the time, the phone is waiting for data on a socket, which is not going to get fast any time soon.
As phones got faster, the situation improved, until a larger screen was added, which brought things back down again as the CPU burns to keep up. The milliseconds latency here and the stuttering FPS there decreases the usability significantly in my opinion.
I think that there is a group of users who don't notice these latency and animation issues. But I do, and I think many users still unconsciously attribute their dislike for Android to these issues. (example: my brother in law, used several Android Froyo phones extensively before settling on a BlackBerry because it felt "faster".)
My EVO 4G gets about 24 hours battery life, with moderate usage during the day. (But again, this is because I don't use HTC's build. With their build, the battery life is barely long enough to boot the phone.)
Responsiveness and framerate-wise, even a simple cursory examination will tell you: WP7 and iOS run laps around poor old Android. I got to play with a Nexus S recently, and the responsiveness is greatly improved over Froyo on N1... but is still noticeably less smooth than either Apple or MS's platforms.
It's an odd day when MS's UX is more polished than somebody else's...
One area where Android's lacklustre performance really hurts usability is diving into menus. iPhone came up with the novel idea of sliding menus - a simple animation that communicates a lot of context to the user (A belongs under B triggers modal window C, all based solely on the direction of the swipe animation). On Froyo+N1 this animation is jerky enough, frequently enough, that this relationship can easily be missed, resulting in more user confusion and worse UI comprehension overall.
Google needs to realize that we are in the year 2011, not 1995. "UI design" is no longer about placing the right buttons in the right places, and animation is often now the defining line between an easily comprehensible user experience and an obtuse one. IMHO graphics/animation performance is the #1 obstacle in Android's way, and the main advantage everyone else seems to have on it (and it's a huge advantage).
When I use a 2.2 phone, my impression is one of annoyance. I don't know if my touches will be detected, and even if they are, I don't know how long I'll have to wait for the phone to respond. Heck, I enjoy using WP7 phones more than Android because of the performance, despite the fact that WP7 is significantly less featureful than Android! Navigating WP7 is a pleasure, and it never feels like the phone stands in your way. I have yet to get this feeling from Android.
Also, is it really fair to compare the N1, a phone that's over a year old, to brand-new WP7 phones?
Google has done a lot of cool stuff for developers in the last few iterations of Android. The NDK has evolved into something quite decent, definitely good enough for writing games now. But responsiveness is the major user experience issue that they've completely failed to address.
I don't have any problems with responsiveness or battery life.
As for battery life, it depends entirely on where you live. I experimented on it for months before finally figuring out that well over 50% of the power draw on the system is the data network (mobile network option when you hold down power). If you have it on when you have really weak/no reception, the phone will burn through the full battery in about 4 hours and it's pretty simple to lose 20% battery in 15 minutes. If you always have a decent connection, your battery life will be fine, if you happen to live in a place where you have holes in your coverage (e.g. NYC urban canyons), battery life is terrible.
After realizing that, I set up a set of Locale rules to shut down the network when I'm on the wireless network at home/work and to shut it down when I lose signal. Combined with the undervolted kernel, I normally get 48 hours on a charge for my normal use (~350MB/mo).
One thing I did notice right away was that the console felt different. It felt like it was maybe more responsive somehow, and not just when it was scrolling a wall of text. It even felt different when I was typing. Somehow, typing at a console (not in X, not via SSH) felt better in FreeBSD than it did in Linux. I couldn't tell you why, and I didn't really understand it myself. It wasn't something I could measure, or even describe, but it left me feeling like FreeBSD was more than 'just like Linux'.
I only used it for a day or so before I realized that I'd screwed up the partitioning and wiped out my Windows partition, and so had to re-reformat, but it left a profound impression.
I feel like iOS is the same way. All the Android phones I've tried (including a friend's Nexus One) have been, theoretically, the same as an iPhone in terms of features and capabilities and even hardware, but even before I got into using them in depth there was just something that felt a little off about tapping on icons or scrolling lists, and I could never put my finger on it, so to speak. Perhaps it's just design decisions or usage patterns that I'd internalized already, but I've always felt like it was something more. Maybe the scrolling behaviour is something I picked up on unconsciously, because while I don't remember it being slow or jerky, I do remember it didn't feel right.
Perhaps the numbers-oriented engineers at Google can't quantify it and so don't prioritize it, or maybe they just don't think it's worth wasting time on, but I think consumers, in the end, will notice. The real question is how many will actually care, and how many will shrug it off because of the benefits (perceived or real) that Android provides them over the iPhone?
Does anyone have a good explanation for this?
I had a similar perception with Solaris on SPARC. It was slower overall but more responsive when interacting on the shell. Sometimes I had high load (10 or more) and did not notice when using a shell in a terminal.
I think Linux got much better since then. I don't feel the difference on current FreeBSD, Solaris or Linux installations.
Thats one of the best descriptions of iOs scrolling I have ever read. It's exactly right.
You specifically state "People who use iPhones" and many others seem to think that the problem is only apparent if you compare it side-by-side. Which suggests to me that it's a fairly marginal concern, particularly compared with other Android flaws that the same people could be working on.
I switched from iPhone (to one of the lowest-end Android phones on the market) and didn't notice this particular issue. I did have general confusion at where things were and how to do things, but by the same token two weeks later my iPhone felt like it was missing a back button.
With subtle effects like this it's easy to sound like audiophiles talking about mp3 encoding. I'd suggest that if you're expecting the hoi polloi to pick up on these subtleties (which may well be imagined or exagerrated in some cases) and carry this fight for you then, like the audiophiles, you're going to be disappointed.
many many webapps do not use the network, and any latency experienced comes from the page's own processing, the browser & the os.
Innovation doesnt come from asking people what they want. It comes from finding new markets and making new means.
Never ocurred to me that GPU would help. I also come from a server programming background and hearing that GPU acceleration on the UI could help is like breathing fresh air.
There's several other more complex things you can deal with if you're writing apps that bring some actual value to the user, other than doing stupid web clients on android. It's not all waiting for data on a socket.
I'm not saying things aren't incredibly responsive, but some of the animation techniques are used brilliantly to disguise lag.
Three examples:
1. Mobile Safari doesn't wait to render what it hasn't downloaded/formated before responding to touch gestures. You swipe and the page scrolls. It might display a checkerboard before it catches up with rendering, but the swipe command takes precendent.
2. Those wizzy transitions when double-tapping the home screen, or returning to the spring-board. They start immediately and actually mask what's going on behind the scenes with regard to processing a request. They're the ultimate illusionist progress-bar. And they need to be tuned finely to present the illusion or else it becomes obvious that they're masking tactics.
When people complain about the performance of new iOS versions on older devices such as the 3G, what they're really noticing is the illusion has been broken as Apple is optimising and fine-tuning the illusions for the latest hardware.
3. Those screengrabbed snapshots of apps that have been completely closed and display when an app is restarted. If the load time is short enough, the illusion of instantly re-entering an app is maintained, even if all you're seeing is a stale image of the app your closed earlier.
I'm not sure if Android employ any of these tricks, but they demonstrate the focus and undocumented nature of Apple's drive to create the illusion of perceived GUI 'smoothness'.
Not only that, but I'm convinced that users are actually more efficient when perceived responsiveness is added: for instance, most users will not start thinking about what to do next on an app until they are seeing the app interface; presenting then with a non-interacting screenshot as soon as possible actually makes the overall interaction more efficient. But the ultimate is what the guys making PCalc discovered ( http://www.dragthing.com/blog/2009/07/how-to-make-your-iphon... ): you can put a non-interactive screengrab in such a way that even though it is a dumb image, touches will actually be recorded and have their effect as soon as the actual interface is set up; users can actually interact very soon. Now that's badass.
(however, most animation stuff doesn't fall in the category of perceived responsiveness, as most often, for instance for transitions, the end result must be ready before the animation begins; also, "undocumented nature"? Apple's obsession with perceived responsiveness is well-known).
(You are right about the other things you objected to in my comment.)
the two tickets, "Support for external keyboard, mouse" and "Browser doesn't render SVG images" both strike me as absolutely essential to making smartphones competitive with PC's. people have been happily attaching fold up keyboards to Palm Pilots for a decade now, SVG is perhaps the only sane route forwards for resolution independent apps and animation, and Apple has found religion on both these topics-- yet Android is obstinately behind.
otoh composited gpu acceleration is not nearly the huge obvious win C. Ying boasts it as. the gpu is more power efficient than the cpu for intensive applications, but a smartphones natural state is not intensive use. mr guy of android points out a number of complexities-- opengl drivers only supporting a single context, opengl acceleration proving less speedy than software implementations. it places huge emphasis on driver support, which any embedded wonk will tell you is a scary prospect.
although gpu acceleration is becoming better supported, keep in mind that the smartphone market expands, both up and down, with broadcom & others putting out chips designed for sub $75 smartphones. android is definitely not winning the race for the top atm, but performance has always been a rather lonely market-- that said, unsmooth experiences dont fly. software graphics focus insures a base level of performance for everyone, and strikes me as the right place to start to insure a speedy experience not subject to external interrupts.
the current silkiness situation is a little sad. it'll get better. more relevant to me is Android not playing in the ecosystem-- external peripherials, UPnP/DLNA, SVG-- these are the real things that will make or break not Android, but the open digital web, and Android is failing, badly, even in compare with non-web-enthusiasts Apple.
I'm not saying the GPU is the answer, and maybe 2.3 is the answer, but ignoring the problem is not.
(Edit: So why does a GC pause stop animation, and not just by taking up CPU time, since moving to the GPU would help in that case? Is the animation code itself written in Java and waiting on the GC? If so, I bet rewriting it in C would be a massive help. Also, yuck.)
The argument in favour of a GPU-accellerated GUI which is not mentioned here is battery life. I haven't been testing this side of things, but just because a CPU can do a 60 FPS GUI doesn't mean it should. Deferring it to the GPU would certainly reduce power consumption while navigating a UI. Whether this is non-negligible in terms of overall battery life, I'm not sure.
The ~40ms input lag shown by most desktop IPS LCD monitors is noticeable even though it falls below the reaction time. On a touch display, it should be even easier to notice input lag, because you can see your finger moving and the screen around it not keeping up. 10ms sensor lag may be below the threshold of perception for circumstances like this, but probably not by much. If you add even a few milliseconds of processing lag after your app receives the touch event, you'll be behind by a frame.
As for touch responsiveness, I don't have enough current knowledge of Android's situation and touch hardware in general to write on that topic.
Both Charles Ying of Satine and wzdd seem sympathetic to this notion that the gpu is a power saving device. Surely it is when doing complex work. On the other hand, most smartphone usage consists of relatively simple screens and basic if any transitions. If it's only going to take 3ms of CPU work to run a 150ms animation, by all means, figure out how to keep the OS from interrupting the high priority graphics task and make the CPU do it. Particularly if the GPU would have to be on across the entire 150ms. If not, what is the cost of copying the graphical content between the GPU and the CPU? How much CPU does it take to initialize and set up the GPU for this extremely simple task you ask of it? Is the CPU going to be able to sleep while the GPU is running the animation, and if not what kind of context switching and control costs are going to be imposed on the CPU to manage the graphics state?
We've barely begun to see multi-core cell phone GPU's, but the point of the GPU is basically to run lots of work at a nice low clock rate where voltages can be dropped. If you dont have a lot of work to do, there's really no sense waking up the GPU and having it's extremely dumb poorly-branching cores chug away at figuring out, say, the SVG animations that a semi-competent ARM core could knock off right quick. You're just wasting power turning on the GPU.
Smoothness and polish, I suspect, are much more a questions of resource allocation than CPU capability, particularly when you've got a sub 640x480 surface and a 1GHz core.
Developer: No you don't. I have the benchmarks to prove it.
Very recent kernels have a new scheduler which supports a new scheduling algorithm which people are anicdotally reporting better interactivity with. Perhaps this will help the situation.
Basically: 1) Android phones can already animate the screen @ 60 fps in software in most cases. 2) When they can't, it's not because the CPU is too slow, it's because there is other stuff happening (GC, etc.). A GPU can't fix that.
Also the widget for editing the web page URL on the Tab was a horrible usability car crash. What's up with that? Disclosure - iPhone user, but Android curious.