How iOS7 Stacks Up : Smartphone OS User Experience Shootout [pdf] | Hacker News Reader

How iOS7 Stacks Up : Smartphone OS User Experience Shootout [pdf]

pfeifferreport.com

25 pointshokkos38 comments

There is a really interesting comment on reddit regarding this survey, specifically about how dodgy its methods was.

"This is not me saying that this is a bad study because I'm a Microsoft fanboy. This is me saying this is a bad study because I'm a scientist.

The "study" says that "cognitive load is the sum of elements a user needs to be familiar with in order to use the operating system." That's actually a good thing to look at, and would be interesting. Unfortunately, they counted the number of pre-installed applications and decided that was somehow linked to the cognitive load? I don't even begin to see a connection between the two.

In the "efficiency and integration" section, they do not actually run a study to see how quickly different tasks can be done (which would be efficiency) -and I don't see anything about integration at all- instead they simply talk about how many short cuts there are- except on Android, which is docked for having too many short cuts.

The customization section showcases 2 problems. 1. They assume that more customization is always better and 2. If what you're going for is customization, than you better make Android a 10 and the next highest operating system a 2. The fact that they have Android and iOS7 anywhere close together shows they did not use any methodology for assigning score.

And finally, the UFX score. I don't even know where to begin. First, he claims that you can close applications on iOS7 and Android to "free up resources," but in reality both camps have said explicitly that their app launcher does not control resource use. He claims that iOS 7 can fit 24 application shortcuts, while Windows Phone can barely fit 12. In reality, if all you want is icons with counters (which is all iOS 7 offers you) then you can fit 28 icons on Windows Phone. He also complains about not having a background image on Windows Phone, never once mentioning how you wouldn't be able to see it (he also lists this under UFX- not sure how this causes any friction. Anyway, he also list this under customization (which makes more sense) and double counts this as a negative). And the oddest one- he claims that the Lumia 920 has "sponsored links on the homescreen." I don't have a Lumia 920- so I can't speak from personal experience, but none of my Google searches reveal anything about this and it is something that I am quite sure would be railed against frequently if true. So in that case, I'm pretty sure it is a lie.

Now, that being said- there are of course problems with Windows Phone. However, I really don't think this article captured any of them." - Weed_O_Whirler

Strange how trying to make something objective can make it inaccurate. For example, I deliberately changed from android to windows phone because I much prefer the windows phone user experience, but it scores the worst in their objective comparison.

All the complaints about windows phone are mostly accurate, but largely irrelevant for me personally. For example, I don't want a background image for the live tiles screen, so I don't care that I can't configure one.

I do agree with their final conclusion that windows phone is a strange mixture of careful user-centric design and complete disregard for actual user needs. There are things which are incomprehensible that they haven't been fixed over the past few years, like not having a separate ringer volume. If, like me, you are lucky to have needs which mostly align with the intended design, then it beats even iOS in usability. But if you are only slightly out of alignment, it is a long fall down the usability ladder.

I only just picked up a Lumia 1020 two days ago after using an iPhone for the last 2 years, so the jury is still out for me on whether I prefer it or not and how well it suits my particular set of needs. I'm not even sure I can accurately define what my particular needs are right now (except that I wanted the 1020's great camera). I wonder if you could elaborate on what needs it is that you think Windows Phone suits so well?

No you can't compare iOS 7 to android until you install my bespoke launcher and remove all the default icons and widgets and customize the phone!

It's not fair picking both up out of a box and comparing the user experience.

You're joking, don't you?

Yes but unfortunately there are people making that argument quite seriously.

When one of the negative points is the number of icons and widgets it says a lot about that report.

Like what? Things that clutter the user experience are certainly valid points. Otherwise we might suggest that the old godaddy website was well designed because it had lots of "features."

Number of apps installed is not what defines clutter.

It's probably worth noting that Pfeiffer, the consulting group which makes this report, seems to be a consultant for marketing departments.

One shouldn't read these reports as a scientific publication (b/c their methods are certainly suspect at best), but rather a way to try and translate product differences into things for the marketing department to emphasize.

Is more customisation really better? I would have left that out of the report as it can be argued either way. High levels of customisation mean you can tailor the device to your needs. On the other hand a device might be designed to work perfectly for 90% of the users and offering customisation options will just lead to confusion and people not getting the best experience.

Who defines what is "best experience"?

What I mean is the setup that works for the majority of users. For example on my iPhone I don't feel I need to customise anything. It's setup well out of the box so lack of customisation options isn't a problem. In fact if given the options I'd probably mess with them and negatively influence the experience. On my Android device I customise it lots. The OOTB experience isn't bad but not perfect and so lack of customisation options would be a problem.

Like when my sound volume stuffed up on my iPhone. There were no settings to recover it... Or when iTunes refused to copy music to my phone, but 3rd party tools would. As soon as iTunes was restarted, it deleted the good "newly added" tracks and corrupt them. My experience with Apple: it works most of the time, but when it doesn't, you're screwed because there's no setting to fix it. Good news: Apple added "reset settings", which would have solved my audio settings quickly.. Without this feature, it took me months to solve the problem. Btw, I LOVE the IOS7 control center !!

Or my SO's iPhone, which started mysteriously filling up with "Other" data that wasn't associated with any app, until she couldn't do anything with the phone since it was perpetually out of space. She had to reset the phone to factory settings, skip iCloud restore, and rebuild all her settings from scratch. All because Apple refuse to let you access the file system to see where the hell that space is going.

What I mean is the setup that works for the majority of users. For example on my iPhone I don't feel I need to customise anything.

Good for you.

However the world is bigger than you and the first thing I would customize if I had an iPhone: Replace the default browser with Firefox, because that's the only browser which has me covered. Except you're not allowed to do that. It can't be done.

That's a score 0 for something which matters a lot to a lot of users: being able to use their preferred browser. And that's definitely a customization.

In fact if given the options I'd probably mess with them and negatively influence the experience.

Pardon me for saying so, but saying anything like that just makes you sound like an apologist, a shill. It instantly makes you 200 times less credible.

> Pardon me for saying so, but saying anything like that just makes you sound like an apologist, a shill. It instantly makes you 200 times less credible.

That's just uncalled for. Can you not perceive that the very same can be said of ”…the first thing I would customize if I had an iPhone: Replace the default browser for Firefox, because that's the only browser that has me covered." comes off in exactly the same manner? The last part comes off as marketing speak to me. Are you an employee of Mozilla, paid or otherwise? As to reducing credibility; petty passive-agressive name calling lowers yours immeasurably.

I really liked someone's observation: Android is about what you do to your phone, iPhone is about what you do with your phone.

Nice, but wrong

I think that's slightly unfair.

The reason a lot of people like Android is that they can customise it to remove even the slightest friction from the things they want to do. They have to make a bit of an investment in time to do that but to them that's a good thing because it will pay off.

Android is absolutely more about what you want to do to your phone than iOS, but I'm not sure it's fair to suggest that it's not about doing things with your phone. That feels a bit like someone just trying to find a comparison to make it into a nice phrase.

This seems like such an arbitrary report. Looking at the "analysis" for UXF it just reads like a list of random annoyances their testers came across.

Nice to see they pick Samsungs version of android, the worst one going.

Yes and its the one most people know?

Let's compare user experience. One phone is straight out of the box. The other we will just take a minute to gain root and reinstall the entire OS. Just to keep things fair.

But it means they're not comparing iOS (and WP and BB10) to Android, they're comparing them to TouchWiz (or whatever Samsung are calling their skin these days). In that case, they should also include HTC Sense and Motorola Blur etc. The fact that TouchWiz is most popular is irrelvant; pretty much any Android skin is probably more popular than BB10, and many are more popular than WP as well.

Relatively no one who walks into a store to buy a phone understands any of those words you just said.

They just know they bought the most popular android phone and I guess this is what android is?

Edit: By your definition you can't compare anything to android because android doesn't exist.

Is it possible to compare windows to linux?

> Relatively no one who walks into a store to buy a phone understands any of those words you just said.

> They just know they bought the most popular android phone and I guess this is what android is?

SifJar was talking to you, not the average person who walks into any store.

> Edit: By your definition you can't compare anything to android because android doesn't exist.

He didn't try to (re)define anything. To give you a clue, when someone talks about Android to me, I think about the Android Open Source Project, the OS managed by Google. It's essentially what the Nexus devices come with.

The problem with the study is that they claim to compare different operating systems, while more precisely, they are comparing different user experiences on different OSs. Big difference, I'm explaining it below, and they should know better.

In the case of Android, most OEMs do everything they can to differentiate their devices from other Android devices, and thus they offer a fundamentally different experience. They even give names to these experiences, e.g. TouchWiz or Sense. So the authors were actually comparing TouchWiz to other operating systems. This is evident by the fact that some of the features they felt were lacking are actually in Android trunk, and some of the features they found problematic are not.

Thus, I feel like the people who did this UX study didn't actually know what exactly they were studying, based on their choice of words (TouchWiz is not mentioned at all).

Exactly, I thought -- before reading the report -- that they're going to compare AOSP. The different skins OEMs put on their phones massively cripple the UX of AOSP, I think pretty much everyone agrees with this. So while it makes sense to include some of the more popular UIs, it's very short-sighted to equate those to Android. It would have been much more interesting if we could see what the OEMs start with, and how they make Android overly complicated and awful.

There is no need to root or reinstall the entire system. You can just install a new launcher app. I think it would be better to have tested against vanilla Android (The one that comes with the nexus). Probably more people has that system than a Windows Phone.

This sure is a thing normal consumers do. Installing a new launcher app.

Go Launcher: 100-500mil installs Nova Launcher: 5-10mil installs Smart Launcher: 5-10mil installs ADW.Launcher: 5-10mill installs

That's just the top 4 and makes it over 115 millions installs. Facebook Messenger has 100-500mil installs. So I think it's safe to say that there is plenty of people that install launchers.

But anyway, I was pointing out that you don't require to reinstall the system to customize the system. In fact, you can make an android look like any of the other ones any time and without too much tech savvy-ness

So which launcher should be used in the comparison? I just checked out Go Launcher and it doesn't look any better than Touch Wiz. Personally I prefer stock Android and got a Nexus 4 because of that. The fact is a large, maybe majority, of Android users are using TouchWiz so it seems fair to use that in the comparison. They probably should have included a few versions of Android (Sense, Stock, most popular launcher app) to improve the results.

That is a lot of effort from you to defend your purchase. They are just telephones, y'know?

If they where just telephones, Nokia would still be the biggest player in the mobile market. The whole point is that they are not just telephones, they're mini computers that you can occasionally use to make phone calls.

The overwhelming majority of buyers use it as it comes out of the box so an out of the box comparison is the only fair one.

It's also one of the most popular versions. Everyone I know that owns an Android device owns a Samsung or a cheap device that can barely be classified as a smartphone. TouchWiz IS Android to these people.

the fundamental flaw of these kinds of survey: if you have enough time to participate then you're not majority, you will never value time-saving features as highly as people who matter will, for example. lots of other things. I never regard experiments using accidental sampling seriously.

Can the experience being behind a wall garden buddy buddy with the NSA ever be truly good, ignoring superficial things like icons and launchers?

Indeed, you make a good point for avoiding the Google Android ecosystem.

Reply on news.ycombinator.com