Jokob Nielsen: Guesses vs. Data as Basis for Design Recommendations
useit.com
useit.com
"100% of the designers who provided external data were right, whereas
"25% of the designers who relied on their personal opinion were right.
"Most strikingly, 75% of guessers were wrong. You'd be better off tossing a coin than asking advice of these people."
A very important point. Confident guessing can send more convincing than data-based hypothesizing, even when it is flat wrong. This is why I appreciate any participant here on HN who can provide data to back up statements, and why I also appreciate comments that ask for data--yes, including from me--if a statement is first made without data.
That claim above is absolutely ridiculous. If it were true you'd do much better by always doing the exact opposite of what they propose. If you can do as well as by doing the opposite of what they guess as by doing real research, then the guys clearly know far more than a random coin.
Consistently getting 75% wrong is difficult.
The data sets are too small to do any meaningful extrapolation from. The important message should've been about falsifying claims.
When you're designing anything you have a whole bunch of assumptions. Those assumptions ("everybody knows how to do X") can be falsified by a single counterexample. But this absolutely doesn't imply that if two people happen to think X that that is a meaningful reason to assume X is correct.
How do we know how many possibilities there are? Maybe there isn't just one exact opposite choice.
But, agreeing with Jakob Nielsen, if guessers are getting it wrong more than half the time when dealing with yes-no questions, it isn't a very good idea to guess, and guesses are poor guidance for important business decisions about Web services.
"Intuition" and "common sense" are nothing more than predjudices. They may be right; after all, prejudices are usually founded in real-life experiences.
The problem comes when you over-generalize those experiences and crystalize that over-generalization as knowledge.
A silly example: you run a test on a website about button colors and red is the best color and that it made a large difference.
There are a lot of lessons you could possible take from this, e.g., 1. Users prefer red buttons over any other color 2. You should always test button color 3. It's important to think about what draws users' eyes and test that, including buttons. 4. It's important to test everything, no matter how small
Learning (1) is probably wrong. Red might work in this situation, but not in another. This is the problem with learning tactics in general. As the competitive landscape evolves or the situation "on the ground" changes specific tactics become less useful, and might even be worse than an alternative you've precluded because of your "expertise."
(2) and (3) are probably good lessons to take away. (3) is better because it opens your eyes to new, possibly fruitful things to test, although (2) isn't bad.
(4) is the opposite of (1) in that it's an overbroad lesson that is hard to apply and if done religiously would probably paralyze the decision-making process.
Anyhow, that's how I think about applying "data-driven" processes to business, marketing, and product decisions.
What I've found is that companies that live or die by data don't have better arguments, necessarily, they just have different arguments.
At the end of the day it's about someone making a decision and taking responsibility for it. Sometimes forward momentum and the feelings of your employees are more important than minute details like that.
The story was more about Marissa Meyer's ego and how she uses the data-driven process at Google to frustrate her internal enemies, IMO, and less about the specific fact that they tested 41 shades of blue.
Personally I'd just make a decision and get on with it. If, for whatever reason, we thought the shade of blue would really, really matter you can always test it latter.
The risk of being data-driven is that you wind up in a world of Cartesian uncertainty. Do I really know ANYTHING? What if this thing I don't think matters really DOES matter? Doesn't that mean I should test every detail no matter how small?
Anyhow, these are issues you have to grapple with when you commit to being data-driven.
There's no simple answer, although it's easy for Google to justify testing every single thing because they have a gargantuan amount of traffic and even an improvement of a fraction of a percent could mean millions of dollars in revenue won or lost.
PS:It's "Jakob"
You mean "it's that Jakob again"? You must be designer.
Oh, I get it: you are a hacker, you mean the typo in the headline.
I like the fact that Jakob took it one extra step and included data that shows making decisions with data is (empirically) better than making decisions without data.
Very meta. :)
But that is what makes what you do stand out and what separates the excellent designer from the mediocre designer.
Just the same that makes the excellent developer stand out, that which cannot be put into a data set.
I challenge anyone to find a common design functionality that couldn't be solved with common sense and intuition.
Something that you could prove the actual results would be better if they where based on data.
The problem of course is that no such data exist.
Where can I see your design work? My company is looking to hire some designers on a contract basis.
I like reading expert articles from people not trying to sell me their expert services.
Yes if you are a complete newbie or academic without any experience about webdesign. Otherwise not in a million years.
The map is not the territory.