Hacker News Reader

He mentions the small sample size (5 people) as a negative, but I wonder if this situation fits Jakob Nielsen's arguments about testing smaller groups.

http://www.useit.com/alertbox/20000319.html

View on HN

Jakob Nielson argues that, specifically in usability testing, 5 people gives you a majority of your flaws but not the entire picture - where ~15 people would give you that. That is to say, more is better but there is diminishing returns. Given this, he argues that rather then expending resources to gather the entire picture for a single iteration, it's even better to test 3 sets of 5 people with design improvements between each set.

I think this argument is slightly different than just saying 5 people is enough for any sort of study. I still would have liked to seen more people. Or at least additional studies with modifications to inspect or confirm specific aspects of their findings.

I actually approve this argument. My personal experience also shows that usability testing can be done with very few people at a time, over many iterations. Obviously, this is not a universal rule. It's useful for practical usability testing, but not for testing all cognitive processes. Perhaps, if the variations are small enough, it may be sufficient.

Reply on news.ycombinator.com