Posted by: bbannan | February 24, 2009

Optimal usability test size? – Savita

Sample Size Oddities

This article considers the optimal usability test size. Jakob Neilsen’s often quoted “Why You Only Need to Test with 5 Users” article is cited, and backed by a new reasoning: Some problems are simply experienced by more people and show up early and repeatedly. As the author points out, however, if you only test a small sample-size group of users, minority voices are not heard. The author purports, “the larger the proportion of a population that holds a given opinion, the fewer people you need to interview when doing user research. Conversely, the smaller the minority of people who share an opinion, the more people you need to interview.” Do you agree? Why? Savita



  1. I think the fatal flaw I’m seeing in this article is that the author does not consider the use of subgroups to learn more about those who fall in a minority category. If I’m reading this correctly, the author is calling for a larger sampling selection that ensures inclusion of subgroup populations when necessary. I’m concerned that because they would still be a smaller portion of the sample, their results would still lost (though some representation is better than no representation at all).

    It seems to make more sense that rather than clumping everyone in together, that the subgroups of interest be drawn out and studied separately from the main group.

  2. The key here seems to be ‘representative sample.’ No matter what other parameters you are testing for, it is necessary to ensure that each group of variability is represented. One thought to consider is that until you begin the interview process, it is difficult to determine the degrees of variability. It is possible that some group may be missed no matter how careful the research; however, it is worthwhile to reach out to as many suspected groups as possible.

    The second thing to consider is the ROI or return on investment. While it would be wonderful to administer the usability test to as many groups as possible, opinion is a volatile commodity, subject to change and misinterpretation. Minority opinions should be heard, unless a particular opinion is positioned so far from center that its input can have no true bearing on the outcome.
    Once you have identified as many of the possible group types as you can, given the type of test and its focus, attempt to test as thoroughly as possible within the scope of the project.

    Here is one final thought. There is, in my opinion, no hard and fast rule for usability testing. The degree of testing and its scope and breadth are the result of a comingling of factors that go beyond the reasoning brought out by the new reasoning for Jakob Neilson’s article. Still, in a totally homogenous group of test subjects, the probability seems to indicate that larger sample may not result in new insight, while a truly heterogeneous sample may yield too many variables to establish a usable trend.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: