Posted by: bbannan | February 15, 2010

Two Views on Recruiting for Usability Testing-Verlinda

In Observing the User Experience, Mike Kuniavsky defines user research as “the process of understanding the impact of design on an audience.” It is easy to get excited about the design aspect of a project. The creative juices of the designers flow to create a prototype, the marketing group storyboards an exciting campaign, and business development is buzzing with the prospect of a new product that will open up revenue streams for the company. What may not be as exciting is finding people in the target audience to test the design, often in iterative cycles. However, this part of the research project cannot be taken lightly or overlooked.  Recruiting the correct people for usability testing is crucial, not only for research, but the ultimate success of the product.

Kuniavsky identifies three basic steps for recruiting participants:

  1. Determine the target audience. Knowing who will use the product is essential before beginning user research. The best process is to use demographics, Web use and technological profiles to create an ideal audience profile.
  2. Finding representative members of that audience. Ask family, friends and co-workers for the names of people who fit the audience profile. Once a database is created, use a screener to filter the right participants for the test.
  3. Convince participants to take part in research. This involves scheduling primary and secondary participants, sending invitations and confirmation notices and making a plan for no-shows.

Other readings and sources about usability testing confirmed these as the basic steps involved in recruiting participants. However, a recent podcast featuring usability testing expert, Dana Chisnell, challenged the details of Kuniavsky’s recruiting plan.

Chisnell’s point of view differed from Kuniavsky in two major ways. First, she believes in recruiting participants based on behavior, not demographics. She argues that just because a person fits the demographic profile of an expected behavior, does not mean that the person executes the behavior. For example, demographics say that young people are more likely to use social networking websites, but there is a subset of young people who reject social media, and therefore would be bad participants for a study even though they fit the profile.

The other point that Chisnell disagrees with is the use of screeners. When recruiting for usability studies, Chisnell prefers to interview potential participants instead of using a multiple choice screener as described by Kuniavsky. She feels that relationships built between the recruiter and participant during the interview results in better quality users, who are more invested in the research and prepared to give honest feedback. She also states that the no-show rate she experiences is much lower because by the end of the interview process people feel they are an important part of the research. Most interestingly, Chisnell feels that screeners eliminate participants who are close to the target audience or who may be useful in testing future versions of the product. She advocates never terminating an interview with a potential group member, even if it appears they do not meet the criteria.

Is one of these methods better than the other or are they different sides of the same coin? I highly recommend that you listen to the podcast and decide for yourself. One thing is certain –research participants who do not match the target audience can render the best planned usability test useless, so recruiting well is a critical task during the research project.

Christiansen, B. (Producer) & Spool, J. (Host). (2009, December 9). Recruiting for Usability Testing Followup. SpoolCast.  Podcast retrieved from : http://www.uie.com/brainsparks/2009/12/09/spoolcast-recruiting-for-usability-testing-followup/

About these ads

Responses

  1. Hi Verlinda,

    I found your post very intriguing. I see Kuniavsky’s point of view as well as Chisnell’s. You ask whether one is better than the other or if their thought it two sides of the same coin. I’m wondering whether it has to be an either/or situation. I almost think it can be combined. I agree, the first step is to determine the target audience. Perhaps by combining the behavior and demographics as I think both are relevant. However, I do think that demographics will end up playing a large role in this. Regarding the disagreement between Kuniavsky and Chisnell on screening, I think at this point this may be a choice that you can make. I agree that relationships between people will increase the number of participants because they feel more invested. At the same time, the multiple choice screener is not a bad way to determine participants. Perhaps a combination of both? Is that possible? Let me know what you think.

    Edith

    • A combination might be the best route. Perhaps a screener to filter in a broad way and then interviews to maximize the value of the participants. Although to Kitty’s point, there needs to be some monitoring of the resources. A two-prong approach is probably more expensive and time-consuming. I guess a balance needs to be struck between resources and a more involved, albeit possibly more beneficial, process.

  2. It seems to me that Chisnell does have some valid points–there may be some people in a target population who don’t execute a certain behavior–but Chisnell’s approach doesn’t seem viable in the real world. In the real world there are time constraints, money constraints, etc. A screener may eliminate a great potential end user, but do you really have the time to personally interview all possible candidates? I think Kuniavsky’s approach is much more realistic.

  3. Interesting questions, Verlinda. While all opinion are not created equal, I think that each potential user can offer something beneficial. Ultimately, I think it comes down to the best use of resources. After all, resources are limited and we have to get the most bang for our research bucks. So, my preference would be to have a mix. Perhaps 80/20, with the majority of users being in the target audience. Maybe it’s just me, but I see validity in the wildcard.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: