Posted by: bbannan | March 1, 2010

Usability testing – Judith

Usability Testing

What is usability?

Before you can test for usability, you need to know what it is. According to usability consultant, Jakob Nielsen, usability is a part of “usefulness” and it is composed of:

  • Learnability: How easy is it for users to accomplish basic tasks the first time they encounter the design?
  • Efficiency: Once users have learned the design, how quickly can they perform tasks?
  • Memorability: When users return to the design after a period of not using it, how easily can they re establish proficiency?
  • Errors: How many errors do users make, how severe are these errors, and how easily can they recover from the errors?
  • Satisfaction: How pleasant is it to use the design?

What is usability testing?

Usability testing is a technique used to evaluate a product by testing it with representative users. Nielsen offers 3 basic rules for usability testing: get representative customers, ask them to perform realistic tasks, and shut up and let them do the talking.

I thought these 3 rules were very interesting, especially the part about listening and not intervening. It is a natural instinct for me as a teacher to intervene and try to help people learn how to do things. I had to learn, when I became an educator of adults rather than high school students, to stand back and let people try things first. I also had to discover, when I do intervene, how to empower my adult students to learn for themselves rather than showing them things. Usability testing takes this all one step further: in doing user testing, I will need to hold myself back and not intervene at all, since the goal is to see what a user can and cannot do on their own. This would be quite difficult for me!

A good practice in usability testing is to get the user to talk about what they are doing as they perform the tasks. I would like to try this, and I even wonder whether it could have its place in the teaching of adults as well. For instance, in tutoring it could be useful to know what a student is thinking as they try to attack a difficult Math problem or to create an outline for an essay. I plan to teach a class at my college in the near future and wonder whether I could incorporate an element of this “flow of consciousness” technique into my teaching. Am I out of order here, do you think? Would my students dislike this? I’d be interested in whether any of my peers have ever tried anything like this in a class or in individual tutoring, and how well it worked.

Measures and results in usability testing

How should you measure success in the tasks that you ask your users to perform? What is the most useful way to collect and represent the data from your testing. Nielsen suggests that you measure success in two ways:

  • Could the user complete the task at all?
  • If so, how long did the task take?

These are good measures because they are quantifiable. To be able to interpret the results in a usability test, most of the results need to be quantifiable so that you can compare them, and plot them or represent them in charts. This can work, not only with tasks, but also with feelings and opinions about the site, so long as you use Likert scales when collecting the user comments.

However, as I mentioned earlier, the usability experts favor tasks over expressions of opinion. This interests me because it brings the realm of psychology into the user testing arena. The idea behind this rule of having the users perform tasks is, not only that this shows whether the product can do what it is supposed to do, but also that what people say is influenced by many factors besides the usability or convenience of the product. They may wish to be polite, please the tester, not offend their boss, appear to be competent and smart, or fulfill some other motivation. The beauty of having them go through tasks is that these other motivating factors are taken right out of the picture. The testing questions become simple: Can the user do the task? How long does it take?

How many participants do you need?

Nielsen say that you only need “5 users to uncover enough usability insights to keep you busy for months”. I can imagine that, though as I read it I thought that it is not the same thing as saying that 5 users can tell you what you need to know. However, Nielsen delights in being provocative. When I read further, I found that most of the usability experts that I read, including Nielsen and Molich, also said that you can identify a design’s most important usability problems by testing with just 5 users. Their idea is that the rate of return falls sharply when you test with more users, since the major issues with a design start to emerge from the first 5 users tested; after that, other testers find the same major issues, along with a few more less typical ones. This finding is helpful in that it reduces the cost and effort needed to perform a usability test, but I would feel happier with the results myself if I could do some sort of larger survey, as well as testing my small group of 5 users in a task-based setting. Perhaps that is because I am not experienced at user testing and have not seen how the results emerge.

Usability testing for mobile phones

I also found some ideas on Nielsen’s web site about user testing for mobile phones in particular. He said that there is a 4th rule, which is to run the test on representative equipment. For mobile applications, Nielsen commented that it is harder to use “representative” equipment, because phones differ so much more than computers do. This is a significant problem, since it is time-consuming and expensive to test an application on many different types of phone. This issue would not apply to my group’s design, if and when our client at IST tests the application with an actual smart phone in the future, since they have already selected a specific smart phone to use and distribute to users, but that is something of a special case, due to the target environment in Africa: in the case of most mobile applications, I can see that the time and costs of testing on a sufficient range of mobile phones is a major stumbling block.



  1. Judith:
    First I would like to say ou have some great resources for usability that I plan to refer back to in the future.

    I find it interesting about only needing 5 participants for usability testing. In my mind, I feel like it depends on the quality of the testers. Where I’m working right now, we have one tester. That’s his job and I think he could probably provide me better feedback on my products than 5 testers off the street. Of course it’s always nice to have an extra eye or 4 to go through the testing. Quality testers need to have a vested interested I think, otherwise they are apt to shrug it off and not really care.

  2. Great post Judith, starting out by defining usability testing with quick definitions that make up usability. The last porting for you post caught my eye. Usability testing for mobile phones and Nielsen’s 4th rule. I also found this method to be extremely difficult , time consuming and expensive. I feel because of the limitations mentioned before, many companies may opt-out of this option, leaving the accessibility and availability on many phones a problem.
    This issue also dose not apply to my groups project, since we are designing on the iPhone. But I think researchers should look into this issue and come up with ways to make this type of testing universal.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: