Posted by: bbannan | March 23, 2011

Usability testing metrics – Jane

As we are all discovering this semester, usability testing is a critical component of design development. In the most obvious ways, determining whether or not a product is usable or useful to someone goes a long way in predicting its ability to function or be successful in the market. (Although, I can remember lots of “programmable” electronics I’ve had in my home that were totally indecipherable and unusable: clearly ineffective or nonexistent usability testing is rife).

Kuniavsky (2003) emphasizes the importance of user testing during the iterations of design development: feedback gleaned from user testing can provide: warning flags, insights into people’s perceptions and behaviors that can inspire new ideas for development, or confirm what works and is liked, or what doesn’t and what people don’t like. It offers the chance for developers to refine their products, release them, or recognize when to cancel them altogether. It also allows developers to quantify improvements or degradations between iterations (Nielsen, 2001b).

So, how is user-testing structured? There are a number of elements to incorporate or consider when designing user testing, each worthy of its own discussion: selecting the users to test, the environment in which to test, what features to test, and what metrics to use.

What are metrics? Turns out, metrics is a bit of a tricky word. I had thought it was simply “measurement.” But I was wrong, or at least partially wrong. According to a number of different online dictionaries, (, a distinction is made between the two: “measurement” is the extent or size of something, where “metric” is the system of measuring something (although Merriam Webster makes no such distinction). As I thought about this (and it took a while to sink in), the distinction started to make sense to me, particularly as it applied to my task at hand here: writing a blog post about usability testing metrics.

What I finally arrived at is that usability testing metrics is the field, or subject, or tool, that addresses how to systematically go about measuring the user experience. “Systematically” is the key word: to accurately measure, and compare user experiences, between users or over time or both, some standard evaluation tool is needed. It is more than just the measurement of the size or extent of something.

As our readings (and other available literature) on the topic suggest, there are several standards, or “metrics” that are generally used in measuring usability. The most commonly referenced, as they are spelled out by the ISO (International Organization for Standardization) include three broad metrics of usability: effectiveness, efficiency, and user satisfaction (impossible for me to directly cite, cited by many, including Alshamari & Mayhew, 2009 ). I found a variety of others which are similar in nature, where all share rating effectiveness as part of the usability metric, but where there are notable differences between them as well:

  • Our government (US Dept of HHS, 2011) spells out a slightly different set of five areas to be measured in usability testing: ease of learning, efficiency of use, memorability, error frequency or severity, and subjective satisfaction. The inclusion of ease of learning and memorability stands out as new or different from the ISO standards.
  • A usability consulting firm in Canada uses five key metrics: effectiveness, efficiency, expectations (both the user’s and the product developer’s), emotions, and experience (Meditative, 2011). I think these criteria break out “satisfaction” into three separate categories for measurement: expectations, emotions and experience.
  • Nielsen (2001) suggests a single, “simple” metric for usability: user success rate, or, the “percentage of tasks that users complete correctly,” which is essentially a measure of effectiveness.
  • Although, Nielsen (2001b) also identifies in other articles or postings a different metric of four measurements that are similar to the ISO ones: success rate (in ISO, this would be effectiveness), time a task requires (ISO equivalent: efficiency), error rate, and satisfaction.
  • Consultants (Lewis & Sauro, 2011) from a quantitative research firm that specializes in measuring usability identifies “SUS” which is the System Usability Scale as one tool for establishing a user’s “perception of usability.” They make the case that although kind of old, it is a relevant, reliable and valid tool, which actually measures both satisfaction and learnability, and which also moderately correlates to other measures of usability such as effectiveness and efficiency. It is a ten item questionnaire developed by DEC back in the 80s which asks a test user the following questions:

1.        I think that I would like to use this system frequently.

2.        I found the system unnecessarily complex.

3.        I thought the system was easy to use.

4.        I think that I would need the support of a technical person to be able to use this system.

5.        I found the various functions in this system were well integrated.

6.        I thought there was too much inconsistency in this system.

7.        I would imagine that most people would learn to use this system very quickly.

8.        I found the system very cumbersome to use.

9.        I felt very confident using the system.

10.      I needed to learn a lot of things before I could get going with this system.


In looking for information on usability metrics, I discovered that, as usual, my initial understanding of an idea was way too simplistic, and that though painful (really, journal articles on metrics are tough to wade through) , I developed a much fuller understanding of what the metrics are trying to capture. For sure it has helped me to understand how to better structure the usability testing for our group.



Alshamari, M. & Mayhew, P. (2009). Technical review: Current issues of usability testing. IETE Technical Review, 26(6), 402-406.

U. S. Department of Health and Human Services. (2011). Retrieved from

Kuniavsky, M. (2003). Observing the user experience: a Practitioner’s guide to user research. San Francisco, CA: Morgan Kaufmann Publishers.

Lewis, J. & Sauro, J. (2011). The Factor structure of the system usability scale. Retrieved from

Meditative: The Results People. (2011).Website usability cheat sheet. Retrieved from

Nielsen, J. (2001). Success rate: The Simplest usability metric. Retrieved from

Nielsen, J. (2001b). Usability metrics. Retrieved from:




  1. Jane,

    Thanks for providing such a thorough post on usability testing metrics. You brought up a great point when you stated, “I developed a much fuller understanding of what the metrics are trying to capture.” I believe a huge part in the process of obtaining usability metrics is understanding what you are trying to accomplish, and then choosing the best method to achieve that. I like the method Nielson describes in the link you provided and was really surprised to see that he suggested testing 20 subjects for collecting usability metrics. I can see now how this process would become expensive and often under-invested by many companies.

    I’m excited to see the results of our usability metrics and experience the process of formulating an overall conclusion on our group’s design usability!


  2. Thank you for sharing this information with us Jane. As we are preparing our usability testing, it is good to see in how many ways it can be conducted. I found the US department of health and human services metrics for usability testing to be interesting. I liked the memorability factor that’s added to it. However, I find the subjective experience to be very broad.

    It’s true that reading about metrics is so complicated. I also thought that metric and measurement were one, so thanks for the clarification.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: