Posted by: bbannan | March 15, 2010

Usability Testing Metrics – Jeff Seeley

Usability (Testing) Metrics. Tullis and Albert, the authors of our text, Measuring the User Experience, dedicate the majority of their book to metrics – from definition and types to use and measurement. A metric is commonly defined as “a way of measuring or evaluating a particular phenomenon or thing” (p. 7). Usability (testing) metrics involve (1) a user, (2) doing something, (3) with a product, system or other thing (p.4). Building on classmate Judith Stevenson’s 1 March 2010 Design + Research article on Usability Testing, metrics and their measurement provide tools and results for determining design and development effectiveness, tracking design progress, providing insights and influencing decision-makers. Metrics replace hunches and feelings with facts. Usability metrics and their measurement can show improvement, decline or indifference (i.e., no change) in a user’s experience with an updated, improved or changed product or process (p. 9).

Our text provides an excellent article (p. 10 – 13) debunking ten (10) common myths about usability metrics including issues concerning time requirements, costs, ineffectiveness at determining causes, conflicts with gut feelings, misunderstandings by management and difficulties with small sample sizes. Usability metrics can: (1) be cost and time effective, (2) address a wide variety of issues and products of any size and (3) be understood and appreciated by management.

There are several variations or types of usability metrics that can be measured including performance, issues-based, behavioral and physiological metrics.

Performance metrics (p. 63) include several measurements:

  • Task success – how effectively users are able to complete a given set of tasks
  • Time-on-task – how much time is required to complete a task
  • Errors – mistakes made during a task. Errors are useful in pointing out particularly confusing or misleading parts of a product, process or interface.
  • Efficiency – amount of effort a user expends to complete a task
  • Learnability – how performance changes over time

Issues-based metrics (p. 99) include determining:

  • What prevents task completion by a user
  • What distracts a user
  • What creates confusion for a user
  • What produces an error for a user
  • What causes something not to be noticed
  • Why a user will assume something is correct when it is not
  • Why a user will assume a task is complete when it is not
  • Why a user performs a wrong action
  • Why a user misinterprets content
  • Why a user does not understand navigation

Behavioral and physiological metrics (p. 167) cover:

Verbal Behaviors:

  • Positive and negative comments,
  • Suggestions for improvement
  • Questions
  • Variation from expectations
  • Confusion
  • Frustration

Non-Verbal Behaviors:

  • Facial expressions
  • Body language
  • May require special equipment to capture data

Usability metrics provide precise, quality measures that can be used to evaluate a system or product. Their purpose is to produce a system or product that is neither under- nor over-engineered. Usability metrics can help a company outrun its competition. As our readings discuss, usability metrics are usually measured in the big three Es, effectiveness, efficiency and emotion:

  • Effectiveness – the accuracy and completeness with which users achieve specified goals
  • Efficiency – the accuracy and completeness of goals achieved in relation to resources
  • Satisfaction – freedom from discomfort, and positive attitudes towards the use of the system

Many of the usability (testing) metrics for software is based on research conducted several years ago (Mayhew, 1992; Nielsen, 1993; Whiteside, Bennet, & Holtzblatt, 1988). Still very useful today, it is equally effective for measuring usability for most products or services from cell phones and computers to websites and software. These usability metrics are the big three Es further refined:

Effectiveness

  • Percent of tasks completed
  • Ratio of successes to failures
  • Workload
  • Number of features or commands used

Efficiency

  • Time to complete a task
  • Time to learn
  • Time spent on errors
  • Percent or number of errors
  • Frequency of help or documentation use
  • Number of repetition or failed commands

User Satisfaction

  • Rating scale for usefulness of the product or service
  • Rating scale for satisfaction with functions and features
  • Number of times user expresses frustration or anger
  • Rating scale for user versus technological control of task
  • Perception that the technology supports tasks as needed by the user

Use of the big three Es can enhance the results and improve the final smart phone software products we are developing as class assignments.

While the misuse of metrics should be avoided (i.e., using metrics where non are needed, presenting too much data at once measuring too much at one or over-relying on a single metric), properly developed and evaluated usability (testing) metrics are cost and time effective, address a wide variety of issues and can be understood and appreciated by management and other decision-makers.

Useful Links:

Jakob Nielsen’s Alertbox, January 21, 2001:

http://www.useit.com/alertbox/20010121.html

Usability Metrics

http://webword.com/moving/metrics.html

NASA Usability Testing Handbook

http://software.gsfc.nasa.gov/AssetsApproved/PA2.5.1.3.pdf

Advertisements

Responses

  1. Jeff, I enjoyed your blog posting about the various types of metrics that can be used in usability testing. You mentioned that one advantage of metrics is that they can “be understood and appreciated by management”. I relate very well to that advantage from experience. Managers are very wary of spending money on what they view as frills and extras, and they can easily view usability testing as such. Conversely, managers respond positively to figures and measures. So the use of metrics to show the increase in tasks completed or the reduction in time needed to get work done goes down very well and persuasively with the decision makers in top management.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: