Posted by: bbannan | March 1, 2010

Formative evaluation – Katrina

In Measuring the User Experience, Tullis and Albert state that a formative usability study is most appropriate when there is still opportunity to make adjustments to the design, which makes sense as formative evaluation is an iterative process where the data gathered is used to continually make improvements in the design.  If it is too late in the design process to make adjustments, then any type of formative evaluation really becomes a waste of resources.

In their article “Usability Evaluation Considered Harmful (Some of the Time)”, Saul Greenberg and Bill Buxton discuss how the purpose of usability evaluation can differ depending on the context.  According to Greenburg and Buxton, generally speaking usability evaluation is valuable for many situations because “it often helps validate both research ideas and products at varying stages in its lifecycle” (p. 111).  However, they argue that usability evaluation may not be appropriate for all interface development situations and that usability evaluation does not guarantee user-centered design. Greenberg and Buxton’s stance is that “the choice of evaluation methodology – if any – must arise from and be appropriate for the actual problem or research question under consideration” (p. 112).

Tullis and Albert list 5 key questions that one should be able to answer when selecting a formative method of evaluation (p. 46):

  • What are the most significant usability issues that are preventing users from completing their goals or that are resulting in inefficiencies?
  • What aspects of the product work well for the users? What do they find frustrating?
  • What are the most common errors or mistakes users are making?
  • Are improvements being made from one design iteration to the next?
  • What usability issues can you expect to remain after the product is launched?

When questions such as these are not considered prior to selecting and/or beginning a usability evaluation method, Greenberg and Buxton discuss several problems that can arise in the interface development process, two of which are discussed below.

Usability Evaluation as Weak Science

When designers decide blindly that usability evaluation MUST automatically be a part of the development process, what can often happen is that a method of evaluation is chosen and research questions and efforts are formulated to fit that method instead of creating the questions and selecting the method most appropriate to address those questions.  The outcome is that, yes, there will be data available, but it will most likely not be useful data that will accurately inform the design process and may result in the design taking a turn that adversely affects the user experience.

Usability Evaluation and Early Designs

Greenburg and Buxton argue that if a usability evaluation is done too early in the design process, it can result in even better ideas never being realized, and they go on to discuss the importance of differentiating between a product sketch as opposed to prototype being used as the focus of evaluation, both of which have a purpose in the design process but different purposes.  When a sketch is used as prototype as part of the evaluation process, obviously, significant problems will most likely be found and the design process can be set back to square one or the designers and developers may then set their energy on focusing on improving that specific design instead of focusing on other, possibly, better designs that may exist but have not yet been seen into fruition, so potentially great ideas are dismissed too early in the design process, resulting in fewer overall designs being considered.

So what does this mean to us as we are engaged in this stage in our design process?

Plan, plan, plan.  I think that from the very start of the design process for any product, be it commercial or academic, a clear plan of purpose and goals must be established (which we have all done) along with a plan for testing or measuring how well that purpose and those goals are being met (where we all currently are).  I think that this goes back to determining who the audience is and what exactly is it that you want them to be able to do as a result of using your product.  I also think that having a clear plan of research established eliminates wasted efforts.  I’m also aware that even with the most clearly laid out plan, unforeseen problems may arise during the development process (The best laid plans of mice and men and all that). When this happens and it appears that the research plan will need to be modified a good question that one should ask oneself is, ““Will this step in the evaluation process help me improve my product?” followed by the question, “How?”.  If definitive answers to these questions exist, then that may be a new or different direction in the evaluation process worthy of pursuing.

Links of interest:

S. Greenberg & B. Buxton. “Usability Evaluation Considered Harmful (Some of the Time).

Journal of Usability Testing. This website contains direct links to fully downloadable free articles.

This is a link to free downloadable forms from the Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests, 2nd Edition,descCd-DOWNLOAD.html



  1. Hi Katrina, thanks for summarizing formative evaluation. This helps add to the conversation we had about it in class last night. I am very interested in this type of evaluation, not only because we are creating a plan for how to conduct our own prototype evaluation for class, but also because I am in the beginning stages of designing my first online training course for work. I plan on incorporating formative evaluation into my design. With that said, I have a question about Greenberg and Buxton’s statement that usability research is not always appropriate for every situation. Were they just talking about not doing it too early in the process (like at the beginning when you’re sketching out ideas because that might prevent ideas from being expressed), or did they give other examples of when usability research is inappropriate?

  2. Hi Katrina, I was interested to read your thoughts and research about formative evaluation. I had not really thought before about the idea that one might test too early in the development process, but had mostly thought about the danger of leaving the testing too late, when changes could not be easily made. However, I can see the point that testing an early “product sketch” might be misleading, since many things would be “wrong” with the design at that stage. Since testing can be quite expensive and a hard sell to employers, it is important to find the right stage or stages at which to test. This seems to a more difficult issue than I had thought up to now. I wonder if there are any guidelines “out there” on how to decide when it is the best time to do formative testing for your design idea or prototype.

  3. Hi Shannon,

    Thanks for your question. The authors did cite other examples of instannces where usability evaluation may hurt the design process. They talk about how it’s important to consider the culture into which a technology or new design is being introduced into. Some cultures may be more reticent about adopting new designs which can skew the results/feedback that might be gained from usability evaluation (which, in my opinion, is more about the audience selection for the usability evaluation than anything else).

    They also talk about the danger of choosing the research method prior to fully formulating the research question. Basically, they were saying that one shouldn’t just automatically assume that usability evaluations are going to be the best way of testing a design and when you do decide that usability evaluation is going to be a method that you are going to employ, then careful planning of when and how it will be carried out is the key to ensuring that you obtain results that can be helpful in the revision of your design.

    I hope this has been helpful.


    p.s. The first link under “Links of Interest” is a link to the article if you’d like to read it to get a more robust look at what I briefly detailed in my blog.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: