Posted by: bbannan | April 20, 2011

Reporting the findings of user-experience research- Ghania Zgheib

In this course, we have learned to evaluate our prototypes and revise them based on methods of user-centered research. We may think that the evaluation of the prototype and the results are the most important, however, if we are working with a client, reporting on the findings needs careful thinking. For our AR mobile applications, we are supposed to report on our user-experience research results twice. The reports should present our data collection methods, the analysis of our data and recommendations for revisions of the prototype. If we were presenting the reports to an existing client, the report should have a certain format because the success and the failure of the product sometimes depend on it.

After reading Kuniavsky’s chapter 17 and other websites, I learned that writing a report needs careful planning and certain formats. Kuniavsky’s recommendations for planning and writing a report are very similar to other websites that I looked at. I present a summary of the strategies that are presented by Kuniavsky, Bruner’s Foundation, SAP Design Guild, and usability.gov.

Planning for the Report

Before beginning to write the report, certain considerations need to be made:

  1. Determine the needs and purposes of your key stakeholders.
  2. Develop a report outline and present it to stakeholders before further development of your report.
  3. Revise the report outline and all other report plans to incorporate key stakeholder suggestions.

Writing the Report

The following points represent the common sections of a report as reported by Kuniavsky, Bruner’s Foundation, SAP Design Guild, and usability.gov.

  1. Subject program description
  2. Clear statement about the evaluation questions and the purpose of the evaluation.
  3. Description of actual data collection methods used
  4. Summary of key findings (including tables, graphs, vignettes, quotes, etc.)
  5. Discussion or explanation of the meaning and importance of key findings
  6. Suggested Action Steps
  7. Next Steps (for the program and the evaluation)
  8. Issues for Further Consideration (loose ends)

A link to a template for a report is provided at the end of this page.

Bruner’s foundation mentions very useful strategies to avoid while writing a report:

  1. Including problems with the methodology in the findings section
  2. Reporting both numbers and percent at the same time
  3. Listing all the responses of the data collected
  4. Compartmentalizing the results
  5. Feeling compelled to use all the information collected
  6. Including conclusions that are not developed by the findings.

Therefore, the presented report represents the outcome of our data collection. Careful reporting is integral to the design process and to the modifications that need to be made to the product.

Related links:

Reporting: Turning data into information: http://www.sapdesignguild.org/editions/edition2/overview_edition2.asp

Usability Reports: http://www.usability.gov/templates/index.html#Usability

Using Evaluation Findings: http://brunerfoundation.org/ei/docs/EvaluativeThinking.bulletin.6.pdf

Advertisements

Responses

  1. Ghania,

    Your posting is quite timely and a good reminder of practical considerations we should be thinking about as we prepare our final presentation. I also found Chauncey Wilson’s practical tips on usability reports that breaks down and simplifies the reporting process:

    1. Always provide your sponsors (clients) with a date for report availability and make sure that the report is ready. Make sure that you schedule time in for report writing (this seems obvious, but is not always done).

    2. Target your report for a particular audience. Developers like as much detail as possible so a long report with the problem, rationale, and possible solution in a table format would be appropriate. If you are giving a 15-minute talk to the senior management group, then a short report (to accompany a 15-minute presentation) with an executive summary and brief details of key conclusions would be more appropriate than a long report.

    3. For a detailed report, consider using one long table to list usability problems. If you use one long table, you can add/delete columns to it for sorting by different attributes. For example, if you have a column that numbers items from 1-N, a column that lists the object where the problem occurred (for example, the menu or a specific dialog box), and a column with a priority, you can sort by priority or object and use the column with numbers to get back to your original ordering. Using one long table, you can quickly copy and paste and sort and create custom version of the report. You could also use a code for which design principle was being violated and sort on principle.

    4. A little tip that has worked from me when I have little time to analyze data is to get large Post-It notes and write one usability problem on each Post-It note as I am observing a participant or reviewing a product, then using affinity diagramming to quickly categorize the problems. This reduces the time it takes to organize data for a report (you don’t have to excerpt things from your notebook or a logging tool. (Thanks to Mary Beth Butler of Lotus for this tip).

    5. Providing the rationale or principle that underlies the problem is more convincing that a person’s opinion. Whenever possible, I always try to provide a basis or rationale for a problem. Too often usability reports are just long lists of problems.

    6. There is a bias toward reporting problems in usability reports. I think that it is useful to have a section in a report that describes what users like about a product. I’ve observed sessions where users like a number of things about a product, but the consultant’s report focused only on the negative.

    7. If there is a clear solution to a problem and you can come up with an alternative screen design, consider putting an illustration of your proposed solution in the document. Make sure that your audience is willing to accept some outside design recommendations though.

    8. Ask your sponsors what they would like to see in a usability report. Outline the major sections of the report in your usability proposal. Ask if the team wants the raw data that supports the problems (make sure that you do not violate privacy when you do this).

    9. Ask for feedback on your report. Usability specialists need feedback too.

    10. Be quantitative whenever you can, even if it is to have a table showing the number of low, medium, and high usability bugs. If you did some think aloud studies or field interviews, you might note the number of people who have a particular problem (for example, of the 10 people interviewed, 9 experienced PROBLEM X) at least once during the testing or field interview).

    I think that #8 is the most valuable tip because we need to keep in mind that clients have very little time and we can maximize the reporting session by providing only the information that they consider necessary and important.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: