After reading Kuniavsky’s chapter 17 and other websites, I learned that writing a report needs careful planning and certain formats. Kuniavsky’s recommendations for planning and writing a report are very similar to other websites that I looked at. I present a summary of the strategies that are presented by Kuniavsky, Bruner’s Foundation, SAP Design Guild, and usability.gov.

Planning for the Report

Before beginning to write the report, certain considerations need to be made:

1. Determine the needs and purposes of your key stakeholders.
2. Develop a report outline and present it to stakeholders before further development of your report.
3. Revise the report outline and all other report plans to incorporate key stakeholder suggestions.

Writing the Report

The following points represent the common sections of a report as reported by Kuniavsky, Bruner’s Foundation, SAP Design Guild, and usability.gov.

1. Subject program description
2. Clear statement about the evaluation questions and the purpose of the evaluation.
3. Description of actual data collection methods used
4. Summary of key findings (including tables, graphs, vignettes, quotes, etc.)
5. Discussion or explanation of the meaning and importance of key findings
6. Suggested Action Steps
7. Next Steps (for the program and the evaluation)
8. Issues for Further Consideration (loose ends)

A link to a template for a report is provided at the end of this page.

Bruner’s foundation mentions very useful strategies to avoid while writing a report:

1. Including problems with the methodology in the findings section
2. Reporting both numbers and percent at the same time
3. Listing all the responses of the data collected
4. Compartmentalizing the results
5. Feeling compelled to use all the information collected
6. Including conclusions that are not developed by the findings.

Therefore, the presented report represents the outcome of our data collection. Careful reporting is integral to the design process and to the modifications that need to be made to the product.

Related links:

Reporting: Turning data into information: http://www.sapdesignguild.org/editions/edition2/overview_edition2.asp

Usability Reports: http://www.usability.gov/templates/index.html#Usability

Using Evaluation Findings: http://brunerfoundation.org/ei/docs/EvaluativeThinking.bulletin.6.pdf

Organizational culture consists of shared beliefs and values established by the organization’s leaders and then communicated and reinforced through various practices (Society for Human Resource Management, 2008). The purpose of fostering this culture is to ultimately shape the employee’s perceptions, behaviors, and understanding related to the company. This culture represents a critical factor in creating organizational value, as culture and talent have the ability to drive performance and shareholder value (Rothrock & Gregory, 2006).

I would like to focus on the concept of user-centered culture. What I like about this concept, is that it centers the culture of the organization more specifically on the end result: the needs of their consumers. I’m not implying that companies are ignoring the needs of their customers, but I can see how organizations would tend to focus their actual culture on the every day practices taking place in the office.

I could use my current situation as an example. I work for an organization that provides training and certification for HR Professionals. Obviously, the success of this company is completely driven by the effectiveness of the training they provide to their customers and its perceived value. This company’s culture upholds ideals such as excellence, integrity, quality and collaboration. These are all ideals that can be applied to the user experience, however I do not feel like that is currently taking place. Instead, these ideals are being applied to the employees and the quality of their production.  Does anyone else experience the same thing at their workplace? In the end, an organization’s culture will push us to simply work to please our bosses, rather than remembering who we are actually designing these products for.

This sounds like a great idea in concept, but there lies a great challenge within to actually shift people’s thinking within the organization and pushing them to place the perspective of the user ahead of their own (Kuniavsky, 2003). I think that attempting to change an organization’s culture represents a key issue in the corporate world today. With companies having decades of success being built upon a culture that has never changed, a shift in thinking regarding this culture would take a great deal of buy-in and research to prove the success of this potential change. This is where effective user research is critical, as it will help provide proof of the gaps between the capabilities of a product and the unsatisfied needs of the user (Kuniavsky, 2003). Having that information presented serves as a starting point to creating a user-centered organizational culture.

References/Resources

Kuniavsky, M. (2003).  Observing The User Experience.  San Francisco, CA: Morgan Kaufmann.

Rothruck, C., Gregory, D. (2006) How Corporate Culture Affects Organizational Value. SHRM Whitepaper.

http://connectedsocialmedia.com/1766/making-corporate-culture-an-asset-forresters-bruce-temkins-tips/

http://www.hieutrung.com/usability-analysis/sharing-ownership-of-ux-user-experience/

http://www.uxmatters.com/mt/archives/2006/03/why-ux-should-matter-to-software-companies.php

http://webuser.bus.umich.edu/cameronk/culture%20book-chapter%201.pdf

http://www.usefulusability.com/the-5-models-of-corporate-user-experience-culture/

http://net.tutsplus.com/articles/general/three-steps-to-fostering-a-user-centered-design-culture/

http://www.shrm.org/TemplatesTools/hrqa/Pages/Doesacompany%E2%80%99scultureaffectorganizationalperformanceandeffectiveness.aspx

Outsourcing work and using consultants can be beneficial in many situations. Whether an organization lacks the proper professionals, is looking for outside-the-box ideas, worries of impartiality, or lacks the proper technology, there are many voids outsourced work can help fill.

Outside of the traditional reasons to outsource work (e.g. no internal know-how), my research led me to several unique ideas. Accessibility of an application could be a large challenge to many of the mobile AR apps we are producing. While most of us have a basic understanding of technology related accessibility issues, the divers and broad accessibility standards and laws are complex and confusing. If we were to sell our application to a government agency, would be meeting appropriate compliance (such as 508)? Outsourcing user experience research in the accessibility arena is not one I had previous thought about.

My user experience research also led me to unique technologies used in the user experience field. Eye tracking is a highly specialized technology that uses cameras to track eye movement on an application. This Youtube video produced by the User Experience consulting company Etre shows how the process works: http://www.youtube.com/watch?v=lo_a2cfBUGc .  Most small to medium companies (even ones with in-house user experience experts) do not have this type of technology, but could definitely benefit from better understanding reactions to their user interface.

Additionally, organizations might have internal politics at play. I have worked with developers who take  criticism (even very positive criticism) very personally.  Etre does an excellent job of describing the work they do in this sound-bite “internal politics often crush objectivity.” Using a consultant would help to deliver “unbiased” results free from office politics.

While outsourcing user experience work can help solve many problems, managing the work to be outsourced is extremely important. As Kuniavsky explains on page 448, consultants cannot perform miracles. Rather, timing is extremely important. Outsourcing a product before it is complete (riddled with bugs or not yet complete) could yield useless results. Try to send out a product at the last minute and will find it hard to solidify a proper understanding of the product with the consultant.     

Also we must be sure we know what we want. A consultant can help us better hone in on our needs, but  as Kuniavisky said “calling a user test a focus group is more than just a faux pa.” Confusion can reign and inadequate results can be delivered when we do not know what we want or need. 

I’m wondering what ideas other groups have regarding outsourcing their UE work. Do you think your UE findings would differ? What resources would you take advantage of that you do not have available in-house? What would some of the downfalls to outsourcing the UE work?

References and Resources:

Kuniavsky, M. (2003).  Observing The User Experience.  San Francisco, CA: Morgan Kaufmann.

http://www.youtube.com/watch?v=lo_a2cfBUGc

http://www.etre.com

http://www.designinginteractive.com/user-experience/why-you-should-outsource-usability-testing/

http://www.softwaremag.com/focus-areas/application-development/commentary/outsourcing-the-user-experience/

I would like to focus here on the importance of competitive intelligence. Competitive Intelligence (CI) provides a bird’s-eye view to allow for predictions and forecasting, facilitating design decision-making.  It’s a snapshot that to help you strategize against the competition. With competitive intelligence, you improve your strategy and performing better against competitors.  With competitive intelligence or competitive benchmarking, you gather enough to know what’s going on in a company but you do not have all the details, nor do you need them right away. You can gather enough information to draw a reasonable conclusion to act immediately.

Competitive intelligence helps companies  define and understand their industry and identify the competition’s strengths and weaknesses. When it’s done well, it can help a company anticipate moves and act swiftly.  An example of competitive intelligence gathering is when major airlines change their fares to respond to the competition.

With the internet, the rapidness at which an organization can gather competitive intelligence is astounding.  It has dramatically accelerated the speed at which companies can find competitive intelligence.  While digging for competitive intelligence, it is important to seek out information not readily available or which falls outside of the competitor’s realm–emerging technologies, non-customers, or unused markets. 

 A designer can learn more about the possible impact of the competitive intelligence gathered by asking users about it during the data collection stage. Perhaps it is an upcoming feature to a website or application which although not currently available, can be described in as much detail as possible to a user to get a reaction and anticipate its success or failure.

 Reference: Kuniavski, M. (2003). Observing the user experience. San Francisco: Morgan Kaufmann Publishers.

 Interesting links:

 http://www.combsinc.com/handbook.htm

 http://www.fuld.com/Tools/

http://www.businessinsider.com/competitive-research-2011-3

 http://www.yadvertisingblog.com/blog/2011/03/25/19-tools-for-competitive-research/

 http://wiki.telfer.uottawa.ca/ci-wiki/index.php/Ethics_in_Competitive_Intelligence#.22The_Bible_of_BI_Ethics.22

 Keywords:  competitive user experience research, competitive intelligence, benchmarking

Tomatoes generate debate, cause perplexity, and sometimes confusion regarding their status as a fruit or a vegetable. Scientifically they are a fruit. However, tomatoes hang out with the vegetables at the store, and frequently are displayed on educational charts as vegetables. When we want to do a competitive analysis of tomatoes the question becomes: Who are the competitors, are they apples and oranges or are they carrots and beets? The answer depends on the context we are using or comparing tomatoes.

Augmented Reality (AR) has elements, functions, and operations of several well-known and established technologies and applications. AR also interacts and perhaps even competes with the real world. This makes it difficult to determine who the competitor is and how to effectively conduct competitive research and analysis. The determination of competitor is driven by the AR application we are designing and who we are designing it for. This search for identity and how AR is benchmark is the subject of AR sites and technology blogs.

AR is an emerging and rapidly changing technology without a diverse and large community of competitors. In each of our team projects the competitors are both in the realm of technology and human physical interaction. This makes a side-by-side or point-to-point comparison difficult. However, the functionalities we are designing and conceptual context we are using do provide some anchor points sufficient for us to use the competitive analysis process.

The pearls of wisdom that roll out of Kuniavsky (2003) discussion on competitive research and analysis are to embrace the process and look in the mirror. Merging Kuniavsky concepts with fundamental elements of good investigation techniques and some AR blogs I came up with these thoughts and questions as essential to an effective functional and operational competitive analysis:

-          Define our own expectations of usability, functionality, and operations. Openly listen for others perspectives and being willing to adapt.

-          Look at other products or activities that have direct or indirect functions similar to our own products. Compare the functionalities based on what they provide and don’t provide the user.  (Watch and listen)

-          Ask the same tough questions of other products we asked ourselves about user interface, functionality, user expectations, feasibility, viability, and sustainability.

-          What does our AR product or the other product provide the user that other current technology does not?

-          What about our AR product or the other product is most valued by the user and why?

-          What are our AR products or the other product’s issues of sustainability and relevancy? (Are we at risk of a “one-hit wonder”?)

-          How does a product and company gain the trust of the user?

-          Don’t make the data try to fit. The difference or often the competitive advantages for someone.

-          What are strengths, weakness, opportunities and threats (SWOT)? (This is the cornerstone of a good decision matrix)

The research, analysis, reflection, and adaptation are constant and ongoing processes to develop and sustain a cutting edge product. Unlike the tomatoes’ crisis of being trapped between two worlds AR products will eventually stand-alone with clear benchmarks.

Bottom line: The outcome of the competitive analysis is actionable information to improve our product.

Some Helpful Resources:

1.  Strength, Weakness, Opportunity, and Threat (SWOT) is an excellent process for competitive analysis. SWOTs organize and focus well-defined information and identify ill-defined domain or problems.  (I would recommend typing into a search engine “SWOT templates” and then select “images” display. This will provide you a wide display of the different usage and types of SWOTs).

SWOT Analysis Worksheet: http://www.mindtools.com/pages/article/worksheets/SWOTAnalysisWorksheet.pdf

Competitive analysis using SWOT: http://office.microsoft.com/en-us/templates/competitive-analysis-using-swot-TC006078428.aspx?CTT=5&origin=HA001165603

Mind Mapping: http://www.mindgenius.com/

2.  AR discussion forums/sites on challenges currently facing AR and updates on new AR developments.

Futuristlens: http://www.futuristlens.com/category/augmented-reality-2/

Augmented Planet: http://www.augmentedplanet.com/2010/06/the-mobile-augmented-reality-competitive-landscape/ (This site has good diversity of information and links to other AR relevant sites).

Kuniavsky had a lot to say and lots of methods to do it.  Through a feature audit, Group 1 needs to identify the key features of our product and identify any competitors out there that may have those same features and attributes.  In addition, once we have discovered some of those competitors we need to find out what features and/ or attributes the competition have that ours doesn’t that make them successful. If time permitted, each of our groups should apply some of the techniques we are using in our rounds and apply them to our competition.  Kuniavsky discusses this in his section on competitive analysis techniques.  Some examples are recruiting, contextual inquiry, focus groups, surveys and usability testing.  Some observations from our focus group discussion in Round 1, our participants began to talk about some key features they would like to see in our product.  Most began describing features on the I-touch and I-phone functionality.  I believe it was because they experienced our prototype on a i-touch mock-up and were expecting similar features.  In the end, the goal of competitive research is to gather enough information to make a meaningful comparison between your product and the competition.

I found a great website entitled, “How to Gather Competitive Research” and it has tremendous amounts of information on gathering competitive research.  According to the author, it is essential to define what competitive questions you want to ask and what kind of research data you need to make meaningful insights. In addition, she shares an acronym that apparently is used in competitive research.  SWOT – Strengths, Weaknesses, Opportunities and Threats.  This helps me to formulate aspects of my research strategies into our product and the competition’s.  As Kuniavsky stated, “Everywhere the competition’s product is more effective than yours reveals a weakness in your product, and vice versa.”

Keywords:  competitive user experience research, competitive intelligence, SWOT analysis

Interesting Links:

http://www.bnet.com/article/how-to-gather-competitive-research/60253

http://www.infrasystems.com/writing-competitive-analysis.html

http://www.inc.com/guides/2010/05/conducting-competitive-research.html

http://www.businessballs.com/swotanalysisfreetemplate.htm

http://www.youtube.com/watch?v=pGH_GSSyiVA

http://www.youtube.com/watch?v=L65wl6DE5Ik

It is important for designers of any web-based design or product to become familiar with log files as understanding their purpose will be beneficial to the long-term success of your product or design.  Log files are used to collect data.  They have the capability to reveal how items were requested, where the request came from, browser and operating system information along with many other details about that particular connection session.  It is up to the designer to determine which information should be captured for general inquiry or design improvements. When considering log files to gain insight into the user experience, the type of log file that you choose will yield different data.  You must be clear on the type of information that needs to be obtained to accurately understand the user’s experience—regardless of the element used to obtain the information, whether it is access or server logs, caching or using cookies.

Another technique for determining a user’s experience is the clickstream analysis approach.  In this approach, you will be able to determine how users move through your site, it can yield collective patterns related to how people move through the site.    Clickstream analysis can reveal statistical data related the next pages and average paths elements.  Clickstream can also be used to assess the purchase path and will track shopping cart abandonment.

In Group 1, in developing geometry lessons using an itouch application and augmented reality, the use of log files and clickstream would be instrumental for teachers and school administrators to observe students’ pattern of behavior when exploring lesson etc.  Since the information will be on the school’s network and the population of the sample size is low, minimal effort will be needed to analyze the data.   Log files and clickstream data can provide vital information and statistical reporting to school administrators for additional buy-in regarding the possible use of augmented reality in other subjects.

Data mining refers to the extraction of hidden predictive information from large databases, is a powerful new technology with great potential to help companies focus on the most important information in their data warehouses. Data mining tools predict future trends and behaviors, allowing businesses to make proactive, knowledge-driven decisions (http://www.thearling.com/text/dmwhite/dmwhite.htm).  This can definitely be useful in the scenario in our group.  Having concrete information to show the school administrators will certainly initiate more interest.

One related article that I found printed in 2007 from the New York Times describes how data mining through the use of technology has been used in various scenarios.  The authors stated how data mining was an emerging phenomenon, yet it seems to be directly in-line with what we are discussing at this time.  Many more organizations are using data mining to

http://www.nytimes.com/2007/05/20/business/yourmoney/20compute.html

 In this digital, augmented, information age,  the use of log files, clickstream analysis and data mining can prove beneficial to the product designers however from a consumer perspective, I am not sure that I want  too much information known about my purchasing behaviors etc.,  as it could have negative implications in the future such as constant cost increases.

Kuniavsky (2003) emphasizes the importance of user testing during the iterations of design development: feedback gleaned from user testing can provide: warning flags, insights into people’s perceptions and behaviors that can inspire new ideas for development, or confirm what works and is liked, or what doesn’t and what people don’t like. It offers the chance for developers to refine their products, release them, or recognize when to cancel them altogether. It also allows developers to quantify improvements or degradations between iterations (Nielsen, 2001b).

So, how is user-testing structured? There are a number of elements to incorporate or consider when designing user testing, each worthy of its own discussion: selecting the users to test, the environment in which to test, what features to test, and what metrics to use.

What are metrics? Turns out, metrics is a bit of a tricky word. I had thought it was simply “measurement.” But I was wrong, or at least partially wrong. According to a number of different online dictionaries, (elook.org, thefreedictionary.com) a distinction is made between the two: “measurement” is the extent or size of something, where “metric” is the system of measuring something (although Merriam Webster makes no such distinction). As I thought about this (and it took a while to sink in), the distinction started to make sense to me, particularly as it applied to my task at hand here: writing a blog post about usability testing metrics.

What I finally arrived at is that usability testing metrics is the field, or subject, or tool, that addresses how to systematically go about measuring the user experience. “Systematically” is the key word: to accurately measure, and compare user experiences, between users or over time or both, some standard evaluation tool is needed. It is more than just the measurement of the size or extent of something.

As our readings (and other available literature) on the topic suggest, there are several standards, or “metrics” that are generally used in measuring usability. The most commonly referenced, as they are spelled out by the ISO (International Organization for Standardization) include three broad metrics of usability: effectiveness, efficiency, and user satisfaction (impossible for me to directly cite, cited by many, including Alshamari & Mayhew, 2009 ). I found a variety of others which are similar in nature, where all share rating effectiveness as part of the usability metric, but where there are notable differences between them as well:

• Our government (US Dept of HHS, 2011) spells out a slightly different set of five areas to be measured in usability testing: ease of learning, efficiency of use, memorability, error frequency or severity, and subjective satisfaction. The inclusion of ease of learning and memorability stands out as new or different from the ISO standards.
• A usability consulting firm in Canada uses five key metrics: effectiveness, efficiency, expectations (both the user’s and the product developer’s), emotions, and experience (Meditative, 2011). I think these criteria break out “satisfaction” into three separate categories for measurement: expectations, emotions and experience.
• Nielsen (2001) suggests a single, “simple” metric for usability: user success rate, or, the “percentage of tasks that users complete correctly,” which is essentially a measure of effectiveness.
• Although, Nielsen (2001b) also identifies in other articles or postings a different metric of four measurements that are similar to the ISO ones: success rate (in ISO, this would be effectiveness), time a task requires (ISO equivalent: efficiency), error rate, and satisfaction.
• Consultants (Lewis & Sauro, 2011) from a quantitative research firm that specializes in measuring usability identifies “SUS” which is the System Usability Scale as one tool for establishing a user’s “perception of usability.” They make the case that although kind of old, it is a relevant, reliable and valid tool, which actually measures both satisfaction and learnability, and which also moderately correlates to other measures of usability such as effectiveness and efficiency. It is a ten item questionnaire developed by DEC back in the 80s which asks a test user the following questions:

1.        I think that I would like to use this system frequently.

2.        I found the system unnecessarily complex.

3.        I thought the system was easy to use.

4.        I think that I would need the support of a technical person to be able to use this system.

5.        I found the various functions in this system were well integrated.

6.        I thought there was too much inconsistency in this system.

7.        I would imagine that most people would learn to use this system very quickly.

8.        I found the system very cumbersome to use.

9.        I felt very confident using the system.

10.      I needed to learn a lot of things before I could get going with this system.

In looking for information on usability metrics, I discovered that, as usual, my initial understanding of an idea was way too simplistic, and that though painful (really, journal articles on metrics are tough to wade through) , I developed a much fuller understanding of what the metrics are trying to capture. For sure it has helped me to understand how to better structure the usability testing for our group.

References:

Alshamari, M. & Mayhew, P. (2009). Technical review: Current issues of usability testing. IETE Technical Review, 26(6), 402-406.

U. S. Department of Health and Human Services. (2011). Usability.gov. Retrieved from http://www.usability.gov/basics/index.html.

Kuniavsky, M. (2003). Observing the user experience: a Practitioner’s guide to user research. San Francisco, CA: Morgan Kaufmann Publishers.

Lewis, J. & Sauro, J. (2011). The Factor structure of the system usability scale. Retrieved from http://www.measuringusability.com/papers/Lewis_Sauro_HCII2009.pdf

Meditative: The Results People. (2011).Website usability cheat sheet. Retrieved from http://www.meditative.ca.

Nielsen, J. (2001). Success rate: The Simplest usability metric. Retrieved from http://www.useit.com/alertbox/20010218.html

Nielsen, J. (2001b). Usability metrics. Retrieved from: http://www.useit.com/alertbox/20010121.html.

Research techniques we have discussed, read, and practiced are rooted in ethnographic research methods.  If you have conducted a task analysis and examined the culture that your target audience works in, this is ethnographic research.  If you have conducted an in-depth interview with a target audience member about the context of their work and the internal processes they follow to perform their tasks, this is ethnographic research.  A recurring idea throughout Mike Kuniavsky’s book (2003) is to observe and gather information from the users in order to facilitate the process of design research by gaining a strong grasp of the contexts within which the users operate.

When I think about it further, ethnographic research is essential to the design research process.  It is not enough to ask users what features they want a product to have.  Ethnographic research gives the researcher/designer the ability to use techniques like interviews, contextual inquiry, focus groups, and others to understand in-depth the user context(s).

One example of an application of ethnographic research is the design project that the IDEO organization (http://www.ideo.com/) undertook in redesigning the checkpoint experience.  The goal of the redesign was to make the airport security more effective and therefore make the environment for the airport passengers and airport and TSA personnel calmer (http://www.ideo.com/work/tsa-checkpoint-evolution).  IDEO used ethnographic research to observe the physical space and interview individuals with different roles within this environment to achieve a greater depth of understanding about the culture of airport checkpoints.  The organization then used their ethnographic research findings to go through the process of developing and refining prototypes of the physical layout from the physical lobby to the checkpoint.  Additionally, IDEO developed a training curriculum for Transportation Security Officers (TSOs) that focused more on the understanding of “behaviors, people, and security measures.”  Through the ethnographic research, IDEO is implementing modifications in the environment to spur change within the TSA culture to focus more on human intelligence and passenger interactions, which will hopefully improve security as a result.

I have found that ethnographic research is broadly defined and is still going through the process of gaining mainstream acceptance despite its use in a range of contexts.  Wilcox (2008) explains how another application of ethnographic research can be used to make design decisions (http://www.dscience.com/articles/Ethnographic%20Article.pdf).  He uses the design of medical devices to illustrate principles of achieving validity in the ethnographic research process.  The author posits that ethnographic research can be done with more of a quantitative foundation to facilitate the design decision making process with more validity.   The author proposes a form of quantitative ethnographic research as an alternative method to surveys.  Gilmore (2002) emphasizes the qualitative aspects of ethnographic research and uses these aspects to make the case that the rich narratives, which are engendered, strengthen the design process (http://www.davidjgilmore.com/professional/writings/185.pdf).  The author makes a distinction between ethnographic research and market research by indicating that “… there is a philosophical difference between research conducted to inspire design and that conducted to validate design—one is about idiosyncracies and little details and the other is about averages and generalities.“  He states that it is important to highlight the different goals of the two forms of research.  Additionally, Gilmore (2002) states ethnographic researchers view every user as having valuable information to contribute to design research.  He contrasts this with market research, which aims to find so-called “average” users for the purposes of observing and gathering information.

To me, it is important to recognize that ethnographic research is continuously evolving.  Each method must be evaluated based on its merits and contributions to the design research process.  The goal of ethnographic research is to understand the users and their cultures at a deeper level.  Intuitively, I feel that this goal contributes to the continuous evolution of the methods used in ethnographic research, which improves the products that are designed by a wide variety of designers.

References:

Kuniavsky, M. (2003).  Observing The User Experience.  San Francisco, CA: Morgan Kaufmann.

IDEO (http://www.ideo.com/)

IDEO TSA Checkpoint Design Case Study (http://www.ideo.com/work/tsa-checkpoint-evolution)

Wilcox, D. (2008). Ethnographic Research and the Problem of Validity. MD&DI. February 2008 edition (http://www.dscience.com/articles/Ethnographic%20Article.pdf)

Gilmore, D. (2002). Understanding and Overcoming Resistance to Ethnographic Design Research. Interactions. Ma y + june 2002 edition (http://www.davidjgilmore.com/professional/writings/185.pdf

Kuniavsky, suggests using not fully functioning prototypes ( paper, wireframes etc.) to develop usability metrics.  At first this seemed a little wrong to me.  How are you supposed to test something that doesn’t work the way it is supposed to yet?  It would be like trying to determine if the food you are eating is delicious before the server brings it to you in a restaurant.  However, looking at the new definition of usability I saw that I was only concerned with one aspect of usability; interaction design.  All of the other methods; visual design, information architecture, navigation design, and user interface design can all be done pre-production.

This amount of variation in evaluation means that usability can be checked throughout many stages of the design process.  In fact, both in the usability.gov website and the power point presentation resources it states that often times a finished product is not necessary to do gather usability information. It is possible design and evaluate a product without even actually physically creating it. This is a fairly new idea but it is one that is gaining popularity and is especially important to our class.

As described above, there are more specific sub areas of usability for which different tests and evaluation are needed. However this does not mean that there are not tools for evaluation that can be used too look at usability as a whole. It is called TAFEI or Task analysis for Error identification. The identification of errors is an analysis  that deals with all aspects of usability . TAFEI allows you to do this look at the all the information that the user would use to operate or learn a system or device and to determine ahead of time where errors may occur.  This is very useful with the repetitive process of design because it allows you to see errors before stages of production and other analysis begin.

When looking at usability it is important to understand what testing metrics work for the sub areas of usability and which ones work for all aspects of usability. Knowing this difference and being able to apply it will make designing evaluation tools and reiterative design that much easier.

If anyone is interested in TAFEI you should check out these books.

Kirwan B.,& Ainsworth L.K. ( 1992). “A guide to task analysis”.
Boca Rotan, Fl.  Taylor and Francis Group LLC.,

Stanton, N., Hedge, A., Brookhuis K., Salas S. & Hendrick, H. (2005). “Handbook of Human Factors and Ergonomics Methods”.
Boca Rotan, Fl.  Taylor and Francis Group LLC.,