Posted by: bbannan | March 22, 2012

Modern Diaries and Perpetual Beta Testing – Melissa

In chapter twelve, Kuniavsky (2003) details the benefits and drawbacks of ongoing user research through the use of diaries (both structured and unstructured) as well as advisory boards, beta testing, and telescoping. Many of the benefits and drawbacks that exist with these ongoing, user research relationships mirror those discussed in earlier chapters in the text. As such, one will not be restating them here, but instead will discuss two other related items—namely, a more modern approach to diaries and perpetual beta testing and the user/researcher relationship.

When the text was published in 2003, the technological landscape was very different. Kuniavsky touches on the use of journals, electronic and paper forms, and emails to collect users’ diary entries. While these methods still work, more options are available today. One believes that if this chapter were written today, Kuniavsky would discuss how blogging software like Word Press, Posterous, and Tumblr (among others) could also be utilized to more efficiently capture data from the user. While accessible from a web interface, these three platforms also accept posts via email which may lessen the anxiety less technologically adept users might feel when trying to record their experiences and observations. An August 2010 USIT blog post describes setting up and using Posterous for user research and Christopher Khalil’s 2009 presentation briefly touches on using other technologies like Twitter, Facebook, or video to perform the research. Khalil posits that using these technologies is now more natural than writing traditional diary entries. If one were participating in user research, a video diary would be one’s preferred method as talking out loud seems more natural and would thus encourage more openness and disclosure. Bryan (2012) echoes these statements and describes video diaries as “[participant] lead” user research instead of researcher guided, which is important because every user is different and users’ relationships to the product are unique to each “[user’s] goals, job, and the other tools they use” (Kuniavsky, 2003). Researchers also benefit from the adoption of the aforementioned technologies in user research. Data is ready to be gathered and analyzed as soon as the user posts to the blog and researchers can be notified via RSS feeds and services like Feed My Inbox ( that new observations are available for their review. In video diaries, the researchers can see the users’ environments and gather data about the context that would not have otherwise shown up in written diaries. Setting these benefits aside, researchers need to determine which data collection methods best suit their research process and not pick a method just because it is new and shiny. The research focus should be on gathering valuable information about the experience as the user interacts with the product in context and in some cases, paper diaries or online forms may be the way to go.

The idea of beta testing as user research was not familiar prior to reading Kuniavsky, but one’s familiarity with beta testing is high. Many web services seem to be in “perpetual beta” (O’Reilly, 2005) or beta testing takes place for a long time. For example, Gmail, launched in 2004, exited beta testing in 2009 (Lapidos, 2009). O’Reilly (2005) states that “users must be treated as co-developers” when products are in perpetual beta, which mimics the approach taken in this course wherein prototype revisions are made after several rounds of user research. Kuniavsky presents beta testing as a usability testing option that occurs after development, but before the product is released. If O’Reilly’s statement regarding users as co-developers holds, could the case be made that beta testing is just another name for what the groups in the course have been doing all along, albeit with less than fully functional prototypes? Once sees both sides of the argument, but is curious as to others’ opinions.

While long-term user research is not feasible in a sixteen-week course, Kuniavsky’s text presents the different types of user research methods that could be used if more time were available. In closing the chapter, he stresses that user research should not be solely focuses on initial experiences because products and users’ relationships with those products change over time and that products can be further developed to “grow with the knowledge and needs of its users” (Kuniavsky, 2003).



Bryan, P. (2012, January 23). Video diaries: a method for understanding new usage patterns. Retrieved March 17, 2012, from UXmatters: insights and inspiration for the user experience community:

Khalil, C. (2009, September 04). The new digital ethnographer’s toolkit: capturing a participant’s lifestream. Retrieved March 17, 2012, from

Kuniavsky, M. (2003). Observing the user experienc: a practioner’s gude to user research. San Francisco: Morgan Kaufmann Publishers.

Lapidos, J. (2009, July 07). Why did it take google so long to take gmail out of “beta”? Retrieved March 16, 2012, from Slate:

O’Reilly, T. (2005, September 30). What is web 2.0: design patterns and business models for the next generation of software. Retrieved March 15, 2012, from O’Reilly:

USiT. (2010, August 13). Using posterous as an online cultural probe (user research diary) – USiT blog. Retrieved March 17, 2012, from USiT blog – user experience at news digital media:



  1. Melissa, A very insightful and well-written post on beta testing and user research – thank you! I like the ideas you’ve added to the Kuniavsky chapter regarding using modern tools such as blogs for diary posts, twitter, and video diaries to capture user-led research. Regarding the use of diary posts, if all members participating in the user research followed each other it might lead to good interactive discussion via comments and even more useful data. Though, I imagine you’d need to caution your users to not read other’s posts until they had recorded their own feelings on the subject matter.

  2. Melissa,

    I completely agree with you that new technologies would improve the diary process for users and researchers. It’s easier for users/participants, since it’s convenient and can be accomplished on the go on most mobile devices, and it’s for researchers to analyze the data since its already digitized. This strikes me as especially true for qualitative research experiments.

    Nielsen uses Twitter in his international usability research, essentially as a diary. Participants are asked to tweet throughout the day as the participate in the research, allowing the researchers to gather the data in near-real time. This was not the only method of gathering feedback that they used, and I wonder if it’s possible to only use Twitter or Tumblr to gather data. I wonder if the character limits would influence the nature of participants’ feedback. I’m thinking of usability research as Kuniavski depicts it, and the importance of the participants thinking out loud. You would certainly get a filtered version via Twitter.

    I suspect that, if we can tweet WWII (, we can successfully conduct a research study using Twitter.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s


%d bloggers like this: