Posted by: bbannan | March 7, 2012

Survey design – Jennifer P.

Kuniavsky (2003) warns the reader about the importance of survey design, saying that poor design can lead to “ask[ing] the wrong people the wrong questions, producing results that are inaccurate, inconclusive, or… deceptive” (p. 304). Chapter 11 spends time introducing the reader to survey research. In a recent webinar, Phillips (2012) shared that survey questions should be written with the respondent in mind. Can they answer the question? Do they understand the meaning of the question? Do they know the answer? Are they able to answer in the terms required? And are they willing to answer the question? It’s very easy to write questions that fit our own schemas and interpretations; however, it’s much more difficult to write for other people. For the purpose of this post, I’ll be talking about how to write good questions that will get what you want to know.

In his text, Kuniavsky (2003) offers a few suggestions for writing good survey questions. I’ll briefly review them here (as you’ve already read them!):

  • Don’t ask respondents to predict their behavior; focus on past behavior
  • Don’t ask negative questions as the words “not” or “never” can be easily overlooked by the respondent
  • Keep questions simple and don’t overload with additional concepts
  • Be specific; avoid using broad terms such as “sometimes” or “rarely”
  • Provide options that don’t make assumptions about the respondent or exclude respondents
  • Keep the language of questions consistent throughout the survey
  • Make sure the questions are relevant to the audience to reduce mortality
  • Use Likert-like scale questions
  • Create follow up questions for certain answers (use “logic”)
  • Include opt-out options such as “not applicable” or “none”
  • Give respondent an opportunity to leave comments

Coming from a perspective of the American Evaluation Association, Ritter and Sue (2007) describe good questions as clear, short, unbiased, and relevant.

  • Clear means that the respondent’s understanding of the question matches the surveyor’s intent. Does the question convey what I want it to? A common obstacle to writing clear questions is the use of jargon or difficult words. Be sure to explain jargon (e.g. ISD means instructional systems design) and that the language level matches your audience. But be weary of simple yet broad terms that may have multiple interpretations (e.g. does the term “weekend” include Friday or is it just Saturday and Sunday?).
  • Ritter and Sue suggest keeping a question short, or no more than 20 words. Long questions can be intimidating to respondents and also run the risk of becoming overly complex and “double-barreled” (and confusing).  An example of a double-barreled question would be, “the materials used in the training session were visually appealing and helped me better understand the content”. Perhaps the materials were visually appealing but actually did not help the learner. How should the respondent answer?
  • Unbiased refers to questions that do not mislead respondents to answering a specific way (also called loaded or leading questions). For example, a biased question for a teacher would be “Are you interested in your students?” Most respondents would likely say “yes” because it would reflect poorly on them if they didn’t.
  • Just as Kuniavsky (2003) indicated, questions must be relevant to the respondents. In order for a question to be relevant, you must be sure to properly identify your target population. Questions that are irrelevant to respondents may be skipped because they don’t know the topic well enough to answer. While others may still choose a favorable answer so that they appear well informed on the topic even if they are not.

Perhaps the best advice offered by Kuniavsky (2003) is to pilot the questions before deployment. It can be difficult to understand your own biases and mistakes. And as the author mentioned, it’s even more difficult to re-run your survey. This is especially true if you already have a small target population to begin with. Last summer I made that mistake with a technology survey for the employees of my company. I realized after I had the data that the questions weren’t really asking what I intended them to. Unfortunately I can’t re-run the survey because I’ll be targeting the same group of 350 employees. In another example, my company surveyed employees about the recent merger. One of questions was not appropriate for the Liket-like scale used on the survey and therefore was confusing. The results showed this confusion as the responses were split across the scale. Through a pilot, you can come closer to ensuring the questions will be gathering the information that you want to know.

 

References

Kuniavsky, M. (2003). Observing the user experience:  A practitioner’s guide to user research. San Francisco, California:  Morgan Kaufmann Publishers.

Phillips, P. (2012, February 28). Evaluation basics:  Four challenges that make or break survey research. Webinar viewed from http://www.trainingmagnetwork.com/main/home

Ritter, L. A. & Sue, V. M. (2007). Using online surveys in evaluation. New Directions in Evaluation, (115), p. 29-36.

Advertisements

Responses

  1. Jen, this is some excellent advice for writing effective survey questions. The tip that stuck out to me the most was the issue of clarity. Group 5 experienced this issue when we revised the first draft of our user survey. We had a potential answer that was something like “less than 2 times per week” and a second choice that said “rarely.” This could have skewed our results if particpants viewed the first option as being the same as “rarely” and they were forced to pick only one answer.

    I also really enjoyed how you related the importance of piloting a survey to your own professional experiences.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: