What might constitute “data” broadly defined in design research? Where might we look for information in regard to a design problem’s optimals and actuals? How might we analyze the information that does not necessarily involve formal qualitative and quantitative research analysis?
- Comment
- Reblog
-
Subscribe
Subscribed
Already have a WordPress.com account? Log in now.
Design+Research 
Rossett on pg. 35 suggests researching optimals by looking at actuals. What current efforts are in place by the client and by other entities like the client? How successful or unsuccessful are they?
Formal sources include the following:
–Client and audience interviews
–Research from journals and news articles
–Press releases from the company
–Secretaries/Admin Assistants – the people who filter the calls the executives hear
Informal sources include the following:
–Company blog (if they have one)
–Watercooler conversations
–Public opinion. “Public” meaning asking people you know what they think of the situation. It’s not scientific, but it could yield new perspectives on the issue.
–Internet communities. Are there groups on Facebook or other discussion forums related to the issue? What does a quick scan of their posts reveal?
In analyzing the information, these perspectives help to find the direction (or the scope) of the planned effort. Identifying trends in the data collected above help identify drivers that further analysis will later explore.
By: Terri Ann Guingab on September 10, 2008
at 3:01Z09
Data broadly defined in design research is anything that helps the designer develop a partnership with his client and increases his understanding of client goals and needs. This data can be gathered from observations, interviews, past designs, and industry research. Rossett writes, “Data are what we gather from sources. They include facts, attitudes, opinions, and actions that turn into information when we organize and infuse them with meaning. Data are thus the basis for the assumptions that we cobble together to make inferences and derive meaning (29).” The goal is to “shake you out of your shoes and into those of the other organizations (Rossett 16). Thus, designers must ask questions that draw them closer to the hearts and minds of their clients and ensure that they view problems, scenarios, and events through an extremely critical designer’s eye. In other words, they need to be able to look at current circumstances and determine whether it acts as a driver or barrier, if an instructional design is warranted, and whether it is an opportunity, problem, development, or strategic planning situation (Rossett 18). At the onset, this can only be done by creating intimate relationships with various stakeholders from the organization. Even when the research gathered is untraditional, designers must be willing to “dive deep” and rely on their own prior knowledge, intuition, and beliefs.
The quality of the performance analysis and the reliability of the sources determines a designer’s ability to distinguish what is actual and optimal. If what the manager says is contradictory to what his employees say, then there is a disconnect between and him and his subordinates and the designer needs to bring those discrepancies to the forefront. This is done after the designer has gathered enough resources to reach a verifiable conclusion. Data gathering is the starting point of any effective design.
By: Cammise McInnis on September 10, 2008
at 3:01Z09
In addition to Terry Ann’s comprehensive list, I think another way to gather information about the optimals is to do research on the company’s key competitors and leaders in their industry. There is also a wealth of information in the organizations internal measures such as productiviity, profit, employee satisfaction surveys, current training programs etc. The relevant internal measures are determined by goals of the training intervention. For example, if the course is about increasing the accuracy level of customer service representatives the relevant internal measures would current accuracy rates, error rates, and current training programs. These would be considered formal data sources.
Informal data sources can often tell the real story behind the numbers in the formal data sources. Informal data sources can help direct the designers to the more relevant data sources,
By: Nancy on September 12, 2008
at 3:01Z09
Rossett (pg. 28) suggests using several sources when collecting data. She says (pg. 29) it’s “facts, attitudes, opinions, and actions…” These are all collected.
Places to find data are numerous. Terri Ann mentioned several. Others would be surveys, letters from customers, performance reviews, mission statements, casual conversations, and interviews. This information will give the actuals in order to get the optimals.
I think that it’s difficult to quantify informal data, but that it’s imperative to pay attention to when working on a project. Asking follow up questions about the informal information and discussing it in the project group helps the team get a better perspective.
By: Deb on September 13, 2008
at 3:01Z09
A) “Data” broadly defined is my view is practically anything that you feel, see, hear, taste. Broadly defined means that there are no specifics to it, therefore I could hear someone on the radio talking about a topic and use that as “data.” If I read a blog about a topic, then I can use that as “data.” Anything that allows me to design a better product is worth taking a look at.
B) Optimals: Administrators or CEOs will tell you what they expect to be happening. Go to the “top dogs” in to the organization, to see what is supposed to be happening.
Actuals: Go to the people that are actually doing. See what they have to say about the situations.
After seeing the “should be” vs “the actual doing” then you are able to assess any gaps or malfunctions in communication between the two.
C) Analyzing information that is not necessarily involve formal qualitative and quantitative research could be tricky, but that could act as a “third piece of the triangle” to see if that actual data holds up or not. If all the research says there is a problem, and through the grapevine you hear about the problems, then it is relevant.
The big thing is to look for gaps that can be improved or ask yourself why are we doing this?
By: Scott Allen on September 13, 2008
at 3:01Z09
What might constitute “data” broadly defined in design research?
Data is the “voice” of design. One of the necessary elements associated with a design research methodology is the acquiring of means necessary to either support or disprove a selected path. This would indicate that testing optimals by looking at actuals in terms of scope of available data is virtually without limits. This also indicates that the definition of “data” can be without limits.
Where might we look for information in regard to a design problem’s optimals and actuals?
If we think about it for a bit, almost anything that has association with the path we are pursuing can be considered “data.” Taking this a step further, we might begin to use data that is more closely associated with the path to create a base for comparison, then begin taking other kinds of information and, by comparison, see if we are able to create a more complex picture of the pursued path. By continuing to select, compare and choose the data we find, we should be able to move toward the correct elements that constitute the path we are seeking.
How might we analyze the information that does not necessarily involve formal qualitative and quantitative research analysis?
Often, when beginning a new project with a client as the unknown, two elements come into play. The first is a “feel” that comes from having interacted with numerous previous clients. While formal qualitative and quantitative research analysis is a proven method for arriving at results, the comparisons that experience can allow can be very important. The second comes from a form of rapid prototyping. Using this methodology, in a brainstorming fashion can be useful if there is sufficient interaction with the client to allow the project group to make rapid decisions, involving rejections and acceptances of gathered data.
Where do these thoughts lead?
All right, now that I have said all of that, I have to add an additional question. “Where do we go from here?” No event is a clone of any other when working with client expectations. We approach the problem, not as instructional designers but as detectives, trying to ascertain what elements the answer will contain. We try to see if we will be able to fulfill all, some, or none of the expectations. We look at the time factor, and at the relative importance of things that may be accomplished.
After going through Allison Rossett’s text for the first time, one of the things I came away with was a feeling of balance, speed, and decision. The tools she talks about are those of any instructional designer trained in the Analysis, Design, Development, Implementation, and Evaluation (ADDIE) method. The difference is in the way she looks at each stage. She believes in putting your effort toward getting the greatest return for your effort. Becoming aware of the front-end of the analysis process will, by its nature cause the needs analysis results. In effect, you are multi-tasking. As you move forward, using rapid prototyping, allows you to quickly refine the areas that you have selected to concentrate on as a result of your analysis of what the performance should be and which parts are doable at this time.
Final thoughts (for now)
As with anything, these thoughts are not concrete. Because of this, I will come back to this blog and add to, refine, and perhaps even remove thoughts and ideas I have posted today. Instructional Design is a malleable process, often changing as ideas are tried and decisions are rethought.
By: Ed Lane on September 14, 2008
at 3:01Z09
Another example of collecting are the drivers. According to Rossett (pg. 44), “Drivers define solutions, they tell us what we need to know now and next”.
There are several Drives Rossett lists:
1. Skills, Knowledge, Information
2. Motivation
3. Environment
4. Incentives
By: Monu Harnal on September 15, 2008
at 3:01Z09
Rossett describes “data, broadly defined” as those “facts, attitudes, opinions, and actions” that can be organized to provide meaningful information. (Rossett, p.29) These include the informal and formal data we collect to define the realities of a situation (actuals), identify the ideal situation (optimals) and create our recommendations. This data is helps to define the problem (Rossett uses, “Clarify the effort” [p. 70]) and rule out possible biases such as history, habit, or generalization. Source for this data may also used to test a hypothesis about what is needed and what might work.
Information gathered towards these ends can come from numerous sources. TerriAnn and other students have covered a lot of them above. Formal sources may include human sources, such as experts, focus groups, or the clients themselves; and documentation, such as accident reports, employee reviews, technical specifications, annual reports, data gathered by other organizations, corporate surveys, and even previous training efforts. Informal sources may include conversations from workers or organizational leaders, observations, and “water cooler talk.” Often this information requires validation, but may prove insightful in directing further research.
Informal information can be analyzed by determining if it is significant (aka, in what way, if any, does this information impact this effort?) and obtaining further formal information. The type of information sought will depend upon the type of information given. Affective information might be confirmed or disproved through employee surveys or focus groups, while other barriers may be confirmed through interviews with workers, discussions with management, or observations of the workplace environment.
By: Cassandra Cloud on September 15, 2008
at 3:01Z09
Interesting discussion and great ideas on where to get data! One of the things to also think about when collecting data about optimals is timeframe. For example, the desired optimal performance six months from now could be very different than the long-term optimal vision for five years from now. So collecting data on where the company is going and their overall long term strategic plan and vision is very important to ensure that what ever training is developed will help and not hinder the ultimate long-term goal. This vision may not be documented and as a result we may need to help our clients establish this vision.
I also really liked Rossett’s suggestions about collecting data from multiple sources (triangulation) including both formal and informal data. I loved her comment that data being collected from informal conversations around the water cooler can be useful data. However, I think in addition to obtaining data from multiple sources obtaining data using different forums is also key. For example, during a group meeting sometimes people do not speak as freely or as frankly as they may in a one on one situation. Conversely, seeing staff interact with one another in a meeting can also highlight internal conflicts that need to be resolved for the training to be successful. So mixing up how you meet with stakeholders, employees, etc. can enrich the quality of data obtained.
Overall, as Rossett stated, the key aspect of the performance analysis is to quickly determine whether there are any show stoppers that indicate that training will not solve the issue at hand. I do not believe formal analysis is needed but you do need to look at the problem from all different angles to make sure you have not missed an important show stopper. This can be challenging as you do not always know what you do not know.
By: Susanne Hobbs on September 15, 2008
at 3:01Z09
Excellent commentary going here which made me think about aspects of communication I had not thought about previously, such as informality or formality of communication context, importance of multiple perspectives on the problem and the issue of inherent bias.
Thank you to all who contributed….and the discussion is by no means over 😉
By: bbannan on September 15, 2008
at 3:01Z09
From my perspective, in design research, data is any information collected from a design problem that can be investigated. This data is anything that can be used to formulate hypotheses and theories——for the sake of creating instructional technology models, any information that serves in contributing to a possible technological learning solution to an instructional problem.
By: Saz on September 29, 2008
at 3:01Z09
In class we discussed a variety of things that can be considered data, ways to collect data, and ways to apply data. Data can be statistics, observations, opinions, ideas, etc. and can be collected through surveys, interviews, focus groups, review or research, etc. Even Rossett broadly defines data as, “Data are ways we gather from resources. They include facts, attitudes, opinions, and actions that turn into information when we organize and infuse them with meaning,” (Rossett, 1999, p. 29). We can apply this information to the performance, needs, tasks analyses and so on through a design project. It is the information that will lead the designer to the solution to the instructional problem.
The definition of the problem is determined by analyzing the optimals and actuals, the performance gaps, the needs gaps, etc. What is the perfect end state and what is actually being accomplished? The problem will be defined dependent upon the type of information collected and from where it is collected. If a designer interviews three different people at a business, in three different positions, she is very likely going to get three different definitions of the problem. So other resources are used; policies, observations, documentation, the work model/product, evaluations, etc.
Despite the various types of information that can be considered data, the question that is most interesting is the third one in the original post, “How might we analyze the information that does not necessarily involve formal qualitative and quantitative research analysis?” Rossett (1999) suggests looking at drivers and barriers, motivators, SKAs, the environment, incentives, etc. However, it is the analysis of data that is going to lead to the problem statement(s), AND it is in analysis that the designer has to be the most careful. Yes, the she has to try and prevent the client’s perceptions and lack of objectivity from influencing the design process and final product as much as possible, but she must also contend with her own. Objectivity is key, and what is interesting about data is that it can be manipulated easily to meet the needs, ideas, and preconceptions of the person analyzing it. Some data can be ignored, other data might be deemed unimportant, or information collected from one are, group, or information source might be considered more important.
So, my point is that the designer must be careful and as objective as possible. It is understood that many projects can last a very long time, it is hard to remain totally objective, but some people can be influenced by their own past experience, by the client, or by the end user. I wonder if any of you have a story and experienced a time when an instructional designer lost their ability to be objective in development of a product and how that influenced the outcome. It could lead to an interesting discussion!
By: Pam Johnston on October 18, 2008
at 3:01Z10
In design research, I think that the instructional developer needs to emulate the funnel analogy that Brenda has discussed in class, when exploring all relevant data. “Data” broadly defined, in the beginning of the developer’s research, might mean anything significant that strikes the developer, including both formal (reports, quantitative research results) and informal data (water cooler conversation, general sentiment around the office, etc). The initial data gathering period for the instructional developer is critical, and although the ramp up time to a project may be stressful and provoke much cognitive dissonance for the instructional developer, it is important to go through that process because it paves that way on the road to understanding the client. And, understanding the client and what the client really needs, not just wants, will enable a good solution to be developed that is effective and meaningful.
To get back to the idea of what might constitute data broadly defined, I think that its important that the instructional developer act as a sponge in the first encounter with the client and any members of the target or related audiences. The developer should note everything from office climate, the way offices/cubes are arranged, the temperament of the employees, etc., to the more formal data that the instructional developer deems necessary in the analysis phase of the project (company reports, demographic information, annual reports, etc.).
After that initial data gathering, the idea of what data will be meaningful to the instructional developer becomes less broad. The instructional developer is able to better gauge what information is important and pursue that avenue, as the funnel gets smaller and the approach more focused.
By: Emily Lloyd on October 24, 2008
at 3:01Z10
What might constitute “data” broadly defined in design research? Where might we look for information in regard to a design problem’s optimals and actuals? How might we analyze the information that does not necessarily involve formal qualitative and quantitative research analysis?
This is an interesting questions to answer at this point in the semester as we’ve spend so much time gathering data to inform our design decisions. Since financial issues are such a hot topic recently there is so much data to draw from. Though we haven’t formally documented it anywhere there were a number of data sources that affected our design ideas. News clips, talk shows, the radio djs talking back and forth to each other, neighborhood discussion, and even junk mail have included “data” that influence which training needs are most recognized and of highest priority.
By: Katie Magin on November 7, 2008
at 3:01Z11