Posted by: bbannan | February 4, 2008

Design research and iterative development

Kuniavsky (2003) writes:

“Every design is a tradeoff.”

“Knowing how to make the right trade-offs is difficult. Like a new organism evolving on an island, an idea isolated in a company is exposed only to certain environmental conditions. It only has to face a certain number of predators, deal with a certain kind of climate, adapt to certain kinds of diseases. The only way to know whether an idea will survive outside its sheltered world is to expose it to the environment in which it will ultimately have to live. However, rather than the shock of the waterfall method-where the final product is pushed out into the big bad world and left to fend for itself, – iterative development attempts to understand the environment and predict how the idea needs to adapt in order to flourish before it is released into the wild.”

What “intersections” do you see between the constructs of design research and iterative development?



  1. Design without research is like trying to find your way to an unfamiliar destination without a map.

    Unfortunately, due to cost and time constraints, design research is a vital step often left out of the development cycle. Back in the early days of the Internet, web sites were created just to get a presence out there for the world to see. But as time has passed, there is much demand for analytic tools and evaluation strategies to let the stakeholder know if their design is really working and to indicate what the user really wants. How does a designer expect to be able to design something that “works” for the end user if they don’t know much about that end user?

    Design research is imperative in creating a design that is functional, educational, and usable for the people it is intended to service. The iterative development process and design research are intimately entangled to inform the design, both of works in-progress and those that have hit the market, to make them better and better as more data become available. Like Kuniavsky says, the design can’t work in a vacuum independent of the environment in which it will ultimately live. So, research is imperative in predicting how the design will be received by the “natives.”

    The lesson here is…the tradeoff is that you may spend more money and time in the beginning on design and research but your ROI (monetary, emotional, satisfaction, functional, etc.) in the end should make up for it many times over with a well designed product that “works” as intended for those who use it. You wouldn’t be able to accomplish this effectively without the intersections between research and iterative development.

  2. Corporate Edict, the Waterfall Method and the Design of Public K-12 Education

    It occurs to me that much of the design of public education, at least extending through the secondary level, is based upon a frightening combination of corporate edict and the waterfall method.

    Corporate Edict
    Following a period of vocal dissatisfaction, a superintendent and/or a school or education board may decide the elements of focus for the entire system and then require all others to enact them. These decisions are often based upon limited information and the personal biases of those who make them. The education system then becomes the pet project of an individual or small group and not the research-based design which would create the best learner outcomes.
    Because of the consideration of only a single perspective, the design of any educational system based on corporate edict is doomed to, as Kuniavsky so eloquently put it, “miss its mark, sometimes spectactularly” (p. 29).

    Waterfall Method
    The ability to design extensive plans is a strength for those who administrate education. Making well-reasoned and valid plans, however, is an entirely different matter. Any educator who has read the requirements of the No Child Left Behind Act knows that the waterfall method, with its numerous assumptions about students, teachers, and schools (many of them clearly inaccurate) and inability to modify expected outcomes, is alive and well in public education. It is obvious that the understanding of the legislators who enacted NCLB does not match the reality of schools.
    In most environments, efficiency that exceeds 90% is reason to celebrate. In public education, nothing less than 100% by 2012 will do. What in the universe works with 100% efficiency? And what makes anyone believe that penalizing schools with pre-existing high success rates for failing to make improvements every year will help?

    The Iterative Design Alternative
    Although an iterative design for education would – and does – make sense, this does not seem to occur to those who make decisions about education. Instead, they cling to what has “always” been done and hope for the best. In terms of learning outcomes, this is a poor execution of administrative power. Therefore, despite the fact that I believe strongly in the potential power of public education, I feel that perhaps a move toward privatization would improve its success – and, as a result, overall public satisfaction in the system.

    Kuniavsky, M. (2003). Observing the user
    experience: A practitioner’s guide to user
    research. San Francisco, CA: Morgan

  3. One aspect that I find intriguing about design research is the consideration beyond empirical research to include what goes on “behind the scenes” and examine the factors that contribute to the unique design of an instructional environment. Of particular interest are the often unpredictable “pragmatic, political, and participatory” factors considered in design (Kelly, in press). It is interesting to see how the focus of design can change based on these factors and despite strong theoretical grounding established for a project. As an immersion student, I have observed how difficult it can be to reconcile the needs of the audience versus the wants of the client, and the direct impact these factors have on design decisions.
    Kelly, A.E. (in press) Introduction. In A. Kelly (Ed.), Handbook of design research in mathematics, science and technology education (tentative title). Mahwah, NJ: Lawrence Erlbaum Associates.

  4. As a teacher in the private sector for 16 years and as a teacher and then Assistive Technology Coordinator in the public school system for the past 10 years, I have similar experiences to the ones described by Cheri. I didn’t really know how the edicts were born but felt their weight as they impacted my workload significantly. I also agree that the highly touted NCLB has many flaws that would be helped tremendously by an iterative procedure that involved gathering and analysis of feedback from all in the school system, teachers, parents, and students. New solutions could emerge from problem solving sessions armed with analysis of feedback. Working as a team to solve educational issues using iterative development would be a novel approach that would give voice to those who are most affected by success and failure of schools. We need to hear the perspective of students as they go through the high stakes testing. We need to hear from the parents and the teachers. If we were all honest we would have to see some benefits yet we could also point out many weaknesses that, if considered, could result in positive changes.

    As one who finds possible technological solutions for my district, I have noticed that often a great deal of money is spent without input from teachers and students. Even after training is provided, often the technology sits on the shelves or is under utilized. Perhaps if they were consulted before purchases and adoptions, greater success might result. And I begin to ponder ways I might begin to get more feedback from teachers and students, and use the analysis to adjust the products and situations to better fit the needs.

    Knowing that collecting feedback and data of use, analyzing the data, adjusting the situation and product, is an ongoing, spiraling process is also vital! How often do we tell students that in writing it may take many drafts before a written piece is ready for publishing, yet, am I forgetting this valuable advice if I do not constantly collect feedback, analyze, and adjust? The pattern of iterative design could be valuable in many areas!

  5. I have read several articles lately on design research and each one of them seems to talk about the importance of iterative development or design. Cobb, Confrey, diSessa, Lehrer, & Schauble (2003) in Design Experiments in Educational Research point out that “perspective and reflective aspects of design experiments result in… iterative design. As conjecture are generated and perhaps refutes, new conjecture are developed and subjected to test. The result is an iterative design process featuring cycles of invention and revision.”

    While the iterative design process mentioned in the above listed article is referencing educational research, I have seen the iterative design process when working on a team to develop software and online portals. Known as the System Development Life Cycle (SDLC), the first step was to conduct feasibility study to determine the needs of the users and the data contained in any legacy systems.

    The second step would be to begin the design/development of the software based on the results of the feasibility study. Usually the first part in the design process was to develop a prototype that the client and a test group of users could provide feedback on. Once the prototype had been approved, development of the actual system would begin. Usability testing would occur at various points in the development process to keep the client and users involved and to provide a validity check. Tweaking of the system design would occur based on client and user feedback.

    Once the system design and development had gotten final approval from the client and test group of users, it would be implemented—rolling the final system out to the entire population of potential users. While the system was live, technical support was provided not only to educate the users on the features of the system, but also to allow them to provide feedback on the system. Their feedback was then incorporated in the next cycle of system updates—creating an iterative design process for software. This process did result in tradeoffs as incorporating some features into the software might conflict with the functionality of other features in the software.

    Cobb, P., Confrey, J., diSessa, A., Lehrer, R., & Schauble, L. (2003). Design Experiments in Educational Research. Educational Researcher, 32 (1), 9-13.

  6. Iterative development is the playing field on which design research is played out. I wonder if a company like Apple engages in what could be called design research. I think of design research as going beyond evaluative research aimed at improving later editions of a product–educational or otherwise–to delve deeper into the nature of human behavior that determines what makes a more effective product more effective.

    Apple is clearly good at evaluative research. As Bill Buxton points out in his book, Sketching User Experiences, Apple’s iPod did not really take off until its third iteration, by which time aspects of its interface had been substantially improved.

    Buxton contends that good designers design for the future and that it is indeed possible to do so by anticipating the evolution of technology. To paraphrase Buxton, the future is already here, it is just very unevenly distributed.

    Can a forward looking company like Apple improve its products through iterative cycles of design and development without yielding new insights into the nature of human learning and behavior? My guess is no. At some level, Apple must be engaging in design research that pays close attention to the contexts in which its products are used and the processes through which its products are designed.

  7. In reading about design research, my first impression was that it was strictly about creating some form of instruction and then improving it with iterations of evaluation and design. After reading more articles, I now realize that design research is much more than that.

    The instruction should be designed with some theoretical underpinnings. The research conducted not only suggests how the instructional intervention might be improved, but also its impact on theory. The intent is not just to improve the instructional product, but by studying the theory behind the instruction be able to make the instruction more widely available to other audiences.

  8. Other interesting aspects that I found as I continue to read the literature on design research are the ideas of transportation and generalizability. Critics of design research methodologies feel that design studies reliance on narrative studies limits the ability to generalize to other settings, which in turn would limit the ability of transportation to other milieus. Without experimental methods included in the analysis, how can the results truly translate to other mediums? But then I also wonder how you can move forward in a study without an in-depth analysis of the context in which it is presented. I feel that context matters, no matter what type of study you are conducting. Are experimental methods able to measure the impact of the local district politics in the development and implementation of an intervention? And that is why iteration is so important, whether you consider iteration in the development of the intervention, or in the actual design experiment itself. It gives you the ability to examine the context further to determine the impact the context has on the development and implementation of the intervention. This is valuable information to gather as one moves forward to introduce a new intervention. These are lessons learned for other product development or product redesign in the future.

  9. In reading more about design research, I realize that the cycles of study and design include the evaluation and evolution of not just the product but the theoretical underpinnings of the product.

    Several of the readings have pointed out the evolution of the product is based on the context in which the product will be used. I began to think about the fact that a product that in conceptualized and developed to work in one context, may need to be modified to take advantage of the affordances of another context. When making those modifications to the original product, in essence a whole other product can be created. This spin-off of the original product then begins to have its own design research iterations. Each of the products, both the original and the spin off, will then be evaluated and modified based on their own contexts.

    It seems to me that the next level of design research then becomes the comparison between these two products. Where do they have similarities? If these similarities are the intersection of their characteristics of each product, what theories can be learned from one product that can be applied to the other product? If there is an intersection of characteristics of each product, how well will improvements to one product carry over to the other product? What if the similarities are characteristics of each product’s context? By being able to appreciate the characteristics of these related products in their own context, design research can be expanded.

  10. The most obvious intersect between design research and iterative development is that they are both done in phases. It is best to gather all of the information in certain stages in order to properly progess to the next phase with some level of confidence. The passage above seems to hint that the product should be tested on small user groups instead of large groups to ensure that the product is not a total failure and that the parts that may need to be re-worked can be re-worked without progressing too far and not being able to turn back.

  11. I think one thing that has been impressed upon me during our readings on design research is not only noting the theories that result through design and research iterations, but also paying attention to the context in which the design research is conducted. In order for the product of design research to be fully transportable, there needs to be consideration of the context in which is was successfully deployed. Without considering the context in which a product was successfully deployed, it may be doomed to failure when transported to a very different context.

    Some of the design research studies indicate that when those studies have been conducted in an artificial, lab-type setting, the results have been varied compared to when the design research study is conducted in a more authentic setting. This is not to say that there is not anything to be gained by conducting a design research study in a lab setting, but the affordances of that setting need to be compared to the affordances of a more authentic setting to ensure successful transportability.

  12. Yes, But Does It Make Sense…?

    Both iterative development and design research are able to yield innovative solutions and allow for continuous improvement through human-centered inquiry. Furthermore, in both systems, designers employ some combination of both quantitative and qualitative research methods. However, the design research process extends beyond routine procedural iteration to include scientifically rigorous standards and professional scholarship. According to Middleton, Gorard, Taylor & Bannan-Ritland (n.d.), the extra steps involved in design research (the research experiment) help to validate not just the theoretical framework of the design, but its very continuation as a product. This seems significant to me as I consider designs I see every day. I think, in fact, that if a number of designs from the past five years, including many high-tech cell phones, had gone through feasibility studies, the product market might have seen vastly different outcomes. Especially in public education, a field in which budgetary constraints have a greater impact than in the private sector, the rigorous, programmatic process of design research is absolutely essential to the development and continual improvement of high-quality instructional designs.

    Middleton, Gorard, Taylor, & Bannan-Ritland (n.d.). The “compleat” design experiment: From soup to nuts. In publication.

    Rawsthorne, A. (2007). Why the overwhelming numbers of design flops. International Herald Tribune. Retrieved February 24, 2008 from the International Herald Tribune Web site:

  13. being new to Design research i believe it is important to understand whatever context or research is at stake. Saying this, it seems to me that iterative design focuses more of address the actual needs of the research audience as opposed to the “waterfall” method where the issues of the design are not followed up on. Like i understand it, Design Research is almost like an unending evolving cycle through which discoveries are made along with progress as well as re-designing goes on.

  14. I enjoyed reading about the experiment Bill Buxton describes in his book Sketching User Experiences: Getting the Design Right and the Right Design. The experiment involves three different sketches representing three distinct designs for a “programmable climate control interface”. There are four experimental groups of reviewers. Their job is to review design sketches and rate the merit of the designs. Group 1 is asked to review and rate all three design alternatives. Each of the remaining three groups reviews only one of the three design alternatives and rates it according to the same scale used by group 1. Group 1 gave design “X” the lowest rating. The group that reviewed only design “X” gave it a significantly higher rating than did group 1.

    One might draw the conclusion that a design option is best evaluated by comparing it to competing design options. This is useful as a practical matter.

    Although this research is clearly focused on an issue relating to design, does that make it design research? My answer is no. The results have implications for the future practices of designers, but does the research really provide new insight into the nature of learning? Not explicitly, as far as I can tell. I think design research happens when the designer has the intention of creating new knowledge in addition to developing a given design. If knowledge is created incidentally in the process of design, that is not research, though it is learning.

  15. The Value of “Expert” Opinions

    Americans have become obsessed with expert opinions. Whether due to educational inflation or the effects of too many true-crime television shows, we tend to believe that decisions are best made by those with esoteric – and often theoretical – knowledge. Design teams are routinely tasked with providing expert solutions for learning/instructional problems. Unfortunately, this reliance on “expert” thinking has lead to “solutions” usable by no one but the experts.

    Fortunately, design research can help to avoid design disasters by testing designs and creating iterations which more closely match the needs of actual users. Ejersbo et al., (n.d.), note that “empirical knowledge about learning is always highly contextualized” (p. 3). In order to understand the true uses and ultimate ramifications of any instructional design, we must not only access, but actually incorporate, the opinions and context of authentic users. We must conduct experiments or inquiries in their locations, with their level of expertise, and then revise the design to better fit their needs (rather than our egos).

    I have given this considerable thought as I have worked to devise a contextual inquiry “script” that my design group may use when we interview and observe our child users. All of our ideas to date are based upon sound ISD theory. We have attended to learning goals and objectives and instructional hierarchies. We have accurately performed all the necessary steps to prototype a theoretically appropriate device for students with ADHD and giftedness. Soon, however, despite our “expert” status, some very practical 10-year-olds may provide us with feedback about how short we have fallen from design perfection. Design research, it seems, may be a humbling experience, but it leads to more effective solutions.

    Ejersbo, L., Engelhardt, R., Frølunde, L., Hanghøj, T., Magnussen, R., & Misfeldt, M. (n.d.). Balancing product design and theoretical insights. Design Research Handbook (in press).

  16. The thing that resonates with me the most is the iterative nature of design research. To me, iterative is not only good to validate assumptions, but really this is how the human brain works best. The human brain is best at adapting, which means it is best at taking something existing and figuring out how to make it better, based on an existing *need*. Once that need is worked out, someone finds another flaw, and finds a solution for that need, and so on. This is in essence, iterative development.

    Only think of all the technological concepts of the past 100 years. The car was once called the horseless carriage because the horse-driven carriage was what people were familiar with and could conceptually relate it to in order to wrap their brains around it. They had no other frame of reference. The laptop exists today not *in spite* of the computer’s beginnings as a machine that took up a whole room, but *because of it*.

    Humans are not good at creating something brand new from thin air. We could never have gone straight to laptops from pen and paper, we needed the intermediary steps of successively smaller computers to refine our notions of what we want and need in a computer, and in fact to refine our notion of what a compute is and should be.

  17. The Affordances of Design Research

    It surprises me how easily designers can increase the affordances of design solutions merely by committing to a process of user research and the creation of successive iterations. It is, of course, true that the collection and application of user data allows designers to tailor modifications to the intersections of demographics, attitudes, and natural behavior in users’ lives, thereby affording them richer, more satisfying experiences. More than this, however, designers can create increased opportunities for user actions through successive designs. Through an ecological approach, taking into consideration the cultural conventions and constraints of users as well as their general technological skills, we can actually extend human capabilities through the refinement of our thought processes.
    Design research, therefore, seems to be the business of creating possibilities for current and future use. What we will do tomorrow depends upon the shared vision of today.

    Kuniavsky, M. (2003). Observing the user experience: A practitioner’s guide to user research. New York: Morgan Kauffman Publishers.

  18. Reading about design research while taking EDIT 590, Educational Research, allows one to see the juxtaposition between the views of design researchers and experimental researchers related to validity. Design researchers specifically consider the context as a part of their experimental design. They take a very organic perspective at educational research and look at the natural environment where their research is being implemented, and in a way, let it grow and evolve naturally, without the constraints that are found in traditional educational research. Critics say that this lack of control in relation to internal and external validity limits the impact that design research has in the educational community. But at least design researchers are considering what the mysterious variable “x” is in relation to the experiment instead of struggling to find the right balance to control for it. I understand that looking at the issues of validity in experimental design is considering what the mysterious variable “x” could be … but in design research, it is almost as if they are embracing variable “x” with open arms! I feel there is some real value in this perspective when moving forward with the design and development of interventions. But as Tiffany Taylor pointed out to me, the iteration that occurs in a design research is one of the tools that design researchers use instead of controlling for internal and external validity found in experimental research. Iteration, coupled with a basis in theory and “auditable trials of documentation” (p. 366) as outlined by Lesh, Kelly & Yoon (in press) all contribute to the evolution of sound design.

    Lesh, R. A., Kelly, A. E., & Yoon, C. (in press) Multitier Experiments in Mathematics, Science, and Technology Education. In A. Kelly (Ed.), Handbook of design research in mathematics, science and technology education (tentative title). Mahwah, NJ: Lawrence Erlbaum Associates.

  19. Research is necessary because it informs design and development. You first find out, through research and analysis, if something even needs to be developed. Then as you develop something, you find out through research what is important to users and what isn’t that important. Finally you find out how well the developed item is doing to fit the needs of the users through research.

    I think it is a very important step in the design process, unfortunately, it does cost a lot, and if organizations do not have a lot of time nor money, this vital step is often over looked and not done. When that happens, the end product might not be very usable for the user or audience it was intended for.

    In iterative development, a product or piece of instruction is subject to further research and refinement, because it might not be usable to the user now or potentially in the future. I think Matthew gave a very good example of iterative development when he mentioned the iPod and how it “took off” after the third iteration. It took up until the third iteration of the iPod for Apple to develop an interface that “worked” with users. It is a simple- easy to use interface that does not require a lot of button pushing. Apple continues to research and develop new designs for the iPod that continue to incorporate modern technology. In fact, just recently they came out with the iPod touch which uses a touch screen for the interface.

    In iterative development, research informs change and helps to move the product or instruction forward and to a broader audience. When you initially design something, you might not have taken into account what the bigger audience wants or needs. For example, when working in the Croquet group, we initially developed our needs analysis data on the input of three geoscience professors. What struck us all is that all three of these professors felt that they needed a virtual world centered on a hurricane- so we developed a prototype around a hurricane. Now we have had the chance to go back and survey a broader audience of geoscience professors and students. We discovered that both geoscience professors and students have a similar need- which is not centered on a hurricane in a virtual world. Through our iterative process we discovered that we were not meeting the needs of geoscience professors in our original design, so we are going back to “the drawing board” so to speak in order to meet the needs of our broader audience in order to make our design more usable to the geoscience professors.

  20. The article “Balancing Product Design and Theoretical Insights” (Ejersbo et al., in press) was especially interesting to me as it is often tough to create a balance between innovative design artifacts and theory-building. The “osmotic model” shows the give-and-take that must go on between these two competing objectives. Iterations of their model (ethnographic, product design, or methodological approaches) and the paths taken through the process, illustrate that one can begin the design-research process at many different points in the cycle depending on the project type. They conclude that is due to constraints from many different factors (organizational, financial, pedagogical, technical, political, etc.) that it is virtually impossible to implement an “ideal” research design project and there is always going to be more emphasis put on either product design or theory (p. 381).

    In personally conducting design research, I often have to stop and reevaluate what the objectives of the project are and what the best route would be to achieving them. Sometimes it is a chicken or the egg scenario. Do we research the users first? Do we design an artifact then have users react to it? Do we start with a theory based on what we think the solution should be? Do we guess at all of it then just jump in wherever our gut tells us too? Through readings and in practice, I’ve come to realize that there is no particular process to follow in design research. You have the standard cyclical elements of researching the user, designing the product, testing it with the user, redesigning based on that research, then starting all over again. However, depending on the constraining factors cited above for your particular project, where you jump into the process and where you end the process could be totally different each time you face a different design problem.

  21. In their article, Multitier Experiments in Mathematics, Science, and Technology Education, Lesh, Kelly & Yoon discuss four design principles that counteract design flaws. Two in particular resonate with me as a student in the Immersion program – the knowledge accumulation principle, and the diversity and triangulation principle. With the knowledge accumulation principle, all those involved in the design process know from the outset that they will be engaging in iterative design, and that documentation of these iterations is necessary to allow an understanding of participants learning. I feel that Immersion is design research in action. The use of a wiki to document our tentative steps, various explorations of the design process, and decisions and revisions as we move through our own iterative team development is an example of the knowledge accumulation principle. Building consensus as a group, while sometimes painful, allowed all perspectives space to explore, absorb, contemplate, and adapt to the overall needs of the group and the project. In this case, the diversity and triangulation principle is evident in the Immersion team’s interaction with each other and in relation to the project. So not only is the design process iterative, the group process needs to be iterative in order to contribute to sound design.

    Lesh, R. A., Kelly, A. E., & Yoon, C. (in press) Multitier Experiments in Mathematics, Science, and Technology Education. In A. Kelly (Ed.), Handbook of design research in mathematics, science and technology education (tentative title). Mahwah, NJ: Lawrence Erlbaum Associates.

  22. Working in corporate and government training, I have definitely seen a lot of trade offs, some valuable and some not. It’s true that deciding which ideas are worth the trade is difficult. Especially since, rather than using what Kuniavsky calls the “waterfall method” of leaving the idea to fend for itself, following iterative development gives the idea a much better chance of succeeding but also requires you to expend more time and effort.

    Design research, like the steps taken in iterative development, strives to understand the environment, past ideas and attempts, and contemporary climate that an idea, method, strategy, etc will have to endure to survive and prosper. I liked Kim’s comparison of iterative development to the System Development Life Cycle used in software development; both development cycles share many of the same steps to make sure an idea can be sustained in an environment, including the audience it will face. They also share the same end goals – creating a product/strategy/training/etc that will be sustainable and developed to be the best for user experiences.

    I also agree with Cheri’s post in which she says that many times people will cling to what has “always” been done simply because it has been done. Cheri referred to that for education, but I believe it is also applicable to corporate training. Conducting the types of research and development required to conduct needs assessments and user research can be expensive, and often foregone simply to save money. Instead, an administrator or instructor will do either what has always been done, make changes based on budget dictations, or will do something new based on what the administrator or instructor believes what should be done rather than what appropriate research dictates. Also, the demands of immediate results lead administrators to forgo the necessary research, design, and development based on time constraints. Time and budget seem to be the biggest hindrance for both design research and iterative development, but if the administrators involved realized the difference in the end results, before it is too late, they would produce much better training and products.

  23. Just thought I would say that I am finding all of your posts very thought provoking and thorough. Your perspectives on this new emergent method are extremely useful as I think about the next article to write about this topic!

    Just wanted you to know that your collective intelligence (analyses and syntheses) are being put to good use in provoking thought!

  24. From Susan Kenney:

    The Stanford Center for Design Research is a community of scholars focused on understanding and augmenting engineering design innovation and design education. We are dedicated to facilitating individual creativity, understanding the team design process, and developing advanced tools and methods that promote superior design and manufacturing of products.

    As I think about all I have read in this BLOG and in assigned materials, although I have an understanding of “iterative development”, I am still confused about the meaning of the term “design research”. I can see how “iterative development” is the process of:
    • Investigating needs, environment, users, materials/devices
    • Problem solving for a viable solution,
    • Testing the solution with a pilot group,
    • Revisiting the solution and adjusting to the new information,
    • Retesting the adjusted solution,
    • Using the gathered information to create a better fit of the solution to the environment and users,
    • And retesting….. etc.
    It seems to be an effective way to develop practical solutions.

    As I understand, design research as stated in Stanford’s Center for Design Research, as harnessing the creativity of individuals into a team process by using specific tools.

    The handbook manuscript made available on line for the course materials, Handbook of Design Research Methods in Education: Innovations in Science, Technology, Engineering, and Mathematics, Edited by Anthony E. Kelly, Richard A. Lesh, and John Y. Baek mention the process of Instructional design including the tools of Analysis, Design, Development, Implementation, and Evaluation, traditionally known as ADDIE as being included in design research to a certain degree. It also includes Bannon-Ritland’s, 2003, idea of multiple cycles within and/or across studies, and the consequence of failing to include iterations that highlight the need for design adjustment and test the successive changes for effectiveness. It speaks of iteration as a core concept in design research.

    Therefore, rather than an intersection, which is often associated as a point or small section where two ideas cross, I propose that design research and iterative development merge and become integral.
    Even as the initial comment in this BLOG described the organism evolving, I liken design to the process of natural selection whereas organisms with the most successful combinations will survive. Iterations will allow the process more control to adjust a design before it is obliterated and the potential good portions are lost.

  25. In the article, “The compleat design experiment: from soup to nuts” the authors describe design as a noun and a verb.
    “Design (the verb) is the creative process by which designers considering a problem in their field of expertise generate a hypothetical solution to that problem. Design (the noun) constitutes the general hypothetical solution often embodied in material form (e.g. a blue print or a physical model).” The compleat design experiment: from soup to nuts. James Middleton Arizona State University Stephen Gorard, University of York Chris Taylor, Cardiff University Brenda Bannan-Ritland
    From the above definitions we can define design as a creative process that results in a physical model of the solution to the problem.
    The iterative research process made me think of the ethnography process. As ethnography looks into what the users need and the iterative process is where the physical model is updated as per the feedback from what the users want and need. Thus design and research complement each other and it is important to know for who and why a particular model or design is being made.
    I found this not only interesting but important to the design process, that the, “design activity consists of an interaction between
    designer and the contextual constraints and is accomplished by proposing the form of an artifact, system, or process, which in turn drives its behavior, which in turn can be compared with its desired function. Because artifact, system, or process is cumbersome when repeated throughout a text, we will use henceforth the general term system to denote the ostensible outcome of a design except where a specific example is given. Both the form (proposed and at some point realized) and the desired function are hypothetical entities that are linked by a theoretical model. In mechanical engineering, this model is called the function structure. The behavior of the system constitutes observable data. Therefore, the degree to which an artifact’s behavior embodies its function can be considered a partial index of both: (a) The quality of the design, and (b) the degree to which the form of the system produced or served as a conduit for the behavior .”(Shooter, Keirouz, Szykman, & Fenves, 2000).
    The engineers use of the iterative, design-and-test cycle is critical for the transformation of both the form of the system. They are dealing with the machines and their marketing and how they can be made useful to the people. But how is this iterative design different in the field of education. I found that each person is different but they can be grouped on some basis and social, cultural, age, gender, etc and thus the quality of the design and the desired behavior as a result of that design would require a lot more than data to be observed and change to be made in the machine. In this case humans are observed and design models are built on basis of this interaction with the design and is improved as the person is observed in its natural environment and his interaction with the design prototype.
    The process indicates that a design contributes not only the development of a theory but also the design practice itself.
    Design Patterns trajectory, “which seeks to identify common factors in promising sequences of activities and define them as patterns that can guide designers of learning environment.” (kali)
    Thus by observing these patterns one can group the humans and can help designers the need of that particular group and design accordingly.

    Yael Kali, The design Principles Database as a Means for Promoting Design-Based Research,
    Bannan-Ritland, Brenda et al., The “Compleat” Design Experiment: From Soup to Nuts

  26. Designing for Meaningful Education

    Lesh, Kelly, & Yoon (n.d.) proposed that “…design research in realistically complex settings should be expected to be multidisciplinary, multiaudience and multistakeholder” (pp. 346-347). I wish this point of view were upheld, not just in professional design literature, but in the larger environment of public K-12 education. However, legislation based upon an extremely limited perspective has forced public education to become a set of “cookie-cutter solutions” to be carried out without regard to either research or significant modification.

    One-size-fits-all legislation belies the complex, interactive nature of both the learning process and the learners themselves. It allows little room for professionals from multiple disciplines to develop and coordinate in meaningful, complementary instruction in order to meet the needs of diverse learners. Fortunately, many experienced teachers, albeit unwittingly, embrace the tenets of design research: shareability, reusability, and modifiability. We seek out, and interact with, each other in order to informally test, adapt, and pass on instructional designs that work. Unfortunately, the process of fighting an inflexible system contributes to the average burnout rate of 5 years for teachers.

    In some ways, the privatization of public education might improve the quality of instructional programming, as it would allow – and might even require – research into the effectiveness of instruction within a variety of settings and with a variety of staff and students. The employment of creative/divergent thinking, the promotion of evolving professional expertise and the reduction or elimination of predetermined outcomes would go a long way toward making public education relevant for a rapidly-changing world.

    Lesh, R. A., Kelly, A. E., Yoon, C. (n.d.). Multitier design experiments in mathematics, science and technology education. Design Research Handbook. In press.

  27. The way I understand the design research and the iterative development process is that research is a necessary part of the process for creating a design because it allows you to study and explore the environment you’re working with. From there you can begin to figure out what fits the needs of the intended users of your design, what ways they learn best, where the obstacles are and what solutions might be realistic, what they consider important. Through multiple iterative cycles of testing your solutions with your target audience and adjusting to their behaviors then testing again, ideally you will have a more effective design in the end based on research conducted throughout the process. Iterative design is refinement of your original ideas based on the research of differing perspectives, behaviors and suggestions you may have never considered from your user base. To me, design research seems to be the logic first step but then the research and the iterative process go hand-in-hand as you continue to further develop and adjust the design.

  28. Today at work I ran right up against one of these tradeoffs between design and research. We’re in the process of putting together a series of seven workshops. With the first of the series looking very good, we took it to some of our instructors for a pilot and preview.

    The loved the content, but they hated our design. Everything from the order in which the material was presented to the convention we used for number the PowerPoint visuals was met with criticism. However, as a group of instructional designers, we happen to think our design is not that bad.

    At this point, we have a few options. We can keep our design and ignore the instructors’ comments. We can use their comments to change the design of the course to what they say they want.

    Or we can take a step back and realize that neither our ideas nor those of this group of instructors should be the final say. We made a mistake by not taking and iterative approach to our design and conducting research throughout the process to ensure we were taking the opinions of our users into consideration.

    While we did vet the content of this series with our instructor community during its development, we did not do so with the design. In an ideal world, we would have vetted the design right along with it. Instead we have to do that research and revamp our design in a much shorter span of time. Additionally, because of the short span of time here, we are unlikely to get the breadth of input we could have gotten earlier, thus opening us up for further criticism when the product is released.

    I think on the surface it can seem that intersecting design and research creates more work because one has to two jobs concurrently when before they were frequently done sequentially. However, the scenario I faced at work today showed that that is simply not true. In today’s digital world, the customer has come to expect instant gratification. (Case in point: While writing this posting, I was able to download a CD from a new musical group I’ve been wanting to check out for a few weeks.) What is the customers’ instant gratification but absolute responsiveness on the part of the provider? If we are investing a lot of time and money into a product that will not be simple to change after it is finalized (like a textbook or anything else that requires a lot of investment in production), then we owe it to the customer to ask his opinion long before we go to market. And we should ask for that opinion as many times as we can before we have to have the design locked down, because while we are hopefully coming closer to the user’s ideal with each iteration, it seems to be very rare that the designer has a full understanding of the user’s wants and needs.

  29. Being new to the discipline of design development I’ve had some difficulty relating to the readings we’ve been assigned this semester. I can understand the concepts to a certain degree but I can honestly say I’ve never had opportunities (other than EDIT 730 & 752) to apply them at an instructional level, and at best that’s been in a condensed version. As I have read the chapters about the iterative development process in our Kuniavsky text, over and over again in my mind I come up with the constant question of when is a design ever good enough? Is it good enough when the funding runs out? Is it good enough when the deadline is looming? As designers won’t we always feel that there is possibility for improvement even though we have done the prior research and analysis of the data for our intended audience? In the corporate world, do you settle for something that would be considered “good enough” or do you constantly strive for that perfect solution? It just seems to me that it’s easy to fall into the recreating mode just to recreate!

    I found Matthew’s post on how the Apple iPod has developed over time very intriguing. This is a good example of how a design has changed to fit its ever-growing audience that continues to use the technology for diverse reasons. I was an early adopter and bought the second generation iPod. At that time it was really used only for music downloads and the price was a concern for many people (other than diehard Apple fans). But as Apple developed iTunesU and educational institutions began seeing a benefit of the iPod for educational purposes, exposing it to an even wider audience of users, they began designing smaller, lower cost versions (Shuffle) as well as video capabilities, the appeal of the multiple uses of the device took over and the iPod became a cultural craze. Now we see the iPod functionalities of ease of use, touchscreen capabilities, awesome resolution showing up on not only the iPod but the iPhone and iTouch. They have set the mark for other companies to follow. Great example and great post by Matthew.

  30. I posted this response early back in Feb. commenting on the first chapter of the research reading. However, for some it went to a different place. Here are my thoughts again.

    That part of the reading about how to assess collective activity in larger group was interesting for me. It is a challenge that we determine if there is general understanding in a large group or not. I think there is no problem to determine a user understanding individualy but in a large group I would be hard. In this reading of – Enabling Innovations in Education and Systematicizing Their Impact, it talks about Toulmin’s basic model which represents four parts: the data, the claim, the warrant, and the backing. The reading did not give much information about the model. However, I did some google search to know about the model in more details. I found more details about the model which makes more sense supported with examples.


    By: Emad on February 2, 2008
    at 7:18 pm

  31. On February 9th Matt Marini wrote:

    “Iterative development is the playing field on which design research is played out… I think of design research as going beyond evaluative research aimed at improving later editions of a product–educational or otherwise–to delve deeper into the nature of human behavior that determines what makes a more effective product more effective.”

    This is a very insightful statement because Matt has touch upon one of the most significant social struggles of the 21st century—how to create a balance between standardizing (design) and customizing (user-centered research and iterative development) learning environments.

    Traditionally, the ISD process has been very designer centric. Using objectivist-based theories and models, instructional design became a matter of manipulating the environment so that all learners would be encouraged to employ the same approach to achieve expected outcomes. The sole purpose of the evaluation process was to affirm the assumptions of the designer. If these assumptions were not affirmed then it was up to the designer to interpret the learners’ experiences and revise his or her initial assumptions. In other words, the learner was considered an inanimate object whose perceptions and experiences have little influence over the direction of the learning environment.

    The designer-centric approach to instructional design was able to thrive when the means of disseminating information was controlled by the few. However, as Ajit Jaokar and Tony Fish point out in their book Mobile Web 2.0, we are in the midst of a revolution in which individuals are playing an ever increasing role in the creation process.

    In addition, technological advancements have made it possible to create functional prototypes that can be assess through out the development process rather than at the end. As a result you can cost effectively create and test prototypes throughout the design and development process and not just at the end.

  32. Matt’s anecdote about the iPod reminded me about what functionality means, and that it has many different aspects. Though there’s no doubt that Apple has done extensive user testing to find out what is usable, I think one of the core tenets of Apple’s success that one misses out on isn’t just that they are functional objects, but that they are also beautiful objects, objects that are perceived as being “cool”, and that beauty and desirability is also party of usability. Aesthetics plays a huge part in the user’s experience and in what can be considered good design, as Don Norman points out in the Design of Everyday Things (among other books). I have an iPod now, and I love it. I’d be lost without it. But before the iPod, I had another mp3 player- a Rio. I hardly ever used it. It did the same thing at its core as an iPod, it played music. But I just couldn’t muster enthusiasm for it, unlike the iPod that I drooled over every time I was in Best Buy. But I refused to buy the iPod because as pretty as it was, it did the same thing that my current mp3 player did, so it seemed silly to waste the money. And then my husband bought me an iPod for Christmas. And I found that aesthetics and delight make a huge difference in the user experience. I use my iPod all the time, and treasure it in a way I never did the other mp3 player, which makes me willing to overlook some of the other faults of the iPod, like proprietary media formats. Beauty is tied integrally with the user experience.

  33. Last night’s conversation about ethnographic research methods served to emphasize the iterative nature of design and research that we have talked about for the past several months. As researcher/designers, we have so many different methods of research at our disposal that it seems almost impossible that we might not get the information we need to create a perfect design. However, certain methods are more or less useful at certain points in the process.

    Based on both my group’s experiences this semester and my professional, I posit that traditional quantitative methods are most useful early in the design. Multiple choice surveys are useful for gathering information from a large population and the limited answer choices make the data easier to analyze. This is useful to ensure that the basic premise of the design will be successful by appealing to a wide audience.

    Some ethnographic research methods are also appropriate during the early stages of design. For example, the “day in the life” method allows the researcher to gather information about the target audience that they may not even realize about themselves.

    Other ethnographic research methods are more useful when a prototype has been developed. In my professional life, we always try to do pilot tests and preview studies with our courses before we release them to the public. These sessions include methods such as storytelling when we ask the users to compare the new product to products they currently use. In the telling of their story, they may reveal gaps in our product that other products fill and we can implement appropriate changes.

    The fact that different research methods are appropriate at different stages of the design process emphasizes the iterative nature of both design and research in developing a quality product or solution.

  34. The question is where does it begin? and not where does it ends? because then we as Instructional designers will be out of jobs!!
    As a designer one can only ask where do I begin what do want/need. In the both the models of ID that is the ADDIOE and ILDF models begin with analyzing the needs and requirements of the audience for whom they are designing. Interestingly I came upon a book called “Tooth pick” now I got you thinking what a useless thing to write about but the author of the book Henry Petroski writes over a hundred page book on the little thing called tooth pick.
    He writes about who invented toothpick and how it was made. The man behind the tooth pick took twenty years of his life trying to build a machine which could make tooth picks. He worked with woodcrafters to mechanical engineers and finally came up with the tooth pick which is now hardly of any recognition.
    Though it has become insignificant of great worth we must remember that there was twenty years of thought and hard work behind it. There was thought for design and research on how to carve this little piece of wood. There was trial after trial after which finally something was accomplished.
    While going through this course I feel the same way that we are testing our product for its effectiveness and usability. There is a lot of research behind the design of the prototype we have developed.

  35. Matt and Nancy’s posts were very thought provoking, about as Nancy put it, “how to create a balance between standardizing (design) and customizing (user-centered research and iterative development) learning environments.”

    During our research this semester for the ADHD-Gifted project, our group has had to consider both standardizing the device we have designed as well as allowing for customization. The biggest trend I have noticed when looking at educational software and devices is that the standardization becomes critical in the features available while the customization is important to the visual and creative aspects. Both pieces come together to create a very usable as well as effective solution. In a good design, the customizable options are considered along with the standard options in the design. In a good iterative development cycle, these options can be built in from the beginning, saving time and eliminating rework.

    In our project, we have had to consider customizable options very early in our design to ensure the device can be usable and desirable to our target audience of children. Luckily, our audience has been very forthcoming with their desires for where to customize 🙂

  36. In general, both theories are alike, their main caracteristic is the development of ideas, hypothesis or projects that have succesful results. Both methods follow paths to ensure the design complies with the objective.
    Iterative development allows the change of audience and other areas since the evaluation is constant and can cover more areas of study or research. Iterative development follows the engineering method as well as design research. In our class project, we were able to apply only Design Research while working in our prototype. The study covered immediate experience of the product with a small group of participants. With time and able to perform Iterative develoment we would have been able to explore the product in depth and its market in a more extensive way.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: