Instructors for quantitative research courses often find that there are many different types of support material for those courses. That is not the case with qualitative courses. Very little support material is available for qualitative research courses. In this paper we describe the creation of one multimedia package that focuses on one type of qualitative research - critical ethnographic techniques. The package was created to help graduate students learn to use five critical ethnographic techniques: meaning fields, validity reconstruction, role analysis, power analysis, and horizon analysis. We used a constructivist instructional design model, R2D2, to guide our development work. It is based on an interpretivist epistemology and a constructivist theory of learning. The result was a multimedia instructional package designed and developed for use in courses teaching qualitative research. The hypermedia, multimedia program called The Critical Researchers Guide to Conducting Qualitative Research (CRIT) has three major components: (1) video cases of middle school settings or sites; (2) definitions and descriptions of qualitative strategies; and (3) application of qualitative techniques. The paper describes revisions and reformulations of the instructional package across the instructional design process.
Webb and Glesne (1992) "attest that teaching qualitative research methods in colleges of education is challenging and exhilarating precisely because such courses call into question students' taken-for-granted assumptions about so many things: the purpose of research, the uses of method, the nature of knowledge, and what it means to be human" (p. 772). They add that qualitative research has emerged in education and brought about the birth of new courses, programs, and special interest groups in the area. In the past, few textbooks dealt specifically with qualitative research. Now there are many texts on qualitative research, such as Denzin and Lincoln's (1994) Handbook of Qualitative Research and Carspecken's (1996) Critical Ethnography in Educational Research: A Theoretical and Practical Guide. Teaching qualitative research in graduate programs can be difficult for a number of reasons. As a relative newcomer to the group of research methods courses taken by graduate students in education and psychology, the literature is scant on approaches to teaching qualitative methods. It is new in two ways: (1) the actual methods of data collection and analysis and (2) the underlying assumptions and epistemologies are often both complex and contradictory to the content students learned in other courses. According to Web and Glesne (1992) a wide variety of approaches are used and many instructors experience difficulty because the subject matter is complex and unfamiliar to students.
Even a cursory reading of the qualitative research methodology literature highlights the diversity in the field (Denzin & Lincoln, 1994; Eisner & Peshkin, 1990; LeCompte & Preissle, 1993; Willis, 1995). We will, however, use a general and relatively flexible framework for deciding whether an approach is "qualitative" or not. According to Strauss and Corbin (1990), qualitative research is any research that produces findings not arrived at by means of statistical procedures or other means of quantification. It can refer to research about people's lives, stories, and behavior, but it also can be about organizational functioning, social movements, or interactional relationships. Qualitative research techniques often rely on observation to collect unique data about the problem under study.
Techniques or methods of research are not, however, the only differences between qualitative and quantitative approaches. Qualitative research differs from quantitative research with respect to philosophical foundations, underlying assumptions, and research methods. Although descriptions of qualitative research methods given by different authors vary considerably, most characterizations of qualitative research emphasize participant observation (or more involved actions such as emancipatory and action research methods) and in-depth interviews. Quantitative methods contrast with qualitative research techniques, in which reliance is placed on the research instrument through which measurements are made. Qualitative research usually consists of three components: (1) data, which can come from various sources, (2) analytic or interpretive procedures that are used to arrive at findings or theories, and (3) written and verbal reports. Some researchers gather data by means of interview and observation, documents, books, and videotapes. The data produced are considered to be rich in detail and closer to the informant's perceived world, while quantitative approaches may lead to an impoverishment of data (Carspecken, 1996).
Understanding the major tenets of research paradigms, specifically qualitative, is essential to conducting good research. According to Borg and Gall (1989), qualitative research is more difficult to do well than quantitative research because the data collected are usually subjective and the main measurement tool for collecting data is the investigator. Therefore, before one can conduct qualitative research effectively, extensive training and practice in the methods are necessary.
One way to think about teaching qualitative research is from the perspective of cognitive flexibility theory (Spiro & Jehng, 1990). Qualitative research methods is a subject that is "ill-structured" and "complex". It is ill structured because you cannot teach precise recipes for conducting qualitative research. Many decisions must be made on the fly as the research proceeds. It is complex because the decisions to be made involve considering many aspects, and often from multiple perspectives. Cognitive flexibility theory (Spiro & Jehng, 1990) was created to guide our thinking about how to teach ill-structured, complex content. According to Spiro, Coulson, Feltovich, and Anderson (1988), teaching content such as qualitative research methods via material that was developed using cognitive flexibility theory and that uses hypertext and hypermedia can be an effective instructional approach. A critical element of this approach is the use of multiple representations that capture the real world complexities of the subject matter, in this case qualitative research.
The increasing interest in qualitative research has, predictably, led to the addition of one or more "qualitative research" courses to the requirements for advanced degrees in education. Instructors for quantitative research courses often find that there are many different types of support material for those courses. That is not the case with qualitative courses. Very little support material is available for qualitative research courses. In this paper we describe the creation of one multimedia package that focuses on one type of qualitative research - critical ethnographic techniques.
According to Denzin and Lincoln (1994), ethnography has had many uses and meanings throughout history. The history surrounding ethnography reveals how multiple uses and meanings are brought to each practice. Critical ethnography differs from traditional ethnography in its attempt to link the detailed analysis of ethnography to wider social structures and systems of power relationships in order to examine the origins of oppression. Critical ethnography raises substantive questions about structural relationships. The intention is to go beyond grasping the subject's meanings in order to relate those meanings to wider cultural and ideological forms. Critical ethnography is a widely used technique in critical social research. The involvement and close attention to detail characteristic of ethnography make it useful for rendering visible the invisible, and for revealing anomalies and common-sense notions.
Carspecken (1996) follows Habermas in distinguishing two basic methodological perspectives that can be employed simultaneously within a critical research project: (1) hermeneutic meaning reconstructions and (2) objectivizing studies of social systems. Hermeneutic meaning reconstruction is totally congruent with interpretivist methodologies; one takes the insider's view of a cultural group and reconstructs tacit cultural themes and structures that members commonly employ to interpret the world, judge the world, and construct their social identities. Objectivizing studies of the social system prioritize an "outsider's" view. Here one seeks social structures that help shape and constrain culture.
Carspecken (1996) developed a five-stage scheme for conducting critical qualitative research, where stages 1-3 employ hermeneutic reconstructive techniques and stages 4-5 emphasize the objectivizing stance in one's search for system phenomena. The preliminary steps include creating a list of research questions, a list of specific items for study, and examining researcher value orientations. The five stages are
These stages were designed to study social action taking place in one or more social sites and to explain this action through examining locales and social systems intertwined with the site of interest. Common subjective experiences and the significance of the activities discovered with respect to the social system at large are assessed (Carspecken, 1996).
This study's emphasis is on the critical ethnographic techniques Carspecken (1996) outlined in stage two of preliminary reconstructive analysis. There are five techniques:
These five techniques for conducting critical ethnographic research were the primary focus of the instructional package developed.
Purpose of the Study. The purpose of this study was to qualitatively create an instructional product using a hypertext system derived from cognitive flexibility theory about the complex and ill-structured domain of qualitative research and its methods, specifically those detailed in Critical Ethnography in Educational Research: A Theoretical and Practical Guide (Carspecken, 1996). A multimedia instructional package was designed and developed for use in courses teaching qualitative research. The hypermedia, multimedia program called The Critical Researchers Guide to Conducting Qualitative Research (CRIT) has three major components: (1) video cases of middle school settings or sites; (2) definitions and descriptions of qualitative strategies; and (3) application of qualitative techniques. This final component is comprised of exemplary illustrations of each concept and associated techniques that were used with other qualitative data sets.
Over the years, many instructional design models have been proposed (Bagdonis & Salisbury, 1994). Andrews and Goodson (1995), for example, identified more than 60 models, but descriptions of over 200 models have been published in the educational technology literature. However, most of the ID models are, however, based on behavioral and information processing theories of learning (Dick, 1996). In fact, Walter Dick and Lou Carey developed the most popular ID model in this tradition in 1968. It is now in its fourth version (Dick & Carey, 1966). Dick and Carey called it an Instructional Systems Design, or ISD, model because it is based on one form of systems theory, as well as behavioral and information processing theories of learning. "Behaviorism is prominent in the roots of the systems approach to the design of instruction" (Burton et al., 1996, p. 57).
While these models have dominated the field for over three decades, they have been criticized recently by a number of scholars. For example, Wilson (1993) felt that rapid change in instructional design comes from the debate initiated by constructivist theorists. In describing instructional design's state of change, Wilson summarized his objections to the established, ISD, approach this way:
The problem can be simply stated: ID, in its present form, is out of sync with the times.
Willis (1995) criticized ISD models because of their behavioral foundation and listed eight characteristics of these ID models that he considered undesirable:
Several other authors have suggested that ID could be based on theories of learning other than behaviorism or information processing. Scholars (Lebow, 1993; Wilson, 1997; Winn, 1992) have offered general suggestions for basing ID on a constructivist foundation. Several instructional designers also have created complete ID models based on constructivist theory. For example, Cennamo, Abell, and Chung (1996) developed the "Layers of Negotiation" model while they were creating a set of video cases for teacher education. They noted that, "Many authors have questioned whether traditional instructional design models are suitable for designing constructivist learning materials" (p. 39). Their model, Layers of Negotiation, is based on the same set of constructivist principles that are the foundations for the instructional material they developed.
The study reported here used another ID model based on constructivist theory-R2D2 (Willis, 1995, 2000).
The Recursive, Reflective Design and Development (R2D2) model was introduced in a journal article by Willis in 1995. The R2D2 model was the methodology used to guide the creation of The Critical Researchers Guide to Conducting Qualitative Research (CRIT).
R2D2 has four overarching principles: (1) recursion, (2) reflection, (3) non-linearity, and (4) participatory design. Recursion allows the designers to revisit any decision, product, or process at any time in the design and development of the product, and make refinements and revisions as needed. Recursion, or iteration, makes the design process a spiral-the same issues and tasks may be revisited many times across the design and development of a particular instructional product.
Reflection is probably best understood by contrasting it with the opposite principle of design-technical rationality. Design based on a technical-rational approach requires developers to follow a set of pre-defined rules that prescribe what is to be done. Reflective design places less faith in preset rules and instead emphasizes the need for the designer to thoughtfully seek and consider feedback and ideas from many sources. For detailed information on the process of reflection in professional practice see Schön (1983, 1987).
The third guiding principle, non-linearity, comes from chaos theory (You, 1994). Instead of providing a linear sequence of steps that must be completed in a certain order, R2D2 suggests a set of focal points that need not be approached in any particular predetermined order. Different projects may call for different starting points. For example, the design process need not begin with a detailed plan that requires development of precise objectives at the beginning of the work. Objectives may, instead, emerge over the design process and not be completely set and clear until the end of the project. Thus, the design process commences wherever it is appropriate and progresses as appropriate.
The last principle, participatory design, is based on the assumption that the context of use is critically important. Further, the people most familiar with those contexts will be the users. Therefore, they should be involved extensively in all phases of the design and development process (Schuler & Namoika, 1993). In R2D2 the idea of participatory design has been expanded beyond end users to include "experts" in the sense Eisner (1979) meant in his connoisseurship model of educational research. Thus, ID using the R2D2 design model involves a participatory team that guides the process. This team typically includes instructional designers, subject matter experts, teachers, and students. Members of the team are often referred to as stakeholders.
These four guiding principles - recursion, reflection, non-linearity, and participatory design - are quite different from the principles that serve as the foundation for many instructional design (ID) models. Most of the existing ID models are based on behavioral and/or information processing theories and proponents of those models have been quite critical of these principles. Merrill (1996), for example, was blunt in his assessment of approaches that try to involve stakeholders in developing a vision of how content should be taught:
A "visioning activity" is a recipe for disaster in the real world of instructional development. It is a dream of academics who value collaborative approaches to knowledge; but, in practice, it often leads to disaster. There is no doubt that "stakeholders" must have a role in determining "ends" (how the learners will be different as a result of instruction), but when "stakeholders" play a significant role in determining "means" (how those changes in the learners will be fostered), then the result is often ineffective instruction that does not teach. (p. 58)
Merrill goes on to say that "The consensus of 'stakeholders' often equals poor learning" and he raises doubts about whether students can play a meaningful participatory role in ID since "students are, for the most part, lazy." When instructional design theorists who work from a behavioral, information processing, or cognitive science paradigm have analyzed the R2D2 ID model, they have generally been critical. Dick (1966) compared R2D2 to his model and found it wanting while Merrill (2000) rejected outright the guiding principles. Merrill has gone so far as to declare that anyone who holds works from a different paradigm cannot call themselves instructional designers:
Those persons who claim that knowledge is founded on collaboration rather than empirical science, or who claim that all truth is relative, are not instructional designers. They have disassociated themselves from the technology of instructional design. We don't want to cast anyone out of the discipline of instructional science or the technology of instructional design; however, those who decry scientific method, and who deride instructional strategies, don't need to be cast off; they have exited on their own. (Merrill, Drake, Lacy, Pratt, & the Utah State University ID2 Research Group, 1996, p. 6)
It is not surprising that scholars such as Merrill are critical of ID models like R2D2 that are based on constructivist theory and interpretivist epistemologies. They are working from behavioral/information processing theories of learning and positivist/postpositivist epistemologies. Constructivist worldviews are based on different, often conflicting, foundational beliefs. The four guiding principles of the R2D2 model naturally derive from an interpretivist epistemology and a constructivist theory of learning. They are contrary to postpositivist epistemology and behavioral/information processing theories of learning.
R2D2 is, however, a particularly appropriate ID model for creating instructional materials to teach qualitative research methods. Many forms of qualitative research are based on some of the same guiding principles as the R2D2 model. Most forms of qualitative research, for example, are recursive and non-linear. The purpose, method and form of the research may change and evolve across the study. This generally calls for a reflective approach to research. A qualitative scholar cannot simply follow technical rules and a detailed, pre-established plan that will lead to a "good" study. Finally, several forms of qualitative research are participatory. Participatory action research and emancipatory research in the critical tradition come quickly to mind.
Thus, while R2D2 has been criticized heavily by some ID scholars who practice from a different paradigm, it seems particularly suited to guide the design of instructional materials for teaching qualitative research methods because the its foundations have much in common with a number of approaches to qualitative research. (And, as would be expected, researchers who use postpositivist and positivist paradigms make some of the same criticisms against qualitative research as postpositivist and positivist ID scholars made against R2D2.)
Focal points are convenient ways of organizing thinking about all the things that need to be done in a design project. R2D2 has three focal points: (1) Define, (2) Design and Develop, and (3) Disseminate. This study involved work in the first two focal points. It is important, however, to note that these are not linear stages. We did not complete all the work related to the define focus, for example, before moving on to work in the design and development focus. R2D2 ID is a spiral, which means some tasks were addressed many times across the project.
Define Focus. This focal point addresses the tasks that would, in a traditional instructional design model, be completed before actual design work begins. The Define focus in the R2D2 model continues throughout the design process. Early in the process, loose and ill defined ideas about the approaches that might be taken, the characteristics of students who will use the product, and the content to be taught guide the design process. These ideas gradually evolve, are revised, and emerge from the ongoing work of design and development. The same is true of the basic purpose of the instructional design project. In this study, we began with a general, if somewhat fuzzy, purpose-to create a multimedia program to support teaching critical qualitative research methods.
An important aspect of define focus is the creation of a participatory group. The learners are participating members of the development team rather than objects to be studied. Their input shapes the project's theme and purpose, the topics to be included, the instructional strategies used, and the look and feel of the interface (Schuler & Namioka, 1993). This project's team included experts such as faculty who teach courses in qualitative research, instructional design specialists, and interface design experts. Graduate students, who had completed qualitative research courses and who were instructional technology specialists, were another important part of the team. The end-users (faculty and students) were involved in the entire design process.
The Design and Development focus has three components: (1) preparation tasks, (2) creation, and (3) procedures.
Preparation tasks. These include selection of a development environment, media selection, and selection of instructional strategies. As with most aspects of the R2D2 model, initial, tentative decisions were made to begin the process, but they were always subject to both evolutionary and revolutionary change. Another preparation task is the selection of a development environment. Several criteria were considered when the initial decisions were made for the development environment. Since the product created is a hypermedia package that uses digital video stored on a CD-ROM, an authoring system capable of supporting CD video was required. Another requirement was cross platform compatibility since both Macintosh systems and Windows systems are popular in higher education. CRIT is non-linear and requires a program capable of creating non-linear information landscapes. In addition, we needed an environment that encouraged and supported recursion and reflection. A desirable environment should make it easy for material in all phases of design and development to be revised and edited. After considering all of these requirements, the authoring system SuperCard 2.5 (Allegiant Technologies Inc.) was selected. Programs created with SuperCard do not require a viewer or the actual software to run them and these executable files are distributable without paying a royalty. Supplemental software such as graphics programs, video editing packages, and audio editing suites were used as needed.
Two additional activities in this area are selection of media and selection of instructional strategies. Much of the material on qualitative research methods is in textual form (e.g., textbooks, articles, and book chapters). A medium that facilitates involving the student in the practice of the procedures with appropriate feedback and mentoring is needed (Crandall, 1993; Wolf, 1993). A hypermedia instructional package that presents video clips on the screen that students can use to practice gathering qualitative data meets this need. CRIT includes computer-controlled video and a computer program that provides instruction, guidance, feedback, and opportunities to compare a novice researcher's observations with those of more experienced researchers. A wide range of media are incorporated into CRIT including text, charts, graphics, animation, sound, and video.
Another reason a hypermedia, multimedia environment was selected is its non-linearity. Learners can make many decisions about what material they study and in what order. For this work, an instructional strategy that is well suited to the purpose is Jacobson and Spiro's (1993) cognitive flexibility hypertext. Hypertext environments promote cognitive flexibility - the ability to use knowledge and skills in flexible and innovative ways (Staninger, 1994). The general framework for CRIT was cognitive flexibility theory and within this framework several instructional approaches were available. The video case studies included on the CD-ROM constitute a rich instructional knowledge base the learner can explore and analyze.
The Creation Tasks. The creation tasks address the work involved in actual creation of the product. In the R2D2 model, a prototype is collaboratively developed with the instructional designer, experts in the relevant specialties, and students. For this study, a prototype was developed that included the elements of:
Drafts of the components of the instructional material were created and evaluated by members of the team (experts and students). The feedback obtained guided a series of revisions and refinements. This was a recursive process that progressed to the final version of the material. CRIT was created by first concentrating on the creation of the components noted above with input from the stakeholders guiding revisions. CRIT is organized as a web of knowledge (i.e., information nodes are linked to others based on their interrelationships). The network's complexity depends on the existing interrelationships.
According to Spiro, Feltovich, Jacobson, and Coulson (1991), "revisiting the same material, at different times, in rearranged context, for different purposes, and from different conceptual perspectives is essential for attaining the goals of advanced knowledge acquisition" (p. 28). CRIT permits students to progress through the information landscape many times from different perspectives. As the product was created, the design team continually revisited the material from different perspectives, as noted earlier, making revisions as needed. The feedback-revise-feedback process occurs across a progressively more "finished" product. It began with components of the product such as the user interface, scenario, and information content. It then progressed to a "single path prototype" (i.e., a version of CRIT that a student progresses through using the one developed path because that is the only path that has been created). From the single path prototype, work progressed to a full alpha version and then to beta versions. The last "beta" version was designated the final product.
Throughout the design and development focus, evaluation plays a critical role. Formative evaluations provide feedback to the development team that improves the product. The participatory design approach is a critical component of the effort to produce a quality instructional package. In addition to the informal evaluations and critiques, there were two formal evaluation strategies: (1) expert and stakeholder appraisal, and (2) student tryouts. Both were used to evaluate many versions of the material across the entire process.
The evaluation strategy for CRIT involved experts and end-users who assisted with two types of evaluation: component feedback and package critique. The R2D2 Model brings the student and experts into the development process in the beginning and uses their advice, evaluations, and feedback at every stage to improve the product (Willis, 1995). The information obtained from component feedback guided further design and development. When the single path prototype of CRIT was available, both experts and end-users provided detailed critiques. That process was repeated many times for both alpha and beta versions.
Procedure. CRIT gives students the opportunity to view a number of video cases. They use hypermedia/multimedia software to navigate through the material. They thus have flexible access to video from classrooms. Viewers see rich visual and verbal cues that can become the basis of their observations. The viewers witness the natural progression of a classroom setting and observe each video case. The program assumes students are already familiar with the particular research techniques they will use. Class discussions and reading assignments in the textbook are common methods of acquiring familiarity. In the program, an expert analysis was done for all video cases regarding the critical qualitative research techniques. Phillip Carspecken, an expert in the field of ethnography, wrote the analyses. The CD-ROM has a SuperCard stack that includes descriptions of techniques, the use of these techniques, practice of these techniques, and reference material. Students can explore the video cases, read the description of the techniques, look at expert analysis of any of the video cases, explore another video case, practice writing an analysis, and look at reference material. With the random access made possible using interactive multimedia, users can juxtapose information in a sequence of their own choosing and examine the content from multiple perspectives (Cennamo et al., 1995).
Decisions about the footage that constitutes a video case for the content included in the instructional package were decided by the team. The video cases were pre-existing data of middle-school students in a multicultural, natural classroom setting. The team searched the tape logs for scenes that represented topics discussed in Carspecken's book such as role settings, meaning fields, validity constructions, and power analysis. A compilation of those video clips, graphics, and sound were used to produce the final videotape. Editing was done on a manual-editing system. The final medium of video is CD-ROM because of its portability and cost.
Dissemination was not a part of the study. However, efforts are being made to obtain a publisher to distribute the product.
The end result of this study was The Critical Researchers Guide to Conducting Qualitative Research (CRIT), an interactive multimedia instructional package that supports teaching critical qualitative research techniques and strategies. The focus of CRIT is to introduce qualitative researchers to specific qualitative techniques, provide practice using those techniques, and provide expert guidance. CRIT can support any graduate course on critical qualitative research but it was designed to specifically compliment Dr. Phillip Carspecken's (1996) book, Critical Ethnography in Educational Research. Students read and discuss relevant chapters in that book, and then use CRIT to study the methods in a more concrete, hands-on format.
This section of the paper is divided into three parts: (1) chronology, (2) product overview, and (3) the results of formative evaluation procedures.
A working prototype of CRIT was produced after about two months of development, but the actual video used in the program was not developed or selected at that point. Working with experts and end users (graduate students), we selected short segments of video from a library of classroom video at the Center for Information Technology in Education at the University of Houston.
The video clips ultimately selected illustrated a variety of onsite classroom events depicting small group interactions and large group discussions. After the initial selection, a rough edit of the video footage was created. This video was reviewed extensively to identify unnecessary segments. The tape was reviewed by potential users of the product (graduate students taking qualitative research courses) and the rest of the stakeholders. Each individual examined it from his or her perspective.
After the videotape segments were selected, they were incorporated into the prototype. The prototype was a "single path prototype." One path through the material, using one of the video cases, was completed. The development team provided feedback on several versions of this single path prototype, and many revisions were made. When a new version generated no new suggestions for revision, the single path prototype was used as a model to develop all the other paths in the program. The first "full" version prototype was designated the alpha version. With feedback and the resulting revisions, the alpha version became the beta version, and then Version 1.0. Selecting the video segments required about two weeks and it took about four months to develop the single path prototype and the alpha and beta versions.
When the icon for CRIT is double-clicked, a title screen appears that contains the graphic on the cover of the book, tical Ethnography in Educational Research (Carspecken, 1996). There are two buttons on the title screen (see Figure 1), one for the first-time users and one for those who have previously used the program.
The "first-time user's" button takes the users to an overview of the program that describes what the program is about and the benefits the learner can expect from using this program (see Figure 2).
Although the user has the option to bypass the overview and start the program by clicking the "Experienced User" button (see Figure 1), it is important for the user to have an overall understanding of CRIT. At this point in the program, the user is presented with a button option that takes him/her to a set of directions on "how to use the program" (see Figure 3).
After reading the directions, the user can navigate through CRIT by clicking the "Start" button. However, the user does not have to read the directions or the overview to start the program as they can always access the help menu with those options. Students also can access the main menu by clicking the "Experienced User" button (see Figure 1).
The main menu consists of five graphics. Within each graphic there are text fields labeled Description, Expert, Practice, and Examples (see Figure 4).
All of the text fields within the graphics are hyperlinked to the appropriate sections of the instructional material:
In addition, on the screen shown in Figure 4 there is an icon (labeled Main Menu), which is one of the navigational tools featured in CRIT (see Figure 4). Once the user starts CRIT, the main menu icon appears on all the screens thereafter. Students can click this icon and return easily to the main menu regardless of where they are in the program.
CRIT covers five different critical qualitative research methods: meaning fields, role analysis, interactive sequences, validity horizons, and power analyses. Each of these methods has a description, a vide-case example, opportunities to practice the method, and examples of how the method has been used in social science research. The Description section on validity horizons, for example, allows the user to review a detailed description of a validity horizon and view a diagram representing a validity horizon (see Figures 5 and 6).
Figure 7 is an example of the Expert section. It is an analysis of meaning fields for one of the video clips.
A student looking at how an expert analyzed the validity horizon in a video clip of a teacher giving her class a spelling test would probably view the video clip first. Users have a set of video controls to play, pause, rewind, fast forward, or stop the video. There is also a button to pop up a text window that contains a transcription of the dialog from the video case. After viewing the video, the student can then look at the expert's data analysis.
The Practice section gives students the option of viewing a video clip, and then conducting their own qualitative analysis. The program provides both the case to be studied (on video that is displayed on the screen) and the forms or format for collecting the data (see Figure 8).
For example, the user can click on the Practice section of Meaning Fields and select a video clip. The choices are to (a) play the video, (b) pause the video, (c) write some claims, (d) view the transcription, (e) enter more claims, (f) save your work by clicking the "Add to Data File" button, or (g) "Go On to Subjective Claims".
Throughout the design and development process, both experts and potential users provided detailed critiques and suggestions for each version of the software. An overview summary of input from stakeholders and the changes that resulted from that participation is explored in the next section.
Input and feedback from students and experts were important components of the developmental process. The student group (5) that evaluated the program included graduate students who had taken the Ethnography and Cultural Studies course at the University of Houston. Those in the expert group (6) included individuals who had expertise in instructional design, instructional technology, and qualitative research.
In the beginning of the design process, the primary goal was to develop the look and feel of the program, (i.e., the user interface). The goal was to design something that would catch learners' attention, be informative, and at the same time not prescribe a set way of navigating through the program. The first attempt at an interface was a rockscape-style background with contrasting color hotspots pointing to the specific research methods (see Figure 9).
This primitive version of CRIT was too "busy" and did not seem to mesh with the subject matter. This was a milestone, however, because it provided a beginning point. Many changes were tried and rejected before the team agreed that we had a pleasing, appropriate background, and a workable user-interface that did not suggest to students that there was one "correct" or linear path through the various options of the program.
Discussion with the expert stakeholders suggested that, ideally, the menu-bar should appear on every screen to provide the user with the options of viewing video, looking at expert analyses, practicing, looking at descriptions of techniques and how they were used, or exiting the program. Making the menu-bar omnipresent provides the user with the flexibility to "jump" anywhere in the program at any time. The input from stakeholders suggested that a thematic or metaphoric menu would fulfill the requirements. Other elements of the program such as the format, video cases for practice, and explanations of the research method including examples and illustrations of data gathered by others, were arrived at rather easily. Two of the experts took the lead in suggesting the case format and the options available to students. The navigational options and how to present them were a different matter. Much of the design and development effort was invested in selection of the video cases and the design of the navigational options.
The process was discouraging at this point. The attempt at a thematic menu was a file cabinet labeled with the topics in the program. Stakeholders felt this theme was too common and that it did not represent the subject matter of the program. The stakeholders' input caused a major revision to the way the options were laid out. Eventually the menu structure was organized around iconic images that stood for each of the research methods covered. Sources for images included Internet, clip art programs, and photo CDs. After considerable searching, graphics were located that could be used legally in the program that figuratively represented the techniques. After some changes based on feedback from stakeholders, consensus on the graphics used for the main menu was reached. Figure 4 shows the graphics used to represent each of the research techniques.
The graphics were chosen, but the task of arranging them on the main menu screen remained. The layout was important because they needed to be arranged so the user could easily navigate through the program with no constraints from the interface. It also was important for the user to have access to all the possible paths of information retrieval from the beginning of the program. For example, the user could click within any of the five figuratively represented techniques and select description, expert analysis, practice, or examples (see Figure 4). There was no requirement for students to complete a particular activity before going on to a second activity.
Once the basic structure of the user interface was in place, the next step was to develop first one path and then full versions of the program.
Work on the prototype took about four months. Initial work was a rough mock-up (built in Authorware) of early design ideas. The interface was demonstrated to several users and experts. Feedback was gathered broadly on the general concepts and approach. Significant changes were made and a new version of the prototype included an extended series of design walkthroughs. Feedback on these walkthroughs led to a number of revisions.
Five individuals with expertise in instructional technology and design used the single path prototype and offered detailed suggestions. Also, four graduate students who were potential users also used the prototype and critiqued it. These individuals were invited to observe, explore, construct, and evaluate their interpretations and opinions of the program. The process of using CRIT was the focus of evaluation in these tryouts. The results of the prototype, alpha, and beta tryouts of CRIT by experts and potential users are combined here. Data came from open interviews and a short questionnaire guided revisions. Each participant spent about an hour on each tryout. Many changes were made as a result of feedback. These changes are described below in the three categories: (1) conceptual, (2) cosmetic, and (3) media and implementation.
Conceptual. Conceptual suggestions were about the major ideas or themes within the program. Below are some of the revisions made based upon feedback from various stakeholders.
The main menu, or "homepage," is the entry point to the content CRIT presents. Main menu implies some type of top-level index. A common assumption about interactive instructional programs based on traditional paradigms is that there is a single starting point to begin the program. A second common assumption is that there is a sequential order for content presentation.
The students who tried the menu-bar liked it, but one of the experts suggested metaphorically presenting the information. Using feedback from one of the expert stakeholders, a file cabinet was used as a main menu. Other experts objected to the file cabinet, commenting that the metaphor implied a linear structure. Taking that into consideration, graphics that would figuratively represent the reconstructive analysis techniques featured in the program were selected. This involved a number of iterations as images were considered, critiqued, and then rejected.
Cosmetic Revisions. Numerous changes were made that, while cosmetic in nature, contributed to the "finished" or "professional" look of the product. For example, all of the individuals in the tryout-sessions had difficulty reading the text on the screen because the background was transparent. Burgundy text on a sand-marble background was originally used. As a result of the comments, all text backgrounds were changed to white opaque backgrounds (see Figure 2). Other cosmetic revisions included changing the "busy" and inconsistent layouts of several screens, and improvements in the Overview section that involved dividing it into two parts. Dividing the section into two parts reduced the amount of text a person had to read in a section and made the instructions for use clearer and easier to understand.
Other changes included adding a video counter to allow students to determine exactly where they were in a video clip as they wrote their analyses. This caused another problem, however. The cursor "jittered" on the screen and was distracting enough to render the video counter unusable. Instead, a full-text transcription of the video was provided in place of the counter. The user has the option to click on the button for a verbatim transcript of each video case.
Finally, experts and users found many spelling and grammatical errors that were corrected as they were identified. The subject matter expert reviewed the content and a number of changes were made based on that review. Also, many of the student users felt the text content was too verbose and cumbersome. The content matter expert revised some of the content focusing on cutting down the amount of text.
Media and Implementation Changes. The original video was on S-video tapes. It was digitally edited and saved using MPEG compression. The size of the video files range from 250 to 350 megabytes; the Authorware program or file is 4 megabytes. With three video clips and the Authorware program, storage capacity became a problem. In order for the video clips and the program to fit on a standard CD-ROM, the video clips were saved in "thousands" of colors, instead of millions of colors. That reduced the size of the video files and made it possible to put CRIT on a CD-ROM.
The overall reaction from stakeholders and students was that this type of program is valuable for learning the content. All believed that the final version of the instructional package was easy to use and was a worthwhile learning experience. The match between content intended for teaching advanced qualitative research techniques, and teaching strategy (video cases) was considered appropriate.
Specific suggestions and areas that would benefit from revision were offered by many of the people who participated in the design and development process. The CRIT video component was noted as an area of potential strength in its representation of techniques. Some participants questioned the length of the video clips and the clips were consequently reviewed. More video data was added so students could see more context and develop a better feel for the situation being portrayed. Video cases in the final version ranged from 2.5 to 3 minutes.
CRIT's navigational component was another strength cited by the stakeholders. In observing some of the tryouts, the instruction was interactive and engaging; learners found themselves engrossed. More knowledgeable learners were comfortable with the navigation as well. Also, learners with a richer and more extensive base of subject-matter knowledge seemed to quickly grasp the "big picture" of the content and the program. The novelty and appeal of working with authentic cases was appealing to the students and was also considered a strength.
One student expressed concerns about the content of the video and the need for the cases to be more ethnically diverse. A range of ethnic backgrounds is represented in the cases-Anglo, African American, Hispanic-and most of the participants felt the video cases were diverse in terms of student representation.
The concept of practicing the qualitative techniques in real-time was highly praised, as well as the user-friendliness of CRIT. Both experts and student evaluators thought the final version was effective in helping students better understand the qualitative research techniques.
If the reactions of stakeholders to CRIT were distilled to a few comments, most would focus on the positive benefits of authentic practice and video cases. The video helped students visualize concepts, and the ease of navigation and logical flow to and from different parts of the program allowed students to concentrate on learning. Students were particularly pleased with the opportunity to see actual video cases to which they could apply the research skills they were learning.
This study illustrates two "trends" in education today. The first is the creation of electronic support materials for all types of content - from kindergarten to graduate school. When instructional designers create electronic content, they often use instructional strategies and approaches that come primarily from the behavioral side of psychology. Some authors even feel that behavioral approaches dominate the educational uses of technology today. CRIT illustrates the point that other worldviews - critical theory and constructivism/interpretivism - can be used to make decisions both about what content should be included and the instructional strategies that will be used. Information technology is a flexible and fluid medium, and in the case of CRIT, it became a powerful means of helping students learn some admittedly complex and difficult research methods.
The second "trend" this study represents is the move away from linear models of instructional design (ID) based on behavioral psychology and an increasing interest in ID models based on constructivist as well as chaos theory. The R2D2 model is non-linear, recursive, reflective, and participatory. Traditional ID models are linear, tend to emphasize the role of the designer as THE expert, and frequently involve very little participation on the part of stakeholders. ID is a well-established field within educational technology, and perhaps because it has its origins in the behavioral psychology in the 1960s and 70s, it has been slower to seriously consider other foundations. Other areas in education and the social sciences have already begun to incorporate epistemologies, methodologies, and teaching/learning strategies from other paradigms. ID, and educational technology in general, has been much slower to adopt other perspectives. This study illustrates how an "alternative" approach to teaching "alternative" content would be supported by material developed using an "alternative" ID model.
The ID model used violated many of the "rules" of traditional design. The designer played a collaborative and facilitative role rather than an expert role. Goals and objectives, as well as content and teaching strategies were not decided in advance by the expert. The act of decision-making was based on the understanding, beliefs, and values of the stakeholders. The process that guided the design and development of CRIT included negotiating a set of shared understanding, beliefs, and values.
In addition, there were no prescribed linear steps; rather questions were addressed in a spiral fashion. The same issues were addressed many times, but often at different levels or with different perspectives because the team developed better understanding of the project as design and development progressed.
Using R2D2, decisions were made but they were never final. The R2D2 model may not be for everyone. This model calls for beginning design and development with "fuzzy," ill-defined goals and objectives that change as work progresses. This can be frustrating for designers who traditionally use a linear model. Frustration arises from constant revision and modifications of the program. Design and development may take less time and effort with a traditional design model that starts with a specific set of prescribed objectives. However, the CRIT program likely would not have been as creative or effective if we had used that type of design model. Constant feedback and input from stakeholders made this project a collage of ideas and creativity. The program created using the R2D2 model is quite different from the one that would have been created using a traditional linear model.
In summary, we feel CRIT is a significant contribution to the resource base for teaching qualitative research methodology. It is more an example of what is needed, however, than a solution. There are hundreds of qualitative data collection and analysis methods. Only a few of them were addressed in CRIT. We need many more of this type of instructional resource. Think, for example, of a CD-ROM or Web-based resource that helps students use any of six or seven different formats to conduct interviews and analyze the results, or to conduct participant observer studies in school classrooms, or collect an analyze historical data on racial biases in local government decisions. Qualitative research methodology is not as cut and dried as many forms of quantitative research, but that does not mean that "anything goes." Becoming a scholar who can conduct quality qualitative research is not easy, and courses that support that goal can benefit from a large library of technology-enhanced instructional resources. We believe R2D2 is a flexible model for guiding the design of such instructional materials, and recommend it to you for consideration.
Allegiant Technologies, 6496 Weathers Place, Suite 100, San Diego, CA 92121
Andrews, D. H., & Goodson, L. A. (1995). A comparative analysis of models of instructional design. In G. Anglin (Ed.), Instructional technology. Past, present, and future (pp. 161-182). Englewood, CO: Libraries Unlimited, Inc.
Bagdonis, A., & Salisbury, D. (April 1994). Development and validation of models in instructional design. Educational Technology, 34(4), 26-32.
Borg, W., & Gall, M. (1989). Educational research: An introduction. White Plain, N.Y., Longman.
Burton, J. K., Moore, D. M., & Magliaro, S. G. (1996). Behaviorism and instructional technology. In D. H. Jonassen (Ed.), Handbook of research for educational communications and technology (pp. 46-73). New York: Simon & Schuster Macmillan.
Carspecken, P. F. (1996). Critical ethnography in educational research: A theoretical and practical guide. New York, NY: Routledge.
Cennamo, K., Abell, S., Chung, M., Campbell, L., and Hugg, W. (1995). A "Layers of Negotiation" model for designing constructivist learning materials. Proceedings of the Annual National Convention of the Association for Educational Communications and Technology (AECT), Anaheim, CA, 95, pp. 32-42.
Cennamo, K. S., Abell, S. K., and Chung, M. L. (1996). A "Layers of Negotiation" model for designing constructivist learning materials. Educational Technology, July-August 1996, 39-48.
Crandall, J. (1993). Professionalism and professionalization of adult ESL literacy. TESOL Quarterly, 27(3), 497-515.
Denzin, N., & Lincoln, Y., (Eds.). (1994). Handbook of qualitative research. Thousand Oaks, CA: Sage.
Dick. W. (1996). The Dick and Carey model: Will it survive the decade? Educational-Technology-Research-and-Development, 44(3), 55-63.
Dick, W., & Carey, L. (1966). The systematic design of instruction (4th ed.). New York: HarperCollins.
Eisner, E. (1979). The educational imagination: On the design and evaluation of school programs. New York: McMillan.
Eisner, E.W., & Peshkin, A. (1990). Qualitative inquiry in education: The continuing debate. New York: Teacher's College Press.
Jacobson, M., & Spiro, R. (1993). Hypertext learning environments, cognitive flexibility, and the transfer of complex knowledge: An empirical investigation. Center for the Study of Reading Technical Report. College of Education, University of Illinois at Urbana-Champaign.
Lebow, D. (1993). Constructivist values for instructional systems design. Five principles toward a new mindset. Educational Technology, Research, and Development, 41(3), 4-16.
LeCompte, M. D., & Preissle, J. (1993). Ethnography and qualitative design in educational research (2nd ed.). San Diego, California: Academic Press.
Merrill, M. (1996, July-August). What new paradigm of ISD? Educational Technology, 36(6), 57-58.
Merrill, M. D., Drake, L, Lacy, M, Pratt, J., & the Utah State University ID2 Research Group. (1996). Reclaiming instructional design. Educational Technology, 36(5), 5-7. Available: http://www.coe.usu.edu/it/id2/reclaim.html
Schön, D. (1983). The reflective practitioner: How professionals think in action. New York: Basic Books.
Schön, D. (1987). Educating the reflective practitioner. San Francisco: Jossey-Bass.
Schuler, D., & Namoika, A. (1993). Participatory design: Principles and practices. Hillsdale, NJ: Lawrence Erlbaum.
Spiro, R., Coulson, R., Feltovich, P., & Anderson, D. (1988). Cognitive flexibility theory: Advanced knowledge acquisition in ill-structured domains. Center For the Study of Reading. Technical Report No. 441, College of Education, University of Illinois at Urbana-Champaign.
Spiro, R., Feltovich, P., Jacobson, M., and Coulson, R. (1991). Cognitive flexibility, constructivism, and hypertext: Random access instruction for advanced knowledge acquisition in ill-structured domains. Educational Technology, 31, 24-33.
Spiro, R., & Jehng, J. (1990). Cognitive flexibility, constructivism and hypertext: Theory and technology for the nonlinear and multidimensional traversal of complex subject matter. In D. Nix & R. J. Spiro (Eds.), Cognition, education, and multimedia: Exploring ideas in high technology (pp. 163-205). Hillsdale, NJ: Lawrence Erlbaum Associates.
Staninger, S. (1994, July-August). Hypertext technology: Educational consequences. Educational Technology, 51-53.
Strauss, A., & Corbin, J. (1990). Basics of qualitative research: Grounded theory procedures and techniques. Newbury Park, CA: Sage Publications, Inc.
Webb, R. B., & Glesne, C. (1992). Teaching qualitative research. In M. D. LeCompte, W. L. Millroy, & J. Preissle (Eds.). The handbook of qualitative research in education (pp. 771-814). San Diego, CA: Academic Press.
Wittgenstein, L. (1958).
Philosophical investigations (3rd ed.). (Trans. G. Anscombe). New
York: Macmillan. Willis, J. (1995). A recursive,
reflective instructional design model based on
constructivist-interpretivist theory. Educational Technology ,
35(6), 5-23. Willis, J. (2000). The maturing of
constructivist instructional design: Some basic principles that can guide
practice. Educational Technology, 40(1), 5-16. Wilson, B. G. (1993).
Constructivism and instructional design: Some personal reflections.
Proceedings of Selected Research and Development Presentations at the
Convention of the Association for Educational Communications and
Technology Sponsored by the Research and Theory Division. 15th , New
Orleans, Louisiana, January 13-14, 1993. Wilson, B. (1997). Reflections on
constructivism and instructional design. In C. Dills & A. Romiszowski
(Eds.), Instructional development paradigms [Pages needed
here]. Englewood Cliffs, NJ: Educational Technology Publications. Winn, W. (1992). The assumptions of
constructivism and instructional design. In T. Duffy & D. Jonassen (Eds.),
Constructivism and the technology of instruction: A conversation
(pp. 177-182). Hillsdale, NJ: Lawrence Erlbaum. Wolf, M. (1993). Mentoring middle-aged
women in the classroom. Adult Learning, 4(5), 8-9. You, Y. (1994). What can we learn from
chaos theory? An alternative approach to instructional design.
Educational Technology Research and Development, 41(3), 17-32.
+Brandie Colón is Senior
Systems Support Specialist with American Express in Ft. Lauderdale,
Florida. Her email address is Brandie.Colon@aexp.com.
Wittgenstein, L. (1958). Philosophical investigations (3rd ed.). (Trans. G. Anscombe). New York: Macmillan.
Willis, J. (1995). A recursive, reflective instructional design model based on constructivist-interpretivist theory. Educational Technology , 35(6), 5-23.
Willis, J. (2000). The maturing of constructivist instructional design: Some basic principles that can guide practice. Educational Technology, 40(1), 5-16.
Wilson, B. G. (1993). Constructivism and instructional design: Some personal reflections. Proceedings of Selected Research and Development Presentations at the Convention of the Association for Educational Communications and Technology Sponsored by the Research and Theory Division. 15th , New Orleans, Louisiana, January 13-14, 1993.
Wilson, B. (1997). Reflections on constructivism and instructional design. In C. Dills & A. Romiszowski (Eds.), Instructional development paradigms [Pages needed here]. Englewood Cliffs, NJ: Educational Technology Publications.
Winn, W. (1992). The assumptions of constructivism and instructional design. In T. Duffy & D. Jonassen (Eds.), Constructivism and the technology of instruction: A conversation (pp. 177-182). Hillsdale, NJ: Lawrence Erlbaum.
Wolf, M. (1993). Mentoring middle-aged women in the classroom. Adult Learning, 4(5), 8-9.
You, Y. (1994). What can we learn from chaos theory? An alternative approach to instructional design. Educational Technology Research and Development, 41(3), 17-32.
Kay Ann Taylor is a Ph.D. student in the Curriculum and Instruction program at Iowa State University. Her dissertation is a study of African America educators in Iowa. Her e-mail address is email@example.com or firstname.lastname@example.org.
Jerry Willis, Ph.D. is Professor and Program Coordinator of the Ph.D. program in Curriculum and Instructional Technology at Iowa State University. His e-mail address is email@example.com.
Colón, B., Taylor, K. A., & Willis, J. (2000, May). Constructivist instructional design: Creating a multimedia package for teaching critical qualitative research [100 paragraphs]. The Qualitative Report [On-line serial], 5(1/2). Available: http://www.nova.edu/ssss/QR/QR5-1/colon.html
Brandie Colón, Kay Ann Taylor, and Jerry Willis
Return to the top of the paper.
Return to the Table of Contents.