Enhancing the Rigor of Qualitative Research: Application of a Case Methodology to Build Theories of IT Implementation
by
Guy Paré
+

The Qualitative Report, Volume 7, Number 4 December, 2002
(http://www.nova.edu/ssss/QR/QR7-4/pare.html)


Abstract

This paper presents and illustrates how the approach proposed by Eisenhardt (1989) for building theories from intensive qualitative research, more precisely case study research, can help information systems and medical informatics researchers understand and explain the inherently dynamic nature of IT implementation. The approach, which adopts a positivist view of research, relies on past literature and empirical data as well as on the insights of the researcher to build incrementally more powerful theories. We describe in some detail how this methodology was applied in a particular case study on IT implementation in the health care context and how the use of this approach contributed to the discovery of a number of new perspectives and empirical insights. Furthermore, we provide insights into the many choices that a researcher must make when adopting this methodological approach. Overall, using Eisenhardt's approach as a starting point, our objective is to provide a rigorous, step-by-step methodology for using case studies to build theories within the information systems and medical informatics fields. We provide several insights to the nature of case research, information on and concrete examples of specific techniques and tools, and guidance on how to improve intensive case research.

Key Words: Case Study Research, Positivist Research, Theory Building, Information System Implementation, and Medical Informatics.

Introduction

Health care throughout the world has become more and more complex and health care systems worldwide face the same three challenges, namely, cost, quality, and access. First, overall health care spending is rising because of aging populations, advancing medical knowledge, the growing cost of drugs and medical technology, and increased investments in information technology (IT). Second, health care policymakers increasingly recognize the extent of medical errors and how IT can help. The recent Institute of Medicine (Kohn, Corrigan, & Donaldson, 1999) study, which claimed that medical errors cause between 44,000 and 98,000 deaths in the U.S. each year, attracted much media attention. As medical horror stories become less acceptable, health care organizations must aggressively adopt IT applications - specifically computer-based patient record (CPR) systems - to reduce the medical error rate. Last, health care consumerism will continue to increase as more people go to the Internet for health information. A recent survey conducted by Fox and Rainie (2000) revealed that 52 million American adults have used the Web to get health or medical information. These "health seekers" say the resources they find on the Web have a direct effect on the decisions they make about their health care and on their interactions with doctors. In short, given the myriad complicated elements in today's health care environment, the geographic dispersion of points of care, the interaction of many specialists and the need to balance the complex set of care steps for every patient, improved quality of care cannot be delivered without substantial application of IT.

Several empirical studies have shown that the use of computer-based applications could have positive impacts on organizational performance (Chan et al., 1997; Raymond et al., 1995) and many agree to say that health care institutions are no exceptions (Edwards, 1999; Hatcher, 1998; Lorenzi & Riley, 1995). But for a variety of reasons, health care has traditionally been a "dabbler" in the IT area. For instance, several large medical centres continue to function with rudimentary paper-based information systems (Lorenzi & Riley, 1995). Fortunately, this is rapidly changing and IT spending in the medical field is projected to increase significantly over the next few years in North America and European countries (Dash, 2000; Garets & Frey, 2001; Gurasekaran, 2000; Potter, 2000). Health care providers and managers as well as clinicians around the world are starting to realize the enormous opportunity to improve quality and access and cut costs through automation. As an example, a province-wide telecommunication network in Quebec, Canada, is being deployed to allow fast, real-time access to clinical data and administrative information and greater expertise sharing among clinicians (Paré & Sicotte, 2001). Several computer-based applications such as intranet, extranet, Web EDI, telemedicine and videoconferencing will be progressively implemented throughout the network along with the deployment and increasing use of smart cards. In North America, computerization of medical records has increased in recent years and it appears that this trend is likely to continue, particularly as technology improves and becomes more affordable and as the demand for health care information increases (Edwards & Purchase, 2000; Potter, 2000). But the current and rapid movement of informatics projects into health care organizations has raised managerial concern regarding the capability of today's institutions to satisfactorily manage their introduction (Lorenzi & Riley, 2000). Implementing clinical applications in extremely complex organizations that operate on a 7-day by 24-hour basis is not easy (Lorenzi et al., 1998).

The risk of IT project failure is alarming high across all industry segments, including health care (Sinclair & Cook, 2000). A survey conducted by the Standish Group revealed that as many as half of all IT projects exhibited significant cost and schedule overruns. Of the $250 billion spent in 1997 on IT projects, 31% of the projects were cancelled prior to completion. Although investment in IT continues to increase in the health care industry, projects continue to fail (Charles et al., 1997; Goddard, 2000). Indeed, a large proportion of clinical and administrative systems are underutilized, do not meet their potential, or fail to be used at all (e.g., Dambro et al., 1998; Goddard, 2000; Lawler et al., 1996). Why do IT failures occur? One explanation is that key actors and managers are not taking prudent measures to assess and manage the risks involved in these projects. Quantitatively oriented researchers have built models that identify a variety of critical factors affecting computer-based system success. Lack of top-management support, poor quality system design, adequate resources not available from the beginning, hostile culture toward the information systems area; lack of commitment to change, inadequate testing, and insufficient project planning figure among the most cited risk factors (e.g., Cho et al., 1992; Linder, 1992; Lorenzi & Riley, 2000; Swanson, 1988). Although there is typically no one single cause in a given implementation case, a snowball effect is often seen, with a shortcoming in one area leading to subsequent shortcomings in other areas (Lorenzi & Riley, 2000). In short, research in this area has mainly resulted in a set of managerial prescriptions which, taken altogether, constitute the "ideal" implementation scenario.

Advocates of project management claim that by countering these threats to success, the incidence of failure can be reduced (Boehm, 1991; Charette, 1989). But researchers know very little about how and why the critical factors included in these models interact and work together to produce success or failure. As a result, researchers lack a full understanding of the implementation process that is necessary to guide health care practitioners to attain positive organizational outcomes. It thus appears important to ask ourselves, as researchers, how to improve not only our research models, but also our methodologies so that the results of our work can be of greater value to practitioners. In our opinion, further progress will require more complex, realistic research models and the development of alternative perspectives for viewing IT implementation projects in health care organizations. We also posit that qualitative research methods can play a major role in shedding light on this complex phenomenon.

There is a growing tradition of using qualitative research approaches to study information technology phenomena (e.g., Nelson et al., 2000; Romm & Pliskin, 1997; Trauth & Jessup,2000; Wixon & Ramey, 1996) and case study research figures among those qualitative methods that have been recognized as having gained increasing acceptance over the past decades in the information systems (IS) field (Benbasat & Weber, 1996; Benbasat et al., 1987; Klein & Myers, 1999; Lee, 1989; Orlikowski & Baroudi, 1991). Towards the end of the 1980s, researchers started to evaluate the usefulness and methodological soundness of case research conducted in IS. One of the earliest contributions was that of Benbasat et al. (1987) who surveyed the case study literature published in four journals and one major conference proceedings for the 1981-1985 period and evaluated their sample based on a number of guidelines associated with the case design and data collection process. They recommended that case researchers should: provide clearer descriptions of where their topics fit into the knowledge building process; detail the case selection criteria; and provide more information about the data collection process. Another key contribution was made shortly after by Lee (1989) who provided an overview of and responded to the methodological problems involved in the study of a single case and summarized what a scientific methodology for IS case studies does, and does not, involve. Lee also demonstrated how to make controlled observations and deductions as well as how to allow for replicability and generalizability with the use of a single case. As a result of contributions like these, case research is now considered as a useful and relevant research strategy within the IS research community (Klein & Myers, 1999).

In this paper, we present and illustrate how the methodological approach proposed by Eisenhardt (1989) for building theory from intensive case study research can help researchers understand and explain the inherently dynamic nature of IT implementation in the health care context. Several pieces of the methodology proposed by Eisenhardt are borrowed from extant literature. Specifically, the approach includes key ideas and concepts from the work of Glaser and Strauss (1967) on grounded theory, Yin (1994, 1999) on case study research, Miles and Huberman (1994) on qualitative data analysis, Van Maanen (1988) on ethnography and Jick (1979) and Patton (1999) on triangulation of data types, to name a few. While these previous writings provide pieces of the process and focus on defending building theory from cases, the work of Eisenhardt focuses on how to actually build theory from case studies. The approach relies on past literature and empirical data as well as on the insights of the researcher to build incrementally more powerful theories.

We used a multiple-case design within the positivist tradition (Yin, 1994). Positivist research, which has its roots in the natural sciences, assumes an objective physical and social world that exists independent of humans. The main role of researchers is to discover this reality by crafting precise measures that will detect and gauge those dimensions of reality that interest the researcher. Understanding phenomena is thus primarily a problem of modeling and measurement (Orlikowski & Baroudi, 1991). Positivist research is concerned with the empirical testability of theories. It works in a deductive manner to discover causal relationships that are the basis of generalized knowledge. Importantly, researchers conducting positivist studies see themselves as impartial observers who can objectively evaluate or predict actions or processes, but who cannot engage in moral judgments or subjective opinion. They are detached from the phenomena of interest (Orlikowski & Baroudi, 1991). Subscribing to such orientation to research, our approach is based on predefined research questions, a consideration of a priori measurable constructs, and it is directed toward the development of theoretical propositions or hypotheses. We also used published methodological guidelines (Lee, 1989; Yin, 1994) to ensure validity and reliability.

In order to provide a practical demonstration of how the methodology proposed by Eisenhardt can be used in the context of clinical information systems implementation, this paper illustrates it with extensive material taken from an actual, published case study (Paré, 2002; Paré & Elam, 1998; Paré et al., 1996; Paré et al., 1997). This in-depth case study examines the implementation process, use and consequences of three CPR systems at a large tertiary care teaching hospital. We show that the Eisenhardt approach to theory-building using case study research can be successfully applied in such context and can contribute to the discovery of new phenomena; reflecting the need for a more in-depth understanding of naturalistic settings, the importance of understanding context, and the complexity of implementing organizational and technological change. In addition, we provide insights into the many choices that a researcher must make when adopting this methodological approach. We also discuss when it is appropriate to follow, to ignore, or to modify the suggestions made by Eisenhardt.

We believe that our paper makes two important contributions to the information systems and medical informatics fields. First, using Eisenhardt's methodology as a starting point, we provide a creative and rigorous approach for using intensive qualitative research to build theories of IT implementation. Qualitative research practiced from a positivist perspective is not all that common in the IS field so the present article represents a key contribution in this discussion. Second, the view that qualitative research in many fields, including information systems, is not rigorous stems from the lack of familiarity with the "practices employed by qualitative researchers to deal with problems […] that include investigator bias, questionable measurement reliability and validity, and the level of appropriateness of analytic techniques" (Rundall et al., 1999, p. 1092). Serving as an exemplar for the positivist case research methodology, the present article makes an important contribution to the development of qualitative research knowledge and skill among information systems and medical informatics researchers. Overall, it provides several insights to the nature of case research, information on and concrete examples of specific techniques and tools, and guidance on how to improve intensive case research.

A Roadmap to Studying IT Implementation Process

This section describes the roadmap proposed by Eisenhardt (1989) for building theories from case study research and illustrates how this methodological framework can serve as a useful guide to researchers interested in studying the complex and dynamic nature of CPR implementation. The roadmap is summarized in Table 1.

Table 1. Process of Building Theory from Case Study Research (Eisenhardt, 1989)

StepsActivitiesReasons
1. Getting Started
  • Definition of research questions
  • Possibly a priori constructs
  • Neither theory nor hypotheses
  • Focuses efforts
  • Provides better grounding of construct measures
  • Retains theoretical flexibility
2. Selecting cases
  • Specified population
  • Theoretical sampling
  • Sharpens external validity
  • Focuses efforts on cases that replicate or extend theory
3. Crafting instruments and protocols
  • Multiple data collection methods
  • Qualitative and quantitative data combined
  • Multiple investigators
  • Strengthens grounding of theory by triangulation of evidence
  • Synergistic view of evidence
  • Fosters divergent perspectives and strengthens grounding
4. Entering the field
  • Overall data collection and analysis
  • Flexible and opportunistic data collection methods
  • Speeds analysis and reveals helpful adjustments to data collection
  • Allows investigators to take advantage of emergent themes and unique case features
5. Analyzing data
  • Within-case analysis
  • Cross-case pattern using divergent techniques
  • Gains familiarity with data and preliminary theory generation
  • Forces investigators to look beyond initial impressions
6. Shaping hypotheses
  • Replication, not sampling, logic across cases
  • Search evidence of "why" behind relationships
  • Confirms, extends, and sharpens theory
  • Builds internal validity
7. Enfolding literature
  • Comparison with conflicting and similar literature
  • Builds internal validity
  • Sharpens external validity
8. Reaching closure
  • Theoretical saturation when possible
  • Ends process when marginal improvement becomes small

Step 1: Getting started

According to Eisenhardt, three issues are of great importance in getting started, namely, the initial definition of research questions, the a priori specification of constructs, and the consideration of a priori theory or hypotheses. Each of these issues will be examined in turn.

Initial definition of research questions

First, an initial definition of one or more related research questions, in at least broad terms, is as important in building theory from case studies as it is in hypothesis-testing research. Without a research focus, it is easy to become overwhelmed by the volume of qualitative data (Yin, 1994). The ultimate intent of our study was to broaden and strengthen our understanding of CPR systems implementation by researching the dynamic nature of the implementation process. More specifically, efforts were directed toward opening the "black box" and providing the story that explains how and why contextual conditions and implementation tactics and strategies interact and work together to produce CPR project outcomes. In pursuit of this objective, two interrelated research questions were initially stated: "What are the laws of interaction which characterize the dynamic nature of CPR systems implementation?" and "How and why do contextual conditions and implementation tactics interact and work together to ensure CPR project success?" As explained below, these research questions provided a well-defined focus to our research and allowed us to specify the kind of data to be gathered.

A priori specification of constructs

With respect to the issue of using existing theoretical constructs to guide theory-building research, two different approaches may be taken (Anderson & Aydin, 1994). In the first, the researcher works within an explicit conceptual framework. A conceptual framework "consists of a selection of concepts and relations among them, grouped so as to enable its users to easily see the major concepts simultaneously in their relations to one another" (Kochen, 1986, p. 93). Therefore, a conceptual framework becomes a "researcher's first cut at making some explicit theoretical statements" (Miles & Huberman, 1994, p. 91). In the second, the researcher tries not to be constrained by prior theory and instead sees the development of relevant theory, hypotheses, and concepts as a purpose of the project. In the present study both approaches were combined since the main intent was to provide freshness in perspective to an already researched topic. Importantly, given that this study was aimed at theory building, not theory testing, the conceptual framework (and its constructs) was used solely as a starting point. As prescribed by Yin (1999), even exploratory case study research should make use of a conceptual framework to define the priorities to be explored. In the context of our study, the use of a framework helped us make sense of occurrences, ensured that important issues were not overlooked, provided a set of constructs to be investigated, and guided our interpretation and focus. Using the research questions as a guide, our conceptual framework (see Appendix I) grouped constructs related to the contextual conditions surrounding most implementation situations (e.g., resources availability, top management support, user attitudes, system characteristics), the tactics and strategies aimed at launching the project, managing the development of the new system and preparing clinicians for the new computer application, and the different criteria commonly adopted to evaluate system success (e.g., system use, user satisfaction with system, individual consequences, satisfaction with overall implementation process). In this light, the conceptual framework provides an insightful way to study the process of implementing clinical applications. It suggests that researchers should carefully pay attention to contextual conditions, human actions (both those of the implementers and the actions of those who are the targets of the implementation), and their interaction in order to better understand CPR systems success. However, as stressed by Eisenhardt (1989), although early identification of possible constructs allows them to be explicitly measured in interviews and questionnaires, it is equally important to recognize that the identification of constructs is tentative in theory-building research. We found this to be true as new factors were found during data collection that needed to be added to the analysis.

Consideration of a priori theory or hypotheses

The primary objective of our study was to develop a process theory of CPR systems implementation. Eisenhardt suggests that theory-building research must begin as close as possible to the ideal of no theory under consideration and no hypotheses to test since preordained theoretical perspectives may bias and limit the findings. However, as stressed by Eisenhardt, it is quite impossible to achieve the ideal of a clean theoretical slate. Hence, although we followed her suggestion in terms of not identifying specific relationships between the constructs identified in our conceptual framework, we found it necessary to make use of a process meta-theory called the teleological view (Van de Ven & Poole, 1995). Indeed, a caveat for existing process models raised by Mohr (1982) seems particularly relevant to systems implementation studies. Precisely, Mohr argues that it is not enough for such models to supply the succession of events (such as in stage models). Rather, he posits that process models must provide the external forces and probabilistic processes constituting the means by which events are understood to unfold. In accord with Mohr, we believe process explanations become more meaningful when situated within a broader or higher level of process theory. The adoption of a particular meta-theory, namely, teleological theory reflected our basic assumptions about the nature of the phenomena being studied; assumptions that were supported by strong evidence in the data.

The teleological view of process theory shapes the study of any IT implementation process in three important ways. First, the implementation of a computer-based information system is conceived as a purposeful endeavour that involves movement through different states toward attaining a specific goal or desired end state. Second there are many possible paths that could have been adopted in order to fulfill a specific end goal. Third, human actions were viewed as based on the actors' perceptions of how likely it is that a particular action will move the process closer toward goal achievement. In sum, by adopting a teleological view, a theory of IT implementation cannot specify what trajectory implementation will follow. At best, we can rely on norms of rationality to prescribe certain paths. Consequently, by adopting a teleological view of IT implementation, we focused our research efforts on understanding how courses of action were selected, developing process explanations related to the movement toward attaining a desired end state, and the accessing the role of human's perception in making progress toward goal achievement.

Step 2: Selecting cases

Selection of cases represents another important aspect of building theory from case studies (Eisenhardt, 1989; Lee, 1989; Yin, 1994). Such research relies on theoretical sampling (i.e. cases are chosen for theoretical, not statistical, reasons). The cases may be chosen to replicate previous cases or extend emergent theory, or they may be chosen to fill theoretical categories and provide examples of polar types (Eisenhardt, 1989). Given the nature of our research, we adopted a literal replication strategy (Yin, 1994) where similar, not contrasting, results were predicted for each case.

As a general rule, the definition of the unit of analysis is related to the way the initial research questions have been defined and the generalizations desired at the project's completion (Yin, 1994). The unit of analysis in our study is therefore the CPR project itself, or more precisely, the series of events and decisions that occurred during each project. Under the literal replication strategy, the number of replications is a matter of discretionary and judgmental choice (Yin, 1994). It depends upon the certainty a researcher wants to have about the multiple-case results. Three distinct organizational units at Jackson Memorial Hospital, a large tertiary care teaching institution located in Miami, agreed to participate in our research project. Consequently, three independent CPR implementation projects became the objects of our research. Appendix II presents the characteristics of our field study sample.

Step 3: Crafting instruments and protocols

Theory-building researchers typically combine multiple data collection methods. The rationale is the same as in hypothesis-testing research; that is, the triangulation made possible by multiple data collection methods provides stronger substantiation of constructs and hypotheses (Eisenhardt, 1989; Yin, 1994). Several researchers (e.g., Kaplan & Duchon, 1988; Lee, 1989) recommend that both quantitative and qualitative data be used in any study if at all possible. Collecting different types of data by different methods from different sources produces a wider scope of coverage, and might result in a fuller picture of the phenomena under study than would have been achieved otherwise (Bonoma, 1985).

We collected both qualitative data and quantitative data in our study. Qualitative data were primarily collected through face-to-face semi-structured interviews. As stressed by Kaplan and Maxwell (1994), the primary goal of interviews is to elicit the respondent's views and experiences in his or her own terms, rather than to collect data that are simply a choice among pre-established response categories. The first step in the research design was to interview individuals who participated in the development of each of the CPR implementation projects along with a small group of user representatives (doctors, nurses, and other clinicians). Each interview started with a brief discussion of the research project, carefully designed to arouse the attention and interest of the interviewee, while not biasing the responses by providing too much information about the conceptual framework. The core of the interviews was semi-structured. Interview guides had been developed and were used during each interview. Basically, each interview guide contained the specific issues to be discussed with the respondent, and questions to be kept in mind during each interview. An excerpt of an interview guide used in our study is presented in Appendix III.

We also encouraged open discussions toward the end of each interview allowing interviewees to ask any questions and add any comments they might want. A total of 95 interviews were conducted over a period of six months. Interviews were conducted for one CPR implementation project before proceeding to another. The average length of each interview was approximately 60 minutes with individuals involved in the CPR implementation project and 30 minutes with user representatives producing a total of 812 pages of transcripts. Appendix IV shows the profile of the interviewees. Each project had been implemented within the last four years, making real time data gathering impossible. Recognizing this limitation, two tactics were adopted in our study to increase construct validity, namely, reconstruction of events using multiple respondents and having key informants review final versions of reports.

Documents and texts also can be valuable sources of qualitative data (Kaplan & Maxwell, 1994; Miles & Huberman, 1994). For instance, in Kaplan's studies of the acceptance and diffusion of medical information systems (1983, 1987, 1988), the author read closely original source documents such as published research papers, popularizations in medical magazines, newsletters and books, and conference reports. In line with Kaplan's work, all documents relevant to the present study, including organizational charts, annual reports, special reports and/or administrative documents, newsletters and other internal publications, user manuals and/or training material, and software vendor's marketing kits were collected and analyzed. In one of the three cases, the researchers read a series of three scientific papers that had recently been published by two of the key actors involved in the implementation process. These papers present the results of a post-audit evaluation effort that took place before, during and after an anticipated four-day system failure. These documents provided precious quantitative information that could be compared with qualitative responses of the interviewees in regard to the value of the electronic charting system over the handwritten method.

Finally, whenever possible, observation completed the qualitative assessment. Observation in qualitative studies produces detailed descriptive accounts of what was going on. Such observation often is crucial to the assessment of a system. For example, Kaplan and Duchon (1988) went to the laboratories to observe what technologists actually did, rather than simply depend on verbal reports or job descriptions. In our own study, observation took place during several training sessions and meetings involving CPR implementation project team members, user representatives and external parties. Direct observation of a few clinicians using the different computer-based information systems was also possible in all three projects. Detailed notes were taken during all observations in order to capture the researchers' impressions and insights.

The second and last step in the research design was to gather quantitative data. As stressed by Eisenhardt (1989), quantitative data "can keep researchers from being carried away by vivid, but false, impressions in qualitative data, and it can bolster findings when it corroborates those findings from qualitative evidence" (p. 538). Questionnaire items are often developed after the researcher has analyzed a series of interviews, observations, and documents (Kaplan & Duchon, 1988). This strategy reflects a fundamental difference between case studies and alternative methods (e.g., survey, laboratory experiment, field studies). In the former, the researcher may have less a priori knowledge of what the variables of interest will be and how they will be measured (Eisenhardt, 1989).

In our study, qualitative data were used primarily to develop or suggest theoretical arguments which could then be strengthened (or weakened) by quantitative support. Survey instruments were developed to collect data that would either confirm or refute our interpretation of the data. Respondents were the identified key informants, namely, the individuals actively involved in the three CPR implementation projects who had been interviewed earlier. To ensure that the responses were valid, the surveys were administered toward the end of data collection, although complete analysis of the interview data had not been completed. The survey contained questions that were based on the preliminary data analysis and therefore had not been discussed during the interviews. Examples of survey questions from case #3 are presented in Appendix V.

In sum, the gathering of both quantitative and qualitative data from multiple sources helped to demonstrate the extent of congruity and consistency between the researchers' and key informants' evaluations and to triangulate over given facts (Patton, 1999). However, because the analysis of the interview data could not be completed prior to the administration of the surveys, some important and relevant constructs were not captured in the survey instruments developed for each case. As recommended by Leonard-Barton (1990), sufficient time should be allowed for analysis between waves of data collection in order to eliminate this problem.

Step 4: Early steps in data analysis

A striking feature of research to build theory from case studies is the frequent overlap of data analysis with data collection (Eisenhardt, 1989). The analytical techniques adopted in the first stage of data analysis in our own research are presented below. Note that these techniques were used to help us identify themes, develop categories, and explore similarities and differences in the data, and relationships among them.

First, field notes were an important means of accomplishing this overlap in our study. As described by Van Maanen (1988), field notes are an ongoing stream-of-consciousness commentary about what is happening in the research. By reviewing our field notes frequently, important issues or conflicting answers provided by different individuals were identified immediately. Selected key informants were interviewed again to clear up any questions and to provide any additional information that was missing. The field notes also were useful in revising the interview guides as the study progressed. Second, once an interview was transcribed, reflective remarks were directly entered into the transcripts within brackets. These remarks were ways of getting ideas down on paper and of using writing as a way to facilitate reflection and analytic insight. They were a way to convert the researcher's perceptions and thoughts into a visible form that allows reflection (Miles & Huberman, 1994; Strauss & Corbin, 1990). In short, reflective remarks helped us start thinking, making deeper and more general sense of what was happening, and explaining things in a conceptually coherent way. Examples of reflective remarks are presented in Appendix VI. Finally, a document summary form was created for each document collected and then filled out in the database. This form put the document in context, explained its significance, and gave a brief content summary (Miles & Huberman, 1994). An example of a document summary form is presented in Appendix VII.

In sum, overlapping data analysis with data collection not only gives the researcher a head start in analysis but, more importantly, allows researchers to take advantage of flexible data collection. Indeed, a key feature of theory-building case research is the freedom to make adjustments during the data collection process. In our study, adjustments included adding questions to interview guides, reviewing more data sources, observing meetings when the opportunity arose to do so, and interviewing previously unknown individuals who were identified during the study as important actors in the three CPR implementation projects.

Step 5: Analyzing data (within-case and cross-case)

Analyzing data is the heart of building theory from case studies, but it is both the most difficult and the least codified part of the process (Eisenhardt, 1989; Miles & Huberman, 1994). Qualitative studies tend to produce large amounts of data that are not readily amenable to mechanical manipulation, analysis, and data reduction (Yin, 1994). Therefore, the basic goal of qualitative data analysis is understanding, i.e., the search for coherence and order (Kaplan & Maxwell, 1994). Inspired by the work of Eisenhardt (1989), our data analysis included two aspects, namely, "Within-Case Analysis" and "Cross-Case Analysis." The analytical techniques adopted during each of these two phases are briefly examined below.

Analyzing within-case data

Within-case analysis typically involves detailed write-ups for each case. These write-ups are often simply pure descriptions, but they are central to the generation of insight because they help researchers to cope early in the analysis process with the often enormous volume of data (Eisenhardt, 1989). However, there is no standard format for such analysis. The procedure followed to analyze each of the CPR implementation projects in our study is summarized in Table 2.

Table 2. Within-Case Analysis Procedure

Step 1: Development of a case study data base
1.1
Gather reflective remarks and observation notes
1.2
Codify and extract data from the transcripts using a validated coding scheme
1.3
Group extracted segments under categories (codes and pattern codes)
1.4
Perform descriptive statistical analyses on quantitative data
Step 2: Development of a logical chain of evidence
2.1
Evaluate the contextual conditions surrounding the implementation project
a) perform a qualitative assessment
b) perform a quantitative assessment
c) verify consistency between qualitative and quantitative assessments
d) explain any significant inconsistency
2.2
Evaluate the extent of implementation success
a) perform a qualitative assessment
b) perform a quantitative assessment
c) verify consistency between qualitative and quantitative assessments
d) explain any significant inconsistency
2.3
Establish a logical chain of evidence between implementation context and success
a) identify the challenges
b) provide the story that explains the extent to which each challenge was overcome
c) build a summary table

As recommended by Yin (1994), we first developed a case study database for each CPR implementation project. Each data base organized the data collected in the case and contained the following elements: (1) raw material (including interview transcripts, researcher's field notes, documents collected during data collection, and survey material); (2) coded data; (3) coding scheme; (4) memos and other analytic material; (5) data displays and matrices (explained below); (6) document summary forms; and (7) general chronological log of data collection.

Coding in qualitative research involves segmenting the data into units (Hammersley & Atkinson, 1983) and rearranging them into categories that facilitate insight, comparison, and the development of theory (Strauss & Corbin, 1990). Codes serve as retrieval and organizing devices that allow the rapid retrieval and clustering of all the segments related to a particular question, concept, or theme. To be consistent with our conceptual framework (Appendix I), the coding scheme developed in our study was divided into three broad categories: (1) contextual conditions, (2) implementation tactics, and (3) implementation success criteria. Appendix VIII shows an excerpt of the coding scheme developed in our study.

Specific rules had to be established to ensure the reliability of the coding scheme and the overall quality of the coding process. First, an initial list of codes was developed based on our conceptual framework. The original list was then used to codify and extract the data from the transcripts associated with case one. As a result of this process, we found the need to add a few codes. Once all transcripts associated with the first project were codified, two coders were selected to determine interrater reliability. After a 10-minute initial briefing by the researchers, each coder was instructed to read coding instructions to become acquainted with the coding scheme. Each coder was asked to assign codes to a series of segments representing contextual conditions, implementation tactics and implementation success criteria. The selected segments were randomly selected from all the segments included in the same category. It is worth noticing that the segments used as examples in the instructions were not selected for the validation process. Once each coder completed the task, the researchers' original coding was supplied, and each coder was instructed to discuss any differences with the researchers. On a pairwise basis, the coders' responses and the researchers' codes were compared. As a measure of interrater agreement, Kappa coefficients ranging between 0.78 and 0.82 were obtained, indicating that the strength of agreement among the three raters was substantial (Landis & Koch, 1977).

How data are analyzed and interpreted represents another key question in positivist case study research. Linking data to propositions can be done in a number of ways, but none has become as precisely defined as the assignment of subjects and treatments conditions in laboratory experiments (how hypothesis and data are connected in psychology). Yin (1994) suggests that every case investigation should have a general analytic strategy, so as to guide the decision regarding what will be analyzed and for what reason. He presents three possible analytic strategies: pattern-matching, explanation-building, and time-series analysis. In order to understand the how and why associated with each CPR implementation project and hence to provide answers to our research questions, we adopted an explanation-building strategy. This strategy is to develop a case description, which would be a framework for organizing the case study. Precisely, a logical chain of evidence was built in several steps. The first task was to identify the challenges encountered during the implementation process. Challenges were identified through an in-depth analysis of the contextual conditions surrounding the implementation project. In turn, for each challenge we described the tactics adopted to cope with the encountered problems, anticipated or not. The extent to which each challenge was overcome was explained by (1) providing evidence of the effectiveness of each coping tactic, (2) identifying and explaining how certain contextual conditions enhanced the effectiveness of coping tactics and (3) explaining how other conditions prevented the adoption of tactics by acting as compensatory mechanisms. As recommended by Yin (1994), each chain of evidence was established by having sufficient citations in the report to the relevant portions of the case study database and developing a case study protocol which clearly indicates the links between the content of the protocol and the initial research questions.

A case study protocol contains more than the interview or survey instruments, it should also contain procedures and general rules that should be followed in using the instruments. It is to be created prior to the data collection phase. It is essential in a multiple-case study, and desirable in a single-case study. Precisely, a typical case protocol should have the following components: (1) An overview of the case study project (objectives, issues, topics being investigated); (2) field procedures (credentials and access to sites, sources of information); (3) interview guides and/or survey instruments; and (4) a guide for case study report (outline, format for the narrative).

The adoption of displays such as matrices, flowcharts, and conceptual maps was also useful in several ways during data analysis. For one thing, they made ideas visible and permanent. For instance, besides indicating who has formal authority over whom and what the role names are among actors, context charts (Miles & Huberman, 1994) were useful in telling us about the quality of the working relationships between actors (or groups of actors) involved in each CPR implementation project. Importantly, these charts showed who the key actors were as well as the role played by every individual. Figures and charts also served two other key functions, namely, data reduction and presentation of data that allows it to be grasped as a whole (Miles & Huberman, 1994). For instance, checklist matrices were used to synthesize the overall evaluation (qualitative and quantitative) of the implementation context and the extent of implementation success. A short glance at these tables allowed us to clearly identify the challenges that were encountered over the course of each project and the extent of project success. Importantly, comparison of matrices showing qualitative and quantitative evidence revealed a large extent of congruity and consistency between the researchers' and the key informants' evaluations. In sum, the use of displays allowed us to draw and verify valid conclusions because each display was "arranged coherently to permit careful comparisons, detection of differences, noting of patterns and themes, seeing trends..." (Miles & Huberman, 1994, p. 92). An example of checklist matrix developed in our study is presented in Appendix IX.

Searching for cross-case patterns

As stressed earlier, the ultimate intent of our research was to gradually build a new theory of CPR implementation. The method of generalization adopted here is "analytic generalization," in which previously developed theory is used as a template with which to compare the empirical results of the case study. Under such logic, when two or more cases are shown to support the same theory, replication may be claimed (Yin, 1994).

This iterative process started with the development and presentation of an initial set of theoretical propositions based on evidence from the first CPR implementation project and the theoretical assumptions associated with the teleological process meta-theory. The initial propositions then became a vehicle for generalizing to the other two projects. As a second step, the emergent propositions from the first project were systematically compared with evidence from the second project. The theoretical propositions were either supported by the evidence, revised, or not supported for lack of sufficient evidence. As a third and final step, the process was repeated when refined theoretical propositions were systematically compared with evidence from the third project. The central idea was to iterate toward a theory that fits the data, where projects which supported the emergent theory enhance confidence in its validity, while projects which did not support the theory often provide an opportunity to refine and extend the theoretical model (Eisenhardt, 1989). Step 6 describes how the series of propositions were derived.

Step 6: Shaping research propositions

The next step of this highly iterative process is to compare systematically the emerging theory with the evidence from each project in order to assess how well or poorly it fits with the data. The central idea is that researchers constantly compare theory and data - iterating toward a theory that closely fits the data (Eisenhardt, 1989).

Based on the teleological process meta-theory, our study proposed a series of four research propositions reflecting the complex and dynamic nature of CPR implementation. Each of these propositions will be presented shortly. Proposition 1 stipulates that the successful implementation of CPR systems represents a purposeful process where change agents socially construct envisioned goals, anticipate challenges (both technical and organizational) ahead, and capitalize on unexpected opportunities. Second, our results clearly indicate that the selection and effectiveness of an implementation strategy depends upon the background, skills, beliefs and motivation of key actors involved in the process. Proposition 3 suggests that key actors' beliefs regarding courses of action that should be adopted are constrained or strongly influenced by the context which surrounds a given project. Through the action of reflective actors, effective implementation strategies are likely to become institutionalized while ineffective ones are likely to be quickly abandoned. Last, our research demonstrates that the process of implementing a CPR system is likely to be characterized by a certain indeterminacy and that every CPR implementation project has a life of its own that cannot be perfectly controlled or predicted. Overall, these four propositions define a preliminary set of laws of interaction that characterizes the dynamic nature of the CPR implementation process. As presently constituted, these propositions are at least one step short of theory formation. At minimum, they are empirical generalizations, that is, they summarize observed uniformities of relationships between predictors and outcomes. At best, they suggest a rudimentary model of CPR implementation process. We will refer the reader to Paré (2002) for an extensive discussion of these four propositions.

The process of shaping propositions is more judgmental in theory-building research because researchers cannot apply statistical tests. The research team must judge the strength and consistency of relationships within and across cases and also fully display the evidence and procedures when the findings are published, so that readers may apply their own standards. Consequently, qualitative data are particularly useful for understanding why or why not emergent relationships hold. When a relationship is supported, the qualitative data often provide a good understanding of the dynamics underlying the relationship, that is, the why of what is happening. This is crucial in the establishment of internal validity. Consequently, having sufficient citations and quotes in each of the three CPR implementation project reports was an important way of ensuring internal validity in our study. Another way of validating researchers' interpretations is by systematically gathering feedback about one's conclusions from participants in the case being studied (Guba & Lincoln, 1989). Two tactics were therefore adopted to further increase the validity and reliability of our research. First, key informants reviewed the draft of each case study report. Second, a site analysis meeting for each project took place at the end of data analysis. Participants in these meetings were those in a position to reflect on the project's "big picture." Table 3 summarizes the four criteria used to assess the quality of any research design and the case study tactics adopted in our study for dealing with them.

Table 3. Case Study Tactics Adopted For Design Quality

Criterion Description Adopted tactics
Construct validity
  • Establishing correct operational measures for the concepts being studied
  • Multiple sources of evidence
  • Review of case study report by key informants
  • Mix of qualitative and quantitative methods
Internal validity
  • Establishing a causal relationship, whereby certain conditions are shown to other conditions, as distinguished from creating spurious relationships
  • Explanation-building strategy with logical chain of evidence
  • Review of case study report by key informants
  • Site analysis meeting
  • Sufficient citations in the case report
  • Checklist matrices
  • Tying propositions to existing literature
External validity
  • Establishing the domain within which a study's findings can be generalized
  • Analytic generalization
  • Tying propositions to existing literature
Reliability
  • Demonstrating that the operations of a study can be repeated, with the same results
  • Validation of coding scheme
  • Case study data base
  • Case study protocol

Step 7: Enfolding literature

An essential feature of theory building is comparison of the emergent concepts, theory, or hypotheses with the extant literature (Eisenhardt, 1989). This involves asking what is similar to, what does it contradict, and why. In pursuit of this objective, for each proposition, we indicated the extent to which it was supported by previous research and the extent to which we have added some new perspective or new idea when thinking about the management of the CPR implementation process. For instance, proposition 2 posits that the selection and effectiveness of implementation tactics and strategies depend on the background, skills, and beliefs of key people involved in the implementation effort. While previous research has acknowledged the importance of having individuals with specific characteristics involved in an implementation effort (Fossum, 1986; Hunsucker & Loos, 1989; Joshi, 1990; Schultz et al., 1987), our study has shown how these characteristics affect both the selection and effectiveness of various implementation tactics employed. This implies that no one set of normative implementation tactics can or should be applied to all projects. Rather, project leaders should ensure that implementation tactics are adopted that complement the experiences, skills, and beliefs brought by each player in the implementation effort.

As another example, proposition one stipulates that successful implementation requires identifying and addressing implementation challenges. This evidence adds support to previous research that has found that most unsuccessful IT implementation projects are the result of poor management, not technical problems (Ewusi Mensah & Przansnyski, 1991; Lyytinen & Hirschheim, 1987). While much of the implementation research has equated good management with knowing what to do (Leonard-Barton & Deschamps, 1988; Lorenzi & Riley, 1995; Schultz et al., 1987), we suggest that good management must also focus on what to look for and think about. In short, we found that tying the emergent theoretical propositions to existing literature enhances internal validity and generalizability of theory building from case study research as suggested by Eisenhardt (1989).

Step 8: Reaching closure

An important issue in reaching closure is when to stop adding cases. Ideally, researchers should stop adding cases when theoretical saturation is reached (Eisenhardt, 1989). Theoretical saturation is the point at which incremental learning is minimal because the researchers are observing phenomena seen before (Glaser & Strauss, 1967). In practice, however, theoretical saturation often combines with pragmatic considerations to dictate when case collection ends. In fact, it is not uncommon for researchers to plan the number of cases in advance.

For pragmatic reasons of time, money, and opportunity, this study involved only three CPR implementation projects. Clearly theoretical saturation cannot be attained with such a small number of cases. Therefore, additional case studies of CPR implementation projects must be conducted to increase the validity and reliability of the theoretical propositions developed in our research. The theoretical propositions would benefit not only from being tested using other clinical applications but also to be tested against recent project failures where projects were abandoned at some point or where systems were not used at all.

As a final remark, the presentation of implications for practice and recommendations for future research constitutes another indication of the overall quality of any scientific approach, including case research. Table 4 highlights those implications and suggestions derived from our exploratory study.

Table 4. Lessons Learned and Suggestions For Future Research

PropositionKey implications for practice Avenues for future research
1
  • Conduct an early assessment of the project context
  • Derive a substantive plan describing potential key issues and challenges
  • Share project vision with all parties
  • Consider unique contextual factors and organization' or unit's history with other projects
  • Research that focuses on understanding variations on risks and challenges of CPR implementation
2
  • Assess potential actors' strengths and weaknesses
  • Compare potential actors' profiles to required competency profiles
  • Aim for complementarity of skills and goals
  • Research on preferences for certain actions or alternatives
  • Research on necessary skills or competencies (and their fit) with implementation situations
3
  • Adopt a project-contingent view
  • Engage in regular reflection and re-assessment of changing context and circumstances
  • Allow flexible evolution and revision to implementation plan
  • Research that integrates implementation situations and context with successful tactics
  • Research on the duality of structure and other influences on CPR project outcomes
  • Research on the influence of organizational learning on future implementation successes
4
  • Assess outcomes from a perspective of indeterminacy - open to evolving criteria for success and duality of outcomes
  • Key actors must be opportunistic in exploiting unexpected and/or problematic events
  • Longitudinal research that follows multiple projects within a given organization over time.

Conclusion

It is clear that current qualitative research standards have evolved and are more demanding for information systems researchers than they were in the early 1980s. Today, qualitative analysis needs to be well documented as a process mainly to help us learn. Purposes of auditing aside, we need to understand more clearly what is going on when we analyze data, to reflect, refine our methods, and to make them more generally useable by others. As this article has shown, case study is a reliable methodology when executed with due care. The key contributions from this article are twofold. First, we offer a rigorous, step-by-step methodology to develop theories and show how a variety of concepts, techniques and tools (e.g., Miles & Huberman, 1994; Patton, 1999; Strauss & Corbin, 1990; Yin, 1994) can be used in applying Eisenhardt's approach. Second, we illustrate how this methodology can be applied in order to contribute to the discovery of a number of new perspectives and empirical insights in the information systems and medical informatics fields.

Nevertheless the contributions of this paper, we still need to consider the overall demand of the approach proposed here on IS researchers. For instance, process research usually results in the collection of large amounts of data vulnerable to subjective interpretation and surpassing human ability to compile. Because of the demands and problems encountered during qualitative research, researchers must have a great interest in and dedication to the object of research (Barley, 1990; Leonard-Barton, 1990). While it is important to gain the trust and confidence of organizational members, it is also important, as positivist researchers, to remain sufficiently detached so as to be objective. Importantly, researchers should not underestimate the time and effort required to conduct these kinds of studies. Experience from the present study shows that investigators must often be willing to spend lunches, evenings, and weekends collecting data at the site. Lastly, qualitative analysis is both the most difficult and the least codified part of the process. We found that displaying data using a variety of matrices, flowcharts, and conceptual maps can be a powerful means for discovering connections between coded data and, ultimately, drawing valid conclusions. Displays facilitate data reduction and help make ideas visible and permanent (Miles & Huberman, 1994). Despite these constraints, qualitative studies remain, we believe, the best approach available for studying complex phenomena like system design and implementation as well as several emerging IT management issues. The reward clearly appears to be a deeper and broader understanding of such phenomena and the ability to contribute significantly to cumulative knowledge in our field.

In conclusion, qualitative studies are gradually becoming more accepted in the information systems field; but meanwhile researchers will have to work harder, be more creative, and come up with new and more robust methodological tools to have their work recognized and accepted in a community that tends to be sceptical of qualitative studies. Applying a well-defined methodology along the lines described in this paper will help to position qualitative studies more in the mainstream of health services management research. There are numerous phenomena in addition to technology implementation and adoption whose underlying dynamics are unknown and thus are good candidates to study using a case study research strategy in information systems research. Examples of common topics of case studies include the impacts of IT on organizations, individuals and societies, the reinvention of information systems by individuals, and the diffusion of technological innovations. We hope that we have provided a detailed guide for carrying out such a research strategy.

References

      Anderson, J. G., & Aydin C. E. (1994). Overview: Theoretical perspectives and methodologies for the evaluation of health care information systems. In J. G. Anderson, C. E. Aydin, & S. J. Jay (Eds.), Evaluating health care information systems, methods and applications (pp. 5-29). Thousand Oaks, CA: Sage.

      Barley, S. R. (1990). The alignment of technology and structure through roles and networks. Administrative Science Quarterly, 35, 61-103.

      Benbasat, I., Golstein, D. D., & Mead, M. (1987), The case research strategy in studies of information systems. Management Information Systems Quarterly, 11(3), 369-386.

      Benbasat, I., & Weber, R. (1996). Rethinking diversity in information systems research. Information Systems Research, 7(4), 389-399.

      Boehm, B. W. (1991). Software risk management: Principles and practices. IEEE Software, 8(1), 3241.

      Bonoma, T.V. (1985). Case research in marketing: Opportunities, problems, and a process. Journal of Marketing Research, 22, 199-208.

      Chan, Y. E., Huff, S. L., Copeland, D. G., & Barclay, D. W. (1997). Business strategy, information systems strategy and strategic alignment. Information Systems Research, 8, 125-150.

      Charette, R. N. (1989). Software engineering risk analysis and management. New York, NY: Intertext.

      Charles, F., Southon, G., Sauer, C., & Noel Grant, C. (1997). Information technology in complex health services. Journal of the American Medical Informatics Association, 4, 112-124.

      Cho, S., Clement, H., Gillis, R., Matthews, S., Salois-Swallow, D., & Tamaki, C. (1992). Framework for planning nursing information systems. Edmonton, Canada: Healthcare Computing & Communications Canada, Inc.

      Dambro, M. R., Weiss, B. D., McClure C. L., & Vuturo A. F. (1988). An unsuccessful experience with computerized medical records in an academic medical center. Journal of Medical Education, 63, 617-623.

      Dash, J. (2000). Health care organizations plan to boost IT spending in 2001. Computerworld, 34(25), 14.

      Edwards N. H. (1999). Technology and healthcare into the future. Journal of Informatics in Primary Care, September, 10-13.

      Edwards, J., & Purchase, E. (2000). Trends and directions in healthcare IT: An international perspective. Strategic Analysis Report, Gartner Group, October 5.

      Eisenhardt, K. M. (1989). Building theories from case study research. Academy of Management Review, 14(4), 532-550.

      Ewusi-Mensah, K., & Przanyski, Z. H. (1991). On information systems project abandonment: An exploratory study of organizational practices. MIS Quarterly, 15(1), 67-88.

      Fossum, B. (1986). A normative model for CIM implementation. Ph.D. Thesis, University of Texas at Austin.

      Fox, S., & Rainie, L. (2000). The online health care revolution: How the web helps Americans take better care of themselves. Washington, DC: Institute of Medicine.

      Garets, D., & Frey, N. (2001). Gartner's vision for healthcare: The next 10 years. Strategic Analysis Report, Gartner Group, 29 January.

      Glaser, B., & Strauss, A. (1967). The discovery of grounded theory: Strategies of qualitative research. London, UK: Wiedenfeld and Nicholson.

      Goddard, B. L. (2000). Termination of a contract to implement an enterprise electronic medical record system. Journal of the American Medical Informatics Association, 7, 564-568.

      Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, CA: Sage.

      Gurasekaran, S. (2000). Heatlh care industry IT forecasts update. Gartner Market Analysis Report, (May 8).

      Hammersley, M., & Atkinson, P.(1983). Ethnography: Principles in practice. London, UK: Tavistock.

      Hatcher, M. (1998). Impact of information systems on acute care hospitals: Results from a survey in the United States. Journal of Medical Systems, 22(6), 379-387.

      Hunsucker, J. L., & Loos, D. (1989). Transition management, an analysis of strategic considerations for effective implementation. Engineering Management International, 5, 167-178.

      Kohn, L. T., Corrigan, J. M., & Donaldson, M. S. (Eds.). (1999). To err is human: Building a safer health system. Washington, DC: National Academy Press.

      Jick, T. (1979). Mixing qualitative and quantitative methods: Triangulation in action. Administrative Science Quarterly, 24, 602-611.

      Joshi, K. (1990). Reorganization of the work system for successful information systems implementation: A case study. Information and Management, 19, 271-284.

      Kaplan, B. (1983). The computer as Rorschach: Implications for management and user acceptance. In R. E. Dayhoff (Ed.), Proceedings of the 7th annual symposium on computer applications in medical care (pp. 664-667). Silver Spring, MD: IEEE Computer Society Press.

      Kaplan, B. (1987). Initial impact of a clinical laboratory computer system: Themes common to expectations and actualities. Journal of Medical Systems, 11, 137-147.

      Kaplan, B. (1988). Development and acceptance of medical information systems: An historical overview. Journal of Health and Human Resources Administration, 11, 9-29.

      Kaplan, B., & Duchon, D. (1988). Combining qualitative and quantitative methods in information systems research: A case study. MIS Quarterly, 12(4), 571-586.

      Kaplan, B., & Maxwell, J. A. (1994). Qualitative research methods for evaluating computer information systems. In J. G. Anderson, C. E. Aydin, & S. J. Jay (Eds.), Evaluating health care information systems, methods and applications (pp. 45-68). Thousand Oaks, CA: Sage.

      Klein, H. K., & Myers, M. D. (1999). A set of principles for conducting and evaluating interpretive field studies in information systems. MIS Quarterly, Special Issue on Intensive Research, 23(1), 67-93.

      Kocken, M. (1986). Are MIS frameworks premature? Journal of Management Information Systems, 2(3), 92-100.

      Landis, J. R., & Kock, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174.

      Lawler, F., Cacy, J. R., Vicianni, N., Hamm, R. M., & Cobb, S. W. (1996). Implementation and termination of a computerized medical information system. Journal of Family Practice, 42, 233-236.

      Lee, A. S. (1989). A scientific methodology for MIS case studies. MIS Quarterly, 13(1), 33-52.

      Leonard-Barton, D. (1990). A dual methodology for case studies: Synergistic use of a longitudinal single site with replicated multiple sites. Organization Science, 1(3), 248-266.

      Leonard-Barton, D., & Deschamps, I. (1988). Managerial influence in the implementation of new technology. Management Science, 34(10), 1252-1265.

      Linder, J. C. (1992). outcomes measurement in hospitals: Can the system change the organization? Hospital and Health Services Administration, 37(2), 143-166.

      Lorenzi, N. M., & Riley, R. T. (1995). Organizational aspects of health informatics: Managing technological change. New-York, NY: Springer Verlag.

      Lorenzi, N. M., & Riley, R. T. (2000). Managing change: An overview. Journal of the American Medical Informatics Association, 7, 116-124.

      Lorenzi, N. M., Riley, R. T., Blyth, A. J., Southon, G., & Dixon, B. J. (1998). People and organizational aspects of medical informatics. Proceedings of MedInfo, 9(Part 2), 197-200.

      Lyytinen, K., & Hirschheim, R. (1987). Information systems failures, a survey and classification of the empirical literature. Oxford Surveys in Information Technology, 4, 257-309.

      Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. Beverly Hills, CA: Sage.

      Mohr, L. B. (1982). Explaining organizational behavior. San Francisco, CA: Jossey-Bass.

      Nelson, K. M., Nadkarni, S., Narayanan, V. K., & Ghods, M. (2000). Understanding software operations support expertise: A revealed causal mapping approach. MIS Quarterly, 24(3), 475-508.

      Orlikowski, W. J., & Baroudi, J. J. (1991). Studying information technology in organizations: Research approaches and assumptions. Information Systems Research, 2(1), 1-28.

      Paré, G. (2002). Implementing clinical information systems: A multiple-case study within a US hospital. Health Services Management Research, 15, 1-22.

      Paré, G., & Elam, J. J. (1998). Introducing information technology in the clinical setting: Lessons learned in a trauma centre. International Journal of Technology Assessment in Health Care, 14(2), 1-13.

      Paré, G., Elam, J. J., & Lima, C. (1996). Implementation of an electronic medical records system: How can health care managers ensure its success? Journal of Information Technology Management, 7(3), 1-16.

      Paré, G., Elam, J. J., & Ward, C. G. (1997). Implementation of a patient charting system: Encountered challenges and adopted tactics in a burn centre. Journal of Medical Systems, 21(1), 49-66.

      Paré, G., & Sicotte, C. (2001). Information technology sophistication in health care: An instrument validation process among Canadian hospitals. International Journal of Medical Informatics, 63(3), 205-223

      Patton, M. Q. (1999). Enhancing the quality and credibility of qualitative analysis. Health Services Research, 34(5), Part II, 1189-1208.

      Potter, K. (2000). 2000 IT spending and staffing survey results. Strategic Analysis Report, Gartner Group, 2 October.

      Raymond, L., Paré, G., & Bergeron, F. (1995). Matching information technology and organizational structure: An empirical study with implications for performance. European Journal of Information Systems, 4, 3-16.

      Romm, C. T., & Pliskin, S. (1997). Playing politics with e-mail: A longitudinal conflict-based analysis. In A. S. Lee, J. Liebenau, & J. I. DeGross (Eds.), Information systems and qualitative research (pp. 362-388). London: Chapman & Hall.

      Rundall, T. G., Devers, K. J., & Sofaer, S. (1999). Overview of the special supplemental issue. Health Services Research, 34(5), Part II, 1091-1099.

      Schultz, R. L., Slevin, D. P., & Pinto, J. K. (1987). Strategy and tactics in a process model of project implementation. Interfaces, 17(3), 34-46.

      Sinclair, G., & Cook, J. (2000). Taking the risk out of IT projects. Chartered Accountants Journal of New Zealand, 79(6), 26-27.

      Strauss, A. L., & Corbin, J. (1990). Basics of qualitative research: Grounded theory procedures and techniques. Newbury Park, CA: Sage.

      Swanson, E. B. (1998). Information system implementation: Bridging the gap between design and design and utilization. Columbus, OH: Irwin.

      Trauth, E. M., & Jessup, L. M. (2000). Understanding computer-mediated discussions: Positivist and interpretive analyses of group support system use. MIS Quarterly, 24(1), 43-79.

      Van de Ven, A.H., & Poole, M.S. (1995). Explaining development and change in organizations. Academy of Management Review, 20(3), 510-540.

      Van Maanen, J. (1988). Tales of the field: On writing ethnography. Chicago, IL: University of Chicago Press.

      Wixon, D., & Ramey, J. (Eds.) (1996). Field methods casebook for software design. New York, NY: John Wiley & Sons.

      Yin, R. K. (1994). case study research, Design and methods Beverly Hills, CA: Sage.

      Yin, R. K. (1999). Enhancing the quality of case studies in health services research. Health Services Research, 34(5), Part II, 1209-1224.

Appendix I: Conceptual Framework

Appendix II: Characteristics of field study sample


Case One Case Two Case Three
SiteJMH - Hospital-wideJMH - Trauma CenterJMH - Burn Center
CPR projectMedical records -
Electronic signature
Nursing flowsheet SystemElectronic patient
Medical chart
Project team compositionJMH's MIS division staffTrauma Center's Medical Computer Systems Laboratory staffExternal IT consulting firm & internal system administrator
Primary users (number)Attending physicians (1,000 approx.)Nurses (100 approx.)Nurses and physicians (20 approx.)
Software development / acquisition approachDeveloped in-house Developed in-housePurchased off-the-shelf
Technological infrastructureMainframeClient-server (LAN)Client-server (LAN)
Project championDirector of Medical RecordsDirector of clinical ApplicationsMedical Director of Burn Center
Ideal implementation situationNoNoNo
Encountered implementation challenges
  • Get resources allocated to the project
  • Satisfy agencies' requirements
  • Design a technically sound system
  • Obtain support of chiefs of clinical services
  • Have all MDs comply and use the system
  • Solve technical problems
  • Get the project under way
  • Establish credibility as project leader
  • Design technically sound systems
  • Have all nurses comply and use the system
  • Get the necessary funds to initiate the project
  • Have the staff comply and use the system
  • Add technical expertise to the local team
  • Maintain and customize the system
  • Search for and buy a second-generation CPR
  • Proceed to system conversion
Project SuccessYesPartialYes

Appendix III: Profile of the interviewees

Case SystemSiteInterview with
team members
Interviews with
users
1Medical records -
Electronic signature
Hospital-wide Dir. of Medical Records
VP Medical Affairs
HIS coordinator
Education coordinators
Systems analysts
Programmers
Number of interviews: 26
Attending physicians
Nurse liaisons
Medical records staff



Number of interviews: 13
2Nursing
flowsheet
system
Trauma CenterDir. Clinical Applications
Ass. Dir. Soft. Tech.
Clinical educators
Head nurses
Programmers
Number of interviews: 19
Registered nurses




Number of interviews: 31
3 Electronic
patient
chart
Burn Center Medical Director
Associate Director
Head nurse
Clinical educator
System administrator
Number of interviews: 12
Registered nurses
Physicians
Resident staff
Physical therapists
Dietician
Number of interviews: 25

Appendix IV: Excerpt of an interview guide

Case # : ___       Interview # : ____       Date : / /___       Time : ______ AM PM

Name of interviewee : __________________ Phone: ______________

Title: ____________________________________________________

INTRODUCTION

Purpose of the meeting: Learn more about the context in which the implementation of the computerized charting system took place as well as the tactics adopted to ensure system success.

IMPLEMENTATION CONTEXT

Previous experience with IT:

Organizational Resources

IMPLEMENTATION TACTICS

External Integration

Championing

IMPLEMENTATION SUCCESS

AFTERTHOUGHTS AND COMMENTS: (if any)

Appendix V: Examples of survey questions (Case #3)

Please indicate your agreement or disagreement with each of the following statements by circling the number that corresponds most closely to your desired response.



strongly
disagree

uncertain
strongly
agree
1. The national movement toward the electronic chart has represented a major driving force for the computerization of the patient chart at the Burn Center. 1 2 3 4 5
2. Innovative initiatives are heavily encouraged and are common practices at the Burn Center. 1 2 3 4 5
3. Burn Center's employees all work toward the achievement of a common goal, that is, improving the quality of patient care. 1 2 3 4 5
4. The innovative climate which characterizes the Burn Center has represented a critical factor in the success of both the <first> and the <second> systems. 1 2 3 4 5
5. Generally speaking, the transition from a paper-based environment to a computer-based environment (<first> system) was difficult due to resistance to change. 1 2 3 4 5
6. The transition from a paper-based environment to a computer-based environment required that clinicians' tasks be modified only slightly. 1 2 3 4 5
7. In general, the introduction of the <first> system has led to minor structural and managerial changes within the Burn Center. 1 2 3 4 5
8. Most clinicians were highly computer literate prior to the implementation of the <first> system. 1 2 3 4 5
9. Most clinicians had negative attitudes toward computers prior to the implementation of the <first> system. 1 2 3 4 5
10. After an initial "chaotic" period, most clinicians gradually adapted to the <first> system and developed positive attitudes toward it. 1 2 3 4 5

Appendix VI: Examples of reflective remarks in transcripts

Excerpts of transcripts Reflective remarks

Case #1
Trainer: "We always provide physicians hands-on training, no overheads [theoretical training] because we want to know who understands and who doesn't. In class, you can easily see who is slow and who needs more help and support."


Clear evidence that the trainers care about user learning and support.]

Case #1
Project champion: "The residents always use the system because it helps them a lot. The attendings, on the other hand, get the residents to do everything. If they need a lab result they have their resident to do it. When the system came online it was the attendings' responsibility to get the lab results using their confidential access codes. Of course, we got some resistance at the beginning. Some gave their code to their residents."


[Effects of hierarchy on system acceptance and use.]

Case #2
Project sponsor: "I think what we need to do is to convince nurses of the utility of the system by developing initially those processes that are important to them, nursing assessments for instance. We've done that. The important aspects of nursing work are computerized first to ease the transition from paper-work to computers."


[Importance of starting the development and implementation process with meaningful applications.]

Case #2
Project champion: "The people in the ICU didn't adapt easily to the changes compared to the nurses in the resus unit. In the ICU, nurses know how their day will go in advance, everything is a routine and the computer was a disruption to them. While in the resus, work is chaotic, constant disruption. You never know what's going to happen in the next five minutes. So, for them, the introduction of the system had not the same meaning and they did not offer resistance."


[Impact of work environment and structural conditions on resistance to change.]

Case #3
Project champion: "There are very few clinicians who are interested in having the full medical record online. My goal is to create an entire electronic chart. I am just proud of the whole document and having it online… I don't think there are that many people who really want to create a whole record. That's what I find frustrating. Buying a software package was the only way to accomplish this quickly."


[Evidence of how the champion's vision about what should be a CPR on the development strategy (acquisition vs. in-house development).]

Case #3
Systems manager: "We have always had a very close relationship with our vendor. It is a bit less close now that there are 41 other sites as opposed to use being their third contract. But still on new applications, our input is always listened to."


[Advantages of being a "Beta" site.]

Appendix VII: Example of a document summary form (Case #1)


Case # :1             Document number: 3
Document name: JMH - 75 years, 1918-1993
Received from: Public Relations


Description of the document: General information about JMH. Type of ownership; Licensed bed capacity; short description; patient services; staff; statistics; revenues; history;

Importance of the document: Provides a lot of relevant information regarding JMH in the same document.

Brief summary of content:
JMH is an accredited, not-for-profit, major tertiary teaching hospital. The facility is the second largest hospital under a single license in the United States. JMH is a designated facility under the jurisdiction of the Public Health Trust of Dade County, Florida.

JMH shares a 67-acre site adjacent to downtown Miami with the University of Miami School of Medicine. Jackson North Maternity Center, a satellite maternity facility, is located in north Dade County. JMH has a total of 1,517 licensed beds, 69 of which are located at the Jackson North Maternity Center.

Comprehensive array of diagnostic and treatment services for medical, surgical, obstetrical and gynecological, pediatric, psychiatric, emergent, ambulatory and rehabilitative patients, as well as specialized intensive and coronary care and neonatal intensive care to both residents of Dade County and patients from throughout the southeastern United States, the Caribbean Basin and Latin America.

JMH employs 11,650 employees, 1,150 of which are physicians. The number of physicians includes approximately 600 University of Miami School of Medicine faculty and approximately 550 residents and housestaff of the hospital. The total number of employees also includes more than 2,200 nurses.

Number of admissions at JMH: 51,407 for adults and pediatric and 10,854 for new borns in 1992. The average length of stay is about 7.3 days and the average daily occupancy rate is about 82.6%. Total number of outpatient visits: 420,000 for the same year. Sources of revenues: $628,969,000 in 1992. Today, JMH constitutes a major medical center in the South Florida region.

How big is JMH? beds: 1,527; admission: 52,298; occupancy: 84.4%; births: 14,407

Affiliation with the University of Miami:

Of the 983 full-time faculty members of the medical school, more than 600 are doctors engaged in clinical practice who are permitted to join the medical staff and can admit patients to JMH.

Appendix VIII: Excerpt of the coding scheme

CONTEXTUAL CONDITIONS
National Movement toward Computerization (EC/NAT_MOV)

A segment which presents evidence of efforts made by national associations, agencies, and hospitals towards computerization.
Organizational Climate (OC/ORG_CLI)
A segment which reflects one's own perception about his/her organization with respect to commitment to technological and/or management initiatives.
Organizational IT Experiences (OC/IT_EXP)
A segment which refers to the memories or experiences that exist in the organization about prior information technology initiatives.
Availability of Organizational Resources (OC/AVA_RES)
A segment which shows the extent to which the organizational resources needed to support the development and implementation of the system were available. Organizational resources do not only include money, but also time, people, hardware, software, and facilities.
Users Skills/Knowledge (WUC/SKILLS)
A segment which describes one's perception of the extent to which users are familiar with the task being automated, are familiar with the computer system, and/or have experience with computers in general.

IMPLEMENTATION TACTICS
External Integration (SD/EXT_INT)

A segment which indicates the adoption of tactic(s) whose objective is to link the project team's work to the users. Examples of external integration tactics include the selection of a user as project leader, the selection of particular users as team members, and the consultation of users on an occasional basis for feedback purposes.
Internal Integration (SD/INT_INT)
A segment which describes the adoption of tactic(s) whose purpose is to ensure that the project team operates as an integrated unit. Well-known internal integration tactics include the selection of experienced IT professionals to be part of the project team; frequent project team meetings; participation of all team members in goal setting; and selection of a high percentage of team members with significant previous work experience and relationships.
Championing (SD/CHAMP)
A segment which illustrates effort(s) made for providing motivation to the project team; generating enthusiasm for the targeted users; providing crucial political support during periods of important decision making; selling the idea to top management; and/or getting senior managers sufficiently interested in the project, etc.
Incremental Approach (UR/INCREM)
A segment which provides evidence that a strategic decision was to introduce the various modules of the computer system in a gradual, step-by-step manner.
Mutual Adaptation (MC/ADAPT)
A segment which portrays how the organization moved ahead with the introduction of the technology, left existing organizational arrangements (e.g., structure) in place, and subsequently attended to organizational changes on a responsive or adaptive basis.

IMPLEMENTATION SUCCESS
System Acceptance and Use (O/ACCEPT_USE)
A segment which expresses one's perception of users' acceptance and/or use of the computerized information system.
Project Progress (P/PROGRESS)
A segment which provides insights as for how the project evolved or progressed over time. Most IT implementation or development processes are usually evaluated in terms of their respect of deadlines and budgets. IT projects are also subject to unavoidable or unexpected problems of all sorts.
Overall Satisfaction with the Implementation Process (P/SATIS)
A segment which makes a suggestion about one's own satisfaction, perception, reflection, and/or evaluation in regard with the overall development and implementation process.

Appendix IX: Example of a checklist matrix (Case #3)

Conditions Qualitative
Assessment
Significance Quantitative
Assessment

Organizational Context

Climate
Past IT Experiences
Financial Resources



Favourable
Neutral
Unfavourable



Teamwork and support
No prior experience
No money available



Favourable [4.8]
Unfavourable [2.9]
Unfavourable [2.7]

Project Context

Presence of key actors
Project complexity
Relation with vendor
Team expertise



Favourable
Favourable
Favourable
Unfavourable



Champion & supporters
Relatively small scope
Good technical support
No technical expertise on site



Neutral [3.3]
Favourable [4.5]
Favourable [4.2]
Unfavourable [1.5]

Work Unit Context

Attitudes toward IT
IT Skills & Knowledge
Expectations



Unfavourable
Unfavourable
Unfavourable



Fears & mixed emotions
Very limited
Confused or unknown



Unfavourable [2.4]
Unfavourable [2.1]
Unfavourable [2.3]

Technological Context

Product capabilities
System usability
Product suitability
System preparedness



Favourable
Favourable
Unfavourable
Unfavourable



Extensive capabilities
User-friendly
Customization needed
Program contains bugs



Favourable [4.7]
Favourable [4.3]
Neutral [3.1]
Unfavourable [2.2]

This matrix summarizes the qualitative and quantitative assessments of the initial contextual conditions surrounding the implementation project. The first column identifies the various conditions found to have had an influence on the extent of implementation success. The second column shows the researchers' judgment (favourable vs. unfavourable) for each contextual condition. The cell entries in the "significance" column include summary phrases or direct quotes that summarize the meaning of each factor for this specific implementation project. The last column shows the results of the quantitative survey (see Appendix III). Qualitative and quantitative evidence clearly reveals that the initial context surrounding the implementation of the CPR system did not match with the long prescribed "ideal" implementation scenario. Indeed, a short glance at this matrix suggests that several implementation challenges were encountered over the course of the project, increasing at once the risk of project failure.

Author Note

+Guy Paré is an associate professor of Management Information Systems at HEC Montreal and a Research Fellow at CIRANO. He received his Ph.D. in Business Administration from Florida International University. His current research interests focus on IT in health care, IT staffing and career issues, IT change agentry, and virtual teams. Journals and conference proceedings where his publications have recently appeared include Communications of the ACM, International Journal of Medical Informatics, Health Services Management Research, Journal of Medical Systems, International Journal of Technology Assessment in Health Care, Proceedings of ICIS, and Proceedings of SIGCPR. He can be contacted at HEC Montreal, Information Technology Department, 3000, Côte-Ste-Catherine, Montreal (Quebec) CANADA H3T 2A7 and his email is:
Guy.Pare@hec.ca.

Article Citation

      Paré, G. (2002, December). Enhancing the rigor of qualitative research: Application of a case methodology to build theories of IT implementation. The Qualitative Report, 7(4). Retrieved [Insert date here], from http://www.nova.edu/ssss/QR/QR7-4/pare.html

Guy Paré
2002 copyright


Return to the top of the paper.

Return to the Table of Contents.