Use of Focus Groups in Survey Item Development
by
Sylvia C. Nassar-McMillan and L. DiAnne Borders
+

The Qualitative Report, Volume 7, Number 1 March, 2002
(http://www.nova.edu/ssss/QR/QR7-1/nassar.html)


Abstract

Focus groups are rapidly gaining popularity as a field research tool. This technique can be particularly effective in survey item development, as illustrated here via development of the Volunteer Work Behaviors Questionnaire. The steps involved in this process, ranging from item generation to finalizing logistics, are outlined. Implications for further research are proposed.

Introduction

Several important steps in developing survey instruments, or questionnaires, are item generation and refinement. While using professional literature is almost always appropriate in selecting items (Jaeger, 1984), a second or supplementary method can include soliciting feedback from individuals in the field (e.g., Loesch & Vacc, 1993). One way this second method has been accomplished is through the use of focus groups (e.g., Fuller, Edwards, Vorakitphokatorn, & Sermsri, 1993; Hughes, 1993; O'Brien, 1993; Wolff, Knodel, & Sittitrai, 1993). Focus groups involve the "explicit use of group interaction to produce data and insights that would be less accessible without the interaction found in a group" (Morgan, 1988, p. 12).

The purpose of this article is to review characteristics and uses of focus groups and to illustrate how they can be applied in developing questionnaire items. The use of focus groups to generate and refine items for the Volunteer Work Behavior Questionnaire (VWBQ) (Nassar-McMillan & Borders, 1999) is described as an example.

Focus Groups

Early uses of focus groups in social science research occurred during the World War II era in assessing public response to wartime propaganda (Greenbaum, 1993; Morgan, 1988). This method was quickly assimilated into marketing research, where it gained widespread use and publicity as a way of soliciting consumer feedback on products and services. Because of a variety of other qualitative methods already commonly used in social science research, including other group interview techniques, focus groups remained underutilized in this realm. In relatively recent years, however, social scientists have rediscovered and begun to re-employ the method, recognizing its unique qualities that had previously gone overlooked (Krueger, 1994; Morgan, 1988).

As Morgan's definition (1988) implied, some of the advantages of focus group interviews versus individual interviews can be likened to those of group counseling versus individual counseling, or, more generally speaking, any group discussions versus individual ones. One obvious advantage is that greater amounts of information can be gathered in shorter and more efficient time spans (Krueger, 1994). Secondly, the group synergy fosters more creativity and therefore provides for a greater range of thought, ideas, and experiences (Vaughn, Schumm, & Sinagub, 1996). The peer validation inherent in such groups also can serve as a catalyst for the expression of these. Finally, researchers are able to observe the interaction between group members, which sometimes provides additional valuable insights regarding a topic or phenomenon (Stewart & Shamdasani, 1990).

Uses of Focus Groups

Researchers often are in need of innovative approaches to gather initial data, especially when little information on a specific topic of interest is known. Focus groups potentially provide such an exploratory approach and may be more effective in certain research processes than more traditional approaches (Greenbaum, 1993; Vaughn, et al., 1996). Focus groups may be used to refine information previously known about a topic or may be designed to elicit new insight and information about a topic by examining it from a new angle.

Focus groups can offer an appropriate medium for work at various steps of the research process, from hypothesis generation to hypothesis testing (Krueger, 1994). More specifically, focus groups can be appropriate for obtaining information, generating research hypotheses, stimulating new ideas or concepts, diagnosing problems with or gathering information about services or programs, providing terminology appropriate for the research, and interpreting experimental results (Stewart & Shamdasani, 1990). Focus groups can supplement quantitative or other qualitative techniques; or they can be utilized as a self-contained technique, resulting in data that are useful in and of itself (Morgan, 1988; Vaughn, et al., 1996). Individual focus groups can be drawn from specific populations; a series of focus groups can serve to compare the groups' reactions to the same concepts. In combining focus groups with surveys, group members not only can help provide language appropriate to their population, as noted, but also "augment pre-testing" of a preliminary version of an instrument (Morgan, 1988).

Initial instrument development, as well as adaptation of existing instruments, for use with different populations necessitates identifying appropriate items for inclusion. Because the populations targeted by these instruments usually represent an excellent resource for obtaining information critical to identifying and selecting items, they sometimes are utilized as such (e.g., Loesch & Vacc, 1993). Engaging these populations as focus group participants can provide an efficient means for the purposes of both item generation and refinement.

Perhaps one of the most intriguing aspects of working with focus groups is that, as yet, there are no definite rules for their use (Krueger, 1994; Merton, Fiske, & Kendall, 1990). Thus, many aspects of focus groups can be adapted to meet specific needs of researchers. For example, groups charged with generating hypotheses (e.g., item generation) should have a relatively unstructured agenda, while ones conducted to test hypotheses (e.g., item refinement) would need a relatively structured one. Moderator role also plays an important part in setting the group agenda, as well as tone. Finally, size of groups and membership characteristics can be adapted to meet research needs. Examples of how these variables were employed will be discussed in the context of a study that utilized focus groups in instrument development. The steps involved in the process, including generating items, identifying respondents, engaging respondents, and finalizing logistics, are described in the example provided next.

Focus Groups: An Application

The literature provides specific examples, albeit few, of how focus groups have been used for developing surveys (e.g., Wolff et al., 1993), improving surveys (e.g., O'Brien, 1993), and adapting surveys (e.g., Fuller et al., 1993); generating information on socially sensitive or undesirable topics (e.g., Jarrett, 1993; Zeller, 1993); and mobilizing communities (e.g., Plant, Landis, & Trevor, 1993). Additionally, several researchers have made recommendations for uses of focus groups in program evaluation studies (e.g., Brotherson, 1994; Flores & Alonso, 1995; Straw & Smith, 1995). Others have utilized such groups to test pre-established ideas or concepts for their appropriateness with a new population (e.g., Fuller, et al., 1993). The literature does not, however, include accounts of focus groups directly utilized in survey item development, although some researchers have conducted focus groups that subsequently affected their item generation process (e.g., O'Brien, 1993). Following is a step-by-step application illustrating an effective use of focus groups in survey item development that can inform other field research and practitioners in their work, particularly as they consider implementing this viable tool.

In developing the Volunteer Work Behavior Questionnaire (VWBQ), focus groups comprised of direct service volunteers were utilized to generate actual questionnaire items and to refine existing ones. Focus groups were deemed as more appropriate than other, more traditional methods because virtually no items had previously been developed specifically for the population in question. Secondly, items generated for related populations needed to be modified in order to attain more appropriate terminology. Written solicitation of input, which sometimes has been used for this purpose (e.g., Loesch & Vacc, 1993), would not have been appropriate for the population due to vast differences among potential respondents' educational level, training, and exposure to research. These differences might have limited the representatives of written feedback, particularly because of the level of written skill that would have been required at this point in the study (e.g., open-ended responses). Although this difference in ability may have diversified the actual focus groups, vast disparities in verbal skills were not anticipated. We felt that the commonalities among volunteers at similar agency types (Krueger, 1994), including minimal verbal competencies, would outweigh any differences in educational level.

The purpose of the study was to identify work behaviors of direct service volunteers, defined as volunteers providing direct services to clients of their respective mental health service agencies. It was established, initially, that volunteers' economic and social contributions to the workforce are tremendous (Brudney, 1990; Cnaan & Amrofed, 1993; Hodgkinson & Weitzman, 1992; Naylor, 1984), but that, in spite of the magnitude of volunteer contributions to agencies' direct service programs, no systematic analysis of their actual work had been conducted. In contrast, work behavior analyses have been conducted for paid helping professionals (e.g., Fitzgerald & Osipow, 1986; Loesch & Vacc, 1993; National Certification Reciprocity Consortium/Alcohol and Other Drug Abuse, Incorporated, 1992). Because knowledge regarding positions of paid employment cannot be generalized to volunteers (Cnaan & Cascio, 1999), an independent evaluation of work tasks also was necessary for the field of volunteerism.

It was decided that a survey instrument would be developed for use in the study, as in other work behavior analyses. Just as in any other survey development process, the questions first had to be identified and put into operational terms (Heppner, Kivlighan, Jr., & Wampold, 1992). In this case, one obstacle was the previous lack of systematic scrutiny of volunteer work behaviors. The first issue, then, was to determine the appropriate items for measuring volunteers' work behaviors. Some items could be taken from questionnaires utilized in related studies (e.g., "counsel clients concerning physical abuse," "use active listening' skills"), but they would need to be adapted to volunteers in mental health service. Additional items also would need to be generated. Pre-testing of the instrument would need to occur prior to the final study.

Generating Items

Focus groups were chosen as a tool that would potentially serve several of these purposes. After the initial item list was generated from related instruments (e.g., Bonner, 1993; Loesch & Vacc, 1993) and descriptions of volunteer programs in the literature (e.g., Cnaan & Amrofed, 1993; Coffman & Coffman, 1993; Cotton & Range, 1992; Miller, Hedrick, & Orlofsky, 1991), the items needed to be adapted to a volunteer population by modifying the terminology appropriately (Krueger, 1994). The focus groups would be charged with this task, along with that of generating additional items. Finally, through conducting the groups at the initial stages of the study, a rough version of the instrument would be pre-tested indirectly.

Identifying Respondents

Next, a reasonable sample needed to be selected. Usually due to the practical necessity of conducting the groups locally, focus group participants do not constitute a random sample. Nonetheless, they should reflect the population of interest (Stewart & Shamdasani, 1990). Volunteers in direct service roles were selected because of our interest in actual service delivery to agency clients. Due to the vast types of agencies providing mental health services to clients, it also was necessary to select a small number of agency types that would be representative of the overall population. Three areas of mental health service representing a wide range of services in the overall field of mental health are hospice, crisis, and general family services (Nassar-McMillan & Borders, 1999). Hospice program volunteers provide many short-to-intermediate?term services, such as companionship and assistance with daily tasks. In contrast, crisis services focus on the immediate-to-short-term amelioration of client symptoms by facilitating problem solving and making appropriate referrals. Family services cover a wider realm of services, ranging from family systems interventions to programs dealing with individuals or issues relating to overall functioning, such as developmental disabilities or mental illness. Thus, by targeting these three groups for participation in the focus groups, we were able to solicit insight and perspectives spanning the issues across types of mental health agencies.

Engaging Respondents

The next three steps in planning and conducting focus groups involve identifying the moderator, generating and pre-testing the interview guide, and recruiting the sample (Stewart & Shamdasani, 1990). In this case, these three steps occurred simultaneously. In selecting the most appropriate moderators or facilitators we considered questions such as whether the principal researcher was too invested to maintain a nondirective agenda if one were deemed necessary, how many groups should be held, and whether the number of groups would dictate having multiple facilitators who needed strict guidelines to ensure consistency. We decided that the principal researcher would serve as the moderator for each focus group because an overall understanding of the study was deemed necessary in order to keep the focus groups on task. In addition, this individual would be able to play the role of "enlightened novice" (Krueger, 1994) because of some prior experience with volunteers and volunteer programs.

Meanwhile, in generating and selecting the interview guide, the level of structure had to be determined, based upon the ultimate goals of this research step. Then topics and interview questions had to be selected accordingly (Stewart & Shamdasani, 1990). Our interview guide was developed through discussion between the principal researcher and a representative from the local Voluntary Action Center, an agency serving as a clearinghouse for volunteer opportunities to determine the most effective interview format. A secondary outcome of the discussion was that extraneous and duplicate items were culled from the list. We also consulted with several volunteers regarding items and sequencing. In this way, we utilized both experts and representatives from the population (Krueger, 1994).

Finally, the sample selected must be recruited (Stewart & Shamdasani, 1990). Here, we had to address the question of how to construct our sample, as well as how many groups to recruit. In order to comprise a focus group study, a minimum of two (Vaughn, et al., 1996) or three (Krueger, 1994) groups must be held. Because hospice, crisis, and family service agencies were chosen to represent the population, it was determined that three focus groups would be held, each one comprised of participants from its own respective agency type. We decided that, in our case, intergroup heterogeneity and intragroup homogeneity with regard to agency type would be best, so that each group could focus on the actual work tasks conducted at their specific agency type. We employed active solicitation in recruiting focus group participants. The agencies were identified with the assistance of the Voluntary Action Center. The principal researcher contacted the director or the volunteer coordinator at each identified agency and requested names of individuals who currently served as volunteers in direct service roles, as well as a volunteer coordinator to represent the agency. Desired focus group size was 4-6 participants, representing ideal mini-group size (Greenbaum, 1993; Krueger, 1994)--large enough to generate discussion, yet small enough to maintain adequate control over the agenda. Each group was comprised of at least 4 participants, including at least one agency volunteer coordinator. Both volunteers and their coordinators (or supervisors) were desirable because of their potentially differing views or perspectives of the direct service volunteer work.

Finalizing Logistics

Conducting groups involves a variety of logistical decisions, such as where the groups should be held, at what time, and for how long (Krueger, 1994). In our case, the groups were scheduled as 2-hour sessions (Greenbaum, 1993). Each one was held at one of the agencies affiliated with particular groups' members. Meeting times were early evening, which had been identified by the majority or the participants as the most convenient time. Light refreshments were served.

In keeping with ethical guidelines (Rubin & Rubin, 1995), the moderator began the groups by discussing the purpose of the groups and obtaining informed consent from participants for their participation in the research as well as for audiotaping. Participants were encouraged to freely share their own thoughts and ideas; it was stressed that the moderator, as the principal researcher, was there to glean information from the groups rather than to provide "correct information." The moderator also made a point of assuring them that their candid responses would have no negative consequences on their volunteer status (Rubin & Rubin, 1995), a position strongly supported by the volunteer coordinators. In fact, the volunteer coordinators expressed their desire to utilize the information they learned in improving their volunteer support function as well as, potentially, client services. Few of the participants knew one another; many volunteered in different agencies or programs. In an effort to build rapport, members were asked to introduce themselves and to briefly share the nature or their current volunteer work.

The group agenda was planned to last approximately 1 1/2 hours. Due to the large number of items (225) to be reviewed by the groups, it was necessary for the group to remain on task, thus, the moderators or facilitator, had to be relatively structured and task-oriented. The participants were asked to listen to each item, and to decide as a group whether or not they performed the task represented by the item as a part of their volunteer duties ("yes" or "no"), or whether they performed some semblance of it, but would describe it in different terminology ("yes--with revisions"). Because we wanted to be cautious in using dichotomous questions (Krueger, 1994), we encouraged the groups to discuss how they would propose to modify or combine items. We wanted to go beyond simple "yes" and "no" responses to better understand the thoughts and perceptions of the volunteers (Merton, 1990). On the other hand, we wanted to achieve a balance between qualitative and quantitative information (Krueger, 1994). The focus group sessions were audiotaped for later review. This review served as a confirmation of the facilitator's notes, as well as to provide an opportunity to consider qualitative outcomes.

The group members, as is characteristic of focus groups, readily engaged in extensive discussion over many of the items. In some cases, group members quickly decided upon the inappropriateness of specific items (e.g., "provide group vocational counseling"). Conversely, group members agreed relatively quickly on the appropriateness of some items with regard to specific tasks as well as in the terminology used (e.g., "use active listening skills", "participate in on-going education and skill training"). Other items generated extensive discussion both of appropriateness of the task to their volunteer work as well as terminology (e.g., discussion on "counsel the bereaved" item led to rewording of item as "support the bereaved"). It was the nature of these discussions within the focus groups that facilitated the greatest refinement of items as well as the subsequent generation of additional items.

The moderator's main role in leading the groups was primarily to help them stay on task as necessary and also to help the group members in articulating their perspectives on the work behaviors presented. While time management by the moderator was required, the agenda was not rigidly adhered to (Greenbaum, 1993). At times, the moderator deemed it necessary or beneficial to probe into deeper issues, skip items that might already have been inadvertently discussed, or to allow new items to emerge from discussion of the pre-generated ones. In probing specific topics, the moderator was careful of not encouraging individual participants to discuss any topic against their will (Vaughn, Schumm, & Sinagub, 1996), however this concern did not emerge within any of the groups. The quick agreement on some items allowed for more extensive exploration of others. The 90-minute framework did not pose a challenge.

Questions during the last stages of conducting focus groups involve determining how the information gained from the groups should or will be used (Stewart & Shamdasani, 1990). Our series of focus groups yielded invaluable information. At the conclusion of the three groups, the principal researcher culled more items from the list by utilizing the feedback from group participants. Items which had received three "no" responses were deleted from the list, with the assumption that if none of the three agency types engaged in the function, it was not representative of overall mental health service volunteer work behaviors. Items with two or three "yes" responses were retained, using the same guideline. Items with one "yes" response were scrutinized closely by analyzing sections of focus group audiotapes to review the rationale for "yes" and "no" decisions by the groups.

The focus groups were designed and executed for the purpose of refining the items on the instruments, as well as generating additional ones. In the case of this study, the focus groups were not deemed solely sufficient for item refinement. Thus, a subsequent pilot survey was conducted to further refine the items comprising the instrument. If the focus groups had been employed strictly for the purpose of adapting an already developed instrument for a different population of interest, this may not have been necessary.

In general, the last step required in processing focus groups is reporting the outcomes (Stewart & Shamdasani, 1990). The final questions we had to address at this step included determining what had been accomplished thus far, whether the focus groups had provided information sufficient to continue with the research, and whether the "report" would take the form of an actual written narrative, a revised version of an instrument, or some other product.

The product resulting from the analysis of the focus groups was actually a second draft of the instrument, comprised of 130 items with corresponding forced choice response options of "yes" or "no." Volunteers and volunteer administrators who participated in the pilot survey step were asked to respond by considering whether or not they performed each item and to provide written suggestions or comments about any of the items. The pilot survey phase resulted in 99 items, which comprised the final instrument entitled the Volunteer Work Behaviors Questionnaire (VWBQ).

Discussion

The use of focus groups as a step in this research study was highly effective. Because no prior research had been conducted on the topic of volunteer work behaviors and very few program descriptions existed in the literature, little was known about what volunteers actually do. The dynamics of the focus groups fostered the generation of numerous critical items which otherwise might not have come about. Secondly, while the moderator presented many items to the groups, in most cases the language, or terminology, simply was not appropriate to describe volunteer tasks. The groups, who in most cases reworded or otherwise modified the items to make them more appropriate to a volunteer population, addressed this issue. Thus, although it can be argued that the acquisition of objective knowledge is impossible (Seale, 1999), we believe that the inclusion of volunteers in developing an instrument to measure volunteer work behaviors added a quality control measure and, thereby, minimized the bias in the selection of specific survey items.

In our subsequent study, in which we actually administered the instrument to a national sample, participant responses were factor analyzed. Of the three factors, which consistently emerged throughout the series of factor analyses (i.e., for frequency, importance, and combined ratings), only seven items on the final instrument did not load onto any of the three factors (Nassar-McMillan & Borders, 1999). Thus, the vast majority of items, which were initially generated or refined by the focus group members, did load onto the factors with a .5 or greater factor loading. It also should be noted that, while participants in the pilot survey phase were asked to provide written feedback regarding the appropriateness of each of the items, the written responses were minimal. This may have been due to lack of interest or initiative; however, it also may have been that the refinement resulting from the focus group process already had been effective and complete.

Among the potential limitations involved in focus group research identified by Stewart and Shamdasani (1990) are limited generalizability of results due to small numbers of participants, as well as interdependent or biased responses of focus group participants. Several of these limitations were inherent in the use of focus groups for generating and refining items for the VWBQ, although we concur with Seale (1999), that these are present in many qualitative (as well as quantitative), research processes.

Although the three focus groups comprised of hospice, crisis, and family service agencies, respectively, were representative of the overall population of interest, participants in the focus groups were not randomly selected. This overall lack of randomization contributes to the limited generalizability of results. On the positive side, the moderator did ensure that none of the participants dominated the discussions, thereby minimizing excessive bias by one or more of the group members.

The focus groups, as did the survey, relied on volunteers' self-reports, posing a second potential limitation. This was controlled, in part, by involving volunteer coordinators in both the focus groups and in the pilot surveys. Relatively small numbers of group participants and the interdependency of responses also are inherent limitations in the use of focus groups as a research tool. Nevertheless, the strong factor structure, which emerged from the analyses of the overall study, indicates that, in this case, the effects of these limitations remained minimal.

Suggestions for Future Use of Focus Groups in Item Development

Several potential pitfalls may be inherent in the use of focus groups, but these can be combated with preventive measures. One problem in this case was that the volunteer coordinators initially were not particularly committed to the concepts of research in general, and thus were reluctant to become involved with the project. Other researchers also may encounter this issue in dealing with the general public. It would behoove researchers to develop a brief narrative, written in lay terms appropriate to the population, and give special attention to the initial contact with key individuals, especially if they are to be asked to solicit focus group participation. This narrative should include the overall objectives of the research as well as its contributions to the field. The narrative also should include a description of what benefits, if any, can be expected to be gained directly by participation in the groups. In this case, several of the volunteer coordinators were able to gain a greater comprehension of the relationship between research and practice through discussion of and participation in the research than they had previously had. If at all possible, research team members should consider soliciting the participation of potential group members directly. Further, both volunteers and volunteer coordinators alike, benefited from their involvement in the focus groups in several ways. The verbal feedback demonstrated an interest in the topic, a feeling of being valued, and also a self-realization of their accomplishments in their volunteer work. One of the volunteer coordinators actually used the focus group participants' experiences in further discussion with volunteers for education and training purposes.

A second point of note is that the homogeneity in agency type of the individual groups greatly contributed to the smoothness of the group process. Although the moderator did attempt to ensure that no individual members monopolized the discussion, this rarely became an issue in the groups. Because the participants of each group volunteered in the same type of agency as their fellow group members, there was very little extreme disagreement within any of the groups. An alternative structure might have been to hold four groups, separating volunteer coordinators out into their own group. We chose not do this because we thought that having their input in the volunteer groups would enrich the discussions.

Group size, however, did prove to be an issue, which affected moderator control over the group discussions. The groups of four-to-five participants were noticeably easier to facilitate than the third group of 6 members. In that group, an alternative structure was employed. The members were divided in half and the halves separated; each subgroup was given half of the list of items. Each subgroup was then instructed to determine appropriate "yes," "no," or "yes" with revision" responses for each item. The moderator rotated between the two subgroups in an attempt to monitor the discussions. At a predesignated time, the two groups reconvened and were directed to bring forth all of the items that had not unanimously received "yes" responses into the full group discussion. Discussion then ensued in the large group following the original format.

If such an alternative format were pre-planned, having two moderators present to work with subgroups might be necessary in order to maintain consistency between the subgroups as well as to insure that both groups stayed on task. It also might be beneficial to employ two moderators for focus groups in general, particularly if the size of the groups is relatively large or the agenda relatively structured. While this structure differs somewhat from the traditional or prescribed mini-groups, experimenting with focus group techniques and processes (Krueger, 1994) and using creativity in research design (Greenbaum, 1993) are integral to increasing the knowledge base of effective focus groups usage in research and practice.

Finally, the extent of quantifiable results of the focus groups was in the form of tallies of "yes" and "no" responses. This was critical to our overall study, because it allowed for a thorough and relatively quantifiable content analysis of the focus group sessions. When the researchers could not reach a quantifiable decision, analyses of audiotaped portions of focus group discussions allowed for determination of items based on qualitative information. The decision of whether to conduct qualitative or quantitative content analyses is another of the many factors that can be manipulated by researchers utilizing focus groups in item refinement. While there are no hard and fast rules regarding analysis of focus group data (Vaughn, et al., 1996), the balance between quantitative and qualitative data collection is a hallmark of focus groups (Krueger, 1994).

Overall, focus groups served their purposes of item development effectively. Many new items were generated and most of the pre-generated ones were modified according to appropriateness of volunteer work, especially with regard to terminology. One of the biggest advantages to using focus groups in this stage of instrument development is their flexibility. Allowing the researcher to make decisions regarding homogeneity versus heterogeneity, size of groups, numbers of groups, quantitative versus qualitative content analyses, among others, contribute to making them an especially viable alternative to more traditional approaches to item development.

References

      Bonner, A. (1993). Competencies in human services: A preliminary survey. Unpublished manuscript.

      Brotherson, M. J. (1994). Interactive focus group interviewing: A qualitative research method in early intervention. Topics in Early Childhood Special Education, 14(1), 101-118.

      Brudney, J. L. (1990). Fostering volunteer programs in the public sector. San Francisco: Jossey-Bass.

      Cnaan, R. A., & Amrofed, L. (1993). Mapping volunteer activity. Proceedings of the annual conference of the Association for Research in Nonprofit Organizations and Voluntary Action (pp. 45-49). Toronto, Canada.

      Cnaan, R. A., & Cascio, T. A. (1999). Performance and commitment: Issues in management of volunteers in human service organizations. Journal of Social Service Research, 24, 1-37.

      Coffman, S. L., & Coffman, V. T. (1993). Communication training for hospice volunteers. Omega, 27, 155-63.

      Cotton, C. R., & Range, L. M. (1992). Reliability and validity of the Suicide Intervention Response Inventory. Death Studies, 16, 79-86.

      Fitzgerald, L. F., & Osipow, S. H. (1986). An occupational analysis of counseling psychology. Journal of Counseling Psychology, 41, 535-544.

      Flores, J. G., & Alonso, C. G. (1995). Using focus groups in educational research: Exploring teachers' perspectives on educational change. Evaluation Review, 19(1), 85-101.

      Fuller, T. D., Edwards, J. N., Vorakitphokatorn, S., & Sermsri, S. (1993). Using focus groups to adapt survey instruments to new populations: Experience from a developing country. In D. L. Morgan (Ed.), Successful focus groups: Advancing the state of the art (pp. 89-104). Newbury Park, CA: Sage.

      Greenbaum, T. L. (1993). The handbook for focus group research. New York: Macmillan.

      Heppner, P. P., Kivlighan, Jr., D. M., & Wampold, B. E. (1992). Research design in counseling. Pacific Grove, CA: Brooks/Cole.

      Hodgkinson, V. A., & Weitzman, M. (1992). Giving and volunteering in the United States. Washington, DC: The Independent Sector.

      Hughes, D., & DuMont, K. (1993). Using focus groups to facilitate culturally anchored research. American Journal of Community Psychology, 21(6), 775-803.

      Jaeger, R. M. (1984). Sampling in education and the social sciences. New York: Longman.

      Jarrett, R. L. (1993). Focus group interviewing with minority populations: A research experience. In D. L. Morgan (Ed.), Successful focus groups: Advancing the state of the art (pp. 184-201). Newbury Park, CA: Sage.

      Krueger, R. A. (1994). Focus groups: A practical guide for applied research. Thousand Oaks, CA: Sage.

      Loesch, L. C., & Vacc, N. A. (1993). A work behavior analysis of professional counselors. Muncie, IN: Accelerated Developments.

      Merton, R. K., Fiske, M., & Kendall, P. L. (1990). The focused interview: A manual of problems and procedures (2nd ed.). New York: Macmillan.

      Miller, W. R., Hedrick, K. E., & Orlofsky, D. R. (1991). The Helpful Responses Questionnaire: A procedure for measuring therapeutic empathy. Journal of Clinical Psychology, 47, 444-447.

      Morgan, D. L. (1988). Focus groups as qualitative research. Newbury Park, CA: Sage.

      Nassar-McMillan, S. C., & Borders, L. D. (1999). Work behaviors of volunteers in social service agencies. Journal of Social Service Research, 24(3/4), 39-66.

      National Certification Reciprocity Consortium/Alcohol and Other Drug Abuse, Incorporated (1992). Study guide for alcohol and other drug abuse counselors for the National Certification Examination. Raleigh, NC: Author.

      Naylor, H. H. (1984, Autumn). Beyond managing volunteers. Speech presented at the national conference of the Association for Voluntary Action Scholars, Blacksburg, VA.

      O'Brien, K. (1993). Improving survey questionnaires through focus groups. In D. L. Morgan (Ed.), Successful focus groups: Advancing the state of the art (pp. 89-104). Newbury Park, CA: Sage.

      Plant, T., Landis, S., & Trevor, J. (1993). Focus groups and community mobilization: A case study from rural North Carolina. In D. L. Morgan (Ed.), Successful focus groups: Advancing the state of the art (pp. 202-224). Newbury Park, CA: Sage.

      Rubin, H. J., & Rubin, I. S. (1995). Qualitative interviewing: The art of hearing data. Thousand Oaks, CA: Sage.

      Seale, C. (1999). The quality of qualitative research. Thousand Oaks, CA: Sage.

      Stewart, D. W., & Shamdasani, P. N. (1990). Focus groups: Theory and practice. Newbury Park, CA: Sage.

      Straw, R. B., & Smith, M. W. (1995). Potential uses of focus groups in federal policy and program evaluation studies. Qualitative Health Research, 5(4), 421-427.

      Wolff, B., Knodel, J., & Sittitrai, W. (1993). Focus groups and surveys as complementary research methods: A case example. In D. L. Morgan (Ed.), Successful focus groups: Advancing the state of the art (pp. 89104). Newbury Park, CA: Sage.

      Vaughn, S., Schumm, J. S., & Sinagub, J. (1996). Focus group interviews in education and psychology. Thousand Oaks, CA: Sage.

      Zeller, R. A. (1993). Focus group research on sensitive topics: Setting the agenda without setting the agenda. In D. L. Morgan (Ed.), Successful focus groups: Advancing the state of the art (pp. 167-183). Newbury Park, CA: Sage.

Author Note

+Sylvia Nassar-McMillan, Ph.D., is an Assistant Professor of counselor education at the University of North Carolina at Charlotte. Her scholarly and service interests include multicultural and career development issues, with an emphasis on program development for underserved populations. She can be contacted at the Department of Counseling, Special Education, and Child Development, 9201 University City Blvd., Charlotte, NC 28223, USA; Phone: 704/687-4768; Email:
snassar@email.uncc.edu.

+L. DiAnne Borders, Ph.D., is a Professor of counselor education and Chair of the Department of Counseling and Educational Development at the University of North Carolina at Greensboro. Her interests include counseling supervision, ethics, school counseling, and adopted children and their families, and she has authored over 50 publications in these areas.

The authors wish to thank Sandy Leonard for her assistance in analyzing focus group content, and to David Morgan for his comments on an earlier draft of this manuscript.

Article Citation

      Nassar-McMillan, S. C., & Borders, L. D. (2002, March). Use of focus groups in survey item development. The Qualitative Report, 7(1). Retrieved [Insert date here], from http://www.nova.edu/ssss/QR/QR7-1/nassar.html

Sylvia C. Nassar-McMillan and L. DiAnne Borders
2002 copyright


Return to the top of the paper.

Return to the Table of Contents.