Community Support

Integrating Process and Outcome Evaluations

I. Introduction and Overview

Evaluators have been collecting information on program process and implementation for many years. They rely on both qualitative and quantitative data to describe the processes of a program's intended operation and to assess its implementation--suggesting why they may or may not be successful in achieving desired outcomes. However, most evaluations making use of qualitative and quantitative methods have presented the qualitative and other process data in a narrative "case study" vehicle, using it to describe programs anecdotally. These narratives have often been difficult to integrate with quantitative outcome data in statistical analyses. Yet it is becoming common to recognize the pressing need to integrate studies of participant outcomes with assessments of program process and implementation, to a achieve a more systematic understanding of why programs work or do not work.

The methods described in this toolkit provide one approach to addressing this need. They are part of an ongoing effort to facilitate the integration of process and outcome data. Developed over the course of several national evaluations of multi-site demonstrations of multi-component service programs they are beginning to be used in integrated statistical analyses of process and outcome data. Evaluations such as these have often used designs that include a process or implementation component in which case studies are developed separately for each project or site, with a summary cross-site report being used to highlight major differences or similarities among sites. While the multiple case-study approach can be very effective in characterizing a set of service programs, the technique has some limitations. First, traditional case studies are often long and tedious, especially if they are written to provide both a detailed program description and synthesis of the issues impacting the program. Long and detailed case studies may not often be read thoroughly or utilized to their full potential, either by program planners or by evaluators and other analysts. In addition, it can be difficult to extract equivalent information from several case study reports to support systematic determination of the reasons for a program's specific outcomes, even when they were written to a common outline.

One recommendation of case study methodologists for improving the quality and utility of case studies is that case study information should be systematically maintained in a case-study database so that validation and re-analysis is possible (Yin, 1989). We have found that this can be efficiently done by developing a project-level database that codes important characteristics of programs, their participants, environments, and their implementation at the project level. A project-level database supports analysts in developing a variety of quantitative descriptions of projects and their implementation histories, which can be used to supplement narratives and project logic models, and it can also support statistical analyses of client-level service and outcome data. Once characteristics have been reliably coded and entered into a database, cross-site analyses can explore relationships between project and community characteristics-especially those identified as relevant by program theory as likely to be associated with either implementation success or improved client outcomes-on the one hand, and project and client characteristics, such as indexes of implementation problems, services integration measures, client-level services, or client outcome measures, on the other. Analyses using project-level databases can benefit from the method's emphasis on careful coding and reliability checking of a wide range of characteristics as the database is developed, in advance of the participant-level analyses.

Perhaps the main use of a project level database is as a practical tool for managing the numerous characterizations of projects, coding them, and making them readily available to analysts. To examine a hypothesized relationship between a feature of program context and the success of its implementation, and the relationship between implementation and outcomes, for example, an analyst may want to tabulate and cross-tabulate characteristics in a variety of ways to produce quantitative descriptions of a program, and to explore possible confounding relationships or competing explanations among covarying background and implementation characteristics on the one hand, and outcomes on the other. It is thus an aid to the judgment process necessary in exploratory, multi--site analyses (Cordray, 1986), but it can also serve as a way to discipline analysts' judgments. Such databases can be open-ended, developing as long as time and resources remain to code new features suggested by preliminary analyses. As new indices and summary measures are suggested by the data or developed by analysts, they can be abstracted, coded or constructed, and added to the database.

This Toolkit provides an outline of procedures developed for linking and integrating data produced from both process and outcome evaluations. It is intended to be used in evaluating the implementation of mental health services and other programs, particularly for comparing characteristics of multiple projects or multiple implementations of a program at different sites. These methods were developed in the course of conducting several national evaluations of multi-site service demonstration programs for homeless persons (NIAAA 1992, 1995; CMHS, 1994). In these evaluations it was considered important to produce standardized descriptions of the context, structure and implementation experiences of individual projects. With these descriptions in hand, evaluators could compare and synthesize analyses of project implementation, and then integrate these project-level data with data collected on individual clients or program participants. The client-level information collected included process data on the types of services delivered to participants, as well as a variety of individual outcomes such as changes in housing status, alcohol and other drug use, and psychiatric symptoms.

The authors of this toolkit believe that program administrators and evaluators of other mental health service programs, including innovative programs being designed and implemented in a single site, will also be interested in methods for systematically coding and rating the environment, structure, and history of their programs. As we indicate in the following section, the benefits may include an enhanced ability to interpret and evaluate program outcomes through comparison with other programs that could not be achieved in more traditional ways.

The successful development of a project-level database is one way of joining, linking, and permitting the integration of qualitative and quantitative, case study and outcome evaluations. As outlined in the following sections, it is also methodologically linked to recent developments in meta-analysis and hierarchical linear models for multi-level analysis of evaluation data.

A. Project-Level Databases, Meta-Analysis and Case Studies

The methods used in developing project-level databases are related to the methods used in some multiple-case study and meta-analytic designs. The following discussion describes the connection that does or can exist between the project-level database and these two methodologies.

Meta-analysis

The "Practical Meta-Analysis" Toolkit by Lipsey and Wilson presents the basic techniques of meta-analysis for mental health service researchers and program evaluators. Meta-analysis or quantitative research synthesis is a family of statistical techniques that allow comparisons and averages of the standardized group differences or effect sizes found in evaluations using different outcome measures. In this approach, the effect sizes are calculated after the evaluations are completed and outcome data have been collected (e.g., Cordray and Fischer 1994). However, one of the key features of meta-analysis is the coding of "study characteristics" to better understand variations among effect sizes (Stock, 1994; Orwin, 1994). The nature of treatments, the characteristics of programs and participants, and features of measurement and research designs are customarily coded into a meta-analytic database, along with one or more effect sizes derived from the study or evaluation. The relationships between program and design characteristics on the one hand, and the effect sizes, on the other, can then be used to address questions such as the following:

Which interventions appear to work best?
Under what circumstances?
For whom?
Which behaviors and/or health variables change?
For which populations?
At what cost and with what benefit?

A PDB can be thought of as a prospective "meta-analytic database" for a multi-site program. It could have the same overall structure and tries to code many of the same characteristics as a database coded for a meta-analysis. However, because a PDB is developed while programs are ongoing, and its design may even begin before some programs have been implemented, it may be structured from the outset to capture the range of project characteristics found in a multi-site study and it can reflect plans for process data collection from all sites, rather than being forced to include just the information available in published reports on completed programs. The term "contents meta-analysis" introduced by Figuerado and Scott (1992) seems fitting for the focus of a PDB, although the enterprise they describe would actually be focused almost exclusively on the methodological characteristics of the studies or evaluations being analyzed, and so might be better referred to as "methodological meta-analysis". Although some aspects of study design and implementation will typically be coded in a PDB, its very extensive emphasis on features of the program characteristics and the community contexts of a demonstration project or site distinguish it from traditional meta-analysis.

Unlike a meta-analytic database, a PDB does not need to include estimates of effect size, which are the focus of meta-analyses. Effect sizes would not necessarily be part of the cross-site statistical analysis of outcomes in a multi-site study, since analysts of a multi-site demonstration may have access to the individual-level outcome data and may choose to analyze the pooled data directly, as in a multi-site clinical trial. However, once effect sizes have been calculated from individual outcome data, the effect sizes may also be considered "project-level data." Each project that compares at least two groups of participants can be characterized by one or more effect sizes. If these effect sizes are calculated for a multi-site demonstration and are entered into a PDB along with the descriptive project-level data, the result is a (small, localized) meta-analytic database. The main difference is that an evaluator constructing a PDB is likely to be much more focused on capturing details of program context, structure, and history, with a view to understanding variations in program implementation. In contrast, a meta-analyst analyzing published studies or final reports is typically focused more on determining average program effects across multiple programs, perhaps with coding of methodological characteristics of the study and the most prominent differences in theoretical orientation or intervention structure. There is likely to be somewhat less emphasis on explaining variation in those effect sizes by reference to characteristics of programs, contexts, participants, and so forth. In fact, poor reporting or variations in the reporting categories may preclude coding many such features from reports prepared by many different authors. Seen this way, the differences are in emphasis and detail, reflecting uses for different purposes and practical constraints, as well as in the formality and extent of the coding, as described in a later section.

The close similarity between the databases constructed for meta-analysis and a PDB means that much of the experience in developing and managing such meta-analyses is relevant to PDBs. The recent Handbook of Meta-analysis (Hedges et al., 1994) for example, contains an excellent chapter on "Managing meta-analytic databases" (Woodworth, 1994) as well as chapters on coding and reliability issues. The advice contained in these chapters applies as well to the development of a PDB.

As we will see later, a PDB developed for a single, multi-group project or a multi-site demonstration can later be brought together with the data in a meta-analytic database using the kind of meta-analytic coding described in the Lipsey and Wilson toolkit. When a PDB is planned with a view to integrating a demonstration program's results with an existing or planned meta-analytic database, the coding of common domains for the PDB can make use of and build on coding schemes already in use for meta-analysis. Although some aspects of study design and implementation will typically be coded in PDBs, the very extensive emphasis on features of the program characteristics and the community contexts of a demonstration project or site distinguish it from traditional meta-analysis, and make it more similar to the formalized "case surveys" whose methods are outlined by Yin (1989). These and other developments in case study methods are therefore relevant to a PDB.

Case study methods

A second connection of PDB methods is with formal methods of multiple-case study analysis, like those described in the toolkit on improving case study methods (by Sechrest et al., 1996). The data collection and coding for a PDB do not themselves constitute case studies, but case studies of program implementation are generally based on similar types of data collection. We have conducted such case studies in connection with evaluations of multi-site demonstrations (NIAAA 1992b; R.O.W. Sciences, 1995). The development of project-level databases in connection with the latter evaluation was motivated partly by the need to extract information in a form more easily incorporated into analyses of services and outcome data than would the narrative implementation case studies.

Although the case-survey approach described in Yin's (1989) textbook on case study research is not in wide use, there are examples and descriptions of case-survey methods going back at least as far as Lucas's (1974). The technique involves generating a closed-ended instrument resembling a survey which is applied to each of a large number of case studies, published or not, by one or more "reader-analysts." The data are then analyzed in many of the same ways a conventional mail or telephone survey would be analyzed, including quantitative summaries and statistical analyses when the number of cases is sufficiently large. As Yin points out, the coding involved can be cross-checked and reliability-assessment techniques can be used to improve and quantify the quality of the resulting data.

Attention to reliability and validity in conducting case studies is a main feature of the Case Study Toolkit, in line with its goal of improving case study methods. A focus on the categorization, rating, and coding of the information in a case study is a feature of both the case survey and PDB approaches. These activities are an application to program evaluation concerns of what Miles and Huberman (1994) call "cross-case analysis" of qualitative research in general. The emphasis in PDB is on coded abstractions of limited, but standardized aspects of multiple sites, projects, or programs, whether or not case studies are being conducted. The goal in single case studies is conducting valid, persuasive, or credible individual case studies of programs. Multiple case studies conducted using the standards advocated in the Sechrest et al. toolkit would, however, be easily coded into a project-level database.

B. Organization of this toolkit

This toolkit is intended to explain and illustrate the concept of a project-level database and its potential uses. It will also help in assessing the appropriateness of developing a PDB for your needs and within the limits set by the resources available. For those who decide that a PDB is worth developing, the toolkit provides assistance in planning for its implementation and analytical uses.

Throughout, the toolkit makes reference to and draws examples from a project-level database developed for the National Institute on Alcohol Abuse and Alcoholism (NIAAA) program Cooperative Agreements for Research Demonstration Projects on Alcohol and Other Drug Abuse Treat-ment for Homeless Persons (NIAAA and NIDA, 1990). A description of the NIAAA Cooperative Agreement program and the its national evaluation are contained in Appendix A, and the coding form used in developing this database is contained in Appendix B. We also refer to a more extensive database being developed for the ACCESS (Access to Community Care and Effective Services and Supports) program of the Center for Mental Health Services (CMHS) (Randolph, 1995) and another database in development that combines project-survey and archival data to describe the National Institute on Drug Abuse (NIDA) Cooperative Agreement for AIDS Community-Based Outcome/Intervention Research Program. These multi-site demonstration programs are described in some detail later, in the context of planned or ongoing analyses that use information from their project-level databases.

Table of Contents | Next