Community Support

Integrating Process and Outcome Evaluations

II. What is a Project-Level Database?

A project-level database is an organized collection of information that describes a program's environment, structure and history using standardized and systematic coding schemes. In this section, we first review different aspects of project-level databases and then present factors to consider when deciding whether or not it is appropriate for your use.

A. Ways of looking at a project-level database

There are several ways of viewing a project-level database. Understanding its components and potential uses can help to clarify its use in evaluation. Several important practical steps should be considered when building a database that can be used when documenting and analyzing program implementation and outcomes.

First, a project-level database is composed of a set of computerized records in an electronic file, structured and maintained through any database, spreadsheet, or statistical package software. Each record contains a set of variables that refer to characteristics of a project or the project environment, created by standardized coding, rating, and measurement of characteristics of programs.

Second, the kind of information in each project-level database record is defined by its place in the hierarchy of "levels" of information that correspond to natural units of organization in human services delivery. These levels of organization in turn correspond to units of analysis and, in evaluation designs, units of assignment to conditions. The levels of information range from the service system and funding level (including national, state, and local levels), to the program and treatment group level.

Figure 1. Levels of organization in multi-site demonstrations

National Programs
Participating States
Participating Cities/Communities
Provider Agencies
Treatment Groups

Figure 1. shows some of the levels at which program-relevant information can be gathered. Lower-level units may be subordinate to several higher levels, with multiple activities or organizations each level. In certain demonstration program areas, such as "innovative case management" demonstrations, for example, several national programs may be ongoing in partially-overlapping sets of states, each of which contains multiple participating communities and provider agencies. At each level any of several entities may have nested within it several units at the next lower level, each of which shares the characteristics of the unit in which they are nested. This hierarchical structure is indicated clearly in Figure 2, which makes the "nested" structure explicit, from the national level down to the level of treatment or comparison groups of participants. In the ongoing ACCESS demonstration, for example, only one national program is being studied, but it is implemented in 9 different states, each with 2 different project sites, one of which is implementing a systems-integration plan, and one of which is not. In the project-level database being developed for this demonstration, each of the 18 different project sites would have the same values for national data such as economic characteristics and program relevant trends. Each set of project or sites within a state will also have the same state data. Local data such as the cost of housing will vary by location, although in 3 of the states the 2 projects are located within the same city, so these projects share their city's characteristics as well.

It may often be desirable to represent separately the characteristics of two or more treatment modalities being compared at a site, leading to the development of separate records for each treatment group. In this case some variables describing differences between the groups will have different values while they will have the same values on many characteristics of the project as a whole, just as two different projects in one city or community will share all that community or city's characteristics but differ on variables describing the design and the implementation experience of each distinct project. For example, in each of the NIAAA homeless demonstration projects, there were at least two--and as many as four--treatment groups being compared within a site. The comparisons were often between an intensive case management group with additional housing and supportive services and a group closer to "usual care." Although the treatment groups in a site have in common all the values of characteristics of their state, city, and, often, of their sponsoring agency, they still differed on key components of service that were being studied. To represent this structure, a separate record was created in the project-level database for each group within a site.

The information in a project-level database can describe features of many different kinds of objects: service systems, treatment models, implementation histories, the average consumer of services in a community, or a target population. The one thing these all have in common is that they can be represented as characteristics "above the individual level." (This distinction is indicated by the dotted line in Figure 2.). In such cases, which will be typical of a PDB used for a multi--site demonstration program, the multiple levels of information described above could be represented separately in a true relational database, in which the different levels of information would be stored in different, linked files. For practical data management purposes and to accommodate the limitations of statistical programs, however, users can restrict the number of levels in the database by collapsing all supra-individual information to the level of treatment groups or projects, and representing higher level characteristics redundantly in each group record. The choice of the term "project-level" was made somewhat arbitrarily during the development of our first multi-site demonstration database, with the primary motive being to distinguish it from the "client-level" database of outcomes and services information.

Third, a project-level database can link qualitative and quantitative evaluations. The link is forged by putting qualitative process data into a form that permits the data to be merged with statistical databases containing more quantitative evaluation data at several levels. When qualitative program and contextual characteristics are coded and incorporated into a PDB, they are made available as discrete variables, to order breakdowns and cross-classifications with frequently collected quantitative data, for example:

client/consumer-level services delivery data
individual-level outcome data, and
measures of implementation success.

In this way, the database can be conceptualized as a systematic abstract of multiple case studies of programs and their implementation. By using standardized terms, categories, scales, and codes, it makes these case-study abstracts comparable across multiple programs, interventions, or replications.

Debate has raged in recent years about whether evaluations of human service programs should focus primarily on quantitative or qualitative information (Lincoln, 1991; Reichardt and Rallis, 1994; Sechrest, 1992) and thoughtful commentators on this debate have called for ways to synthesize the two approaches (Cordray, 1993; Datta, 1994). Dennis et al. (1994) outline a variety of ways of linking qualitative and quantitative evaluations of substance abuse programs, several of which are consistent with the PDB approach. The most noteworthy of these is the evaluation of repeatedly observed implementation problems against other sources of information as a way of building a body of hypotheses about program implementation theory.

Fourth, as the link between different types and levels of data, a PDB is an analytical tool that supports recently developed estimation techniques for multi-level statistical analysis that permit analysts to examine simultaneously the relationships of project and context characteristics, consumer or client characteristics, and assessments of individual outcomes. The first to report using these hierarchical linear models in program evaluation have also noted that they are ways of bringing qualitative program and participant characteristics into the analyses of quantitative data (Seltzer, 1994). A project-level database is a logical tool for use in these analyses.

It should be clear from this discussion that a project-level database is to be distinguished from implementation "case-study databases," which can consist of nothing more than boxes of documents or well-ordered file cabinet drawers that contain the source materials for case studies of program implementation--even when multiple case studies will be constructed using these materials. The essence of the project-level database idea is abstracting and coding--summarizing and systematizing--the wealth of qualitative and quantitative process and background information typically obtained on new programs in systematic and standardized ways.

B. Deciding when to use a project-level database

To an evaluator responsible for designing or implementing an evaluation of a multi-site program, the value or the necessity of developing something like a project-level database may be immediately apparent. A careful coding of project characteristics permits the systematic coding of the many sources of cross-site variation in multi--site demonstrations. These can include, for example:

Project context
Program "logic"
Project implementation
Project operations

Project staffing

Changes in the treatment over time

For persons responsible for monitoring the implementation of multi--site projects, it may be easy to see how the databases used to track and monitor implementation could play a role in analyses of the projects' implementation or outcome. But for the evaluator of a single, local innovative program, say, a variation on a case management model expected to improve it's effectiveness for homeless persons, the benefit of thinking about program components in standardized ways may not be readily apparent. So it may be worthwhile to review some of the larger issues and additional uses that would make these methods of interest to both evaluators and program directors of innovative mental health service programs.

One reason for trying to construct explicit descriptions of program components is the articulation and analysis of a program's logic or the "theory" of how it produces its effects on recipients. In recent years, the importance of having detailed and explicit representations of the way program elements are supposed to fit together and produce their beneficial effects on participants has been emphasized by many evaluators and also by critics of program evaluation (Bickman, 1990; Chen, 1990, 1994; Leviton, 1994; Lipsey, 1990). The urgency with which they put forward their recommendation derives from the disappointment at the number of negative or null outcomes of evaluations of new programs. The goal is to "open up the black box" of an intervention, so that both successful treatments and apparent failures to improve outcomes can be better understood by reference to program components and measures of their implementation. The role of process information is to facilitate the understanding of the mechanisms of program success or failure, as opposed to merely providing measures of success or failure. Detailed and reliable knowledge of program environment or context and component choices also facilitates the process of program replication in different community or service system contexts. Even without an interest in program theory for its own sake, there is often a practical need to understand why a particular implementation or realization of a model [e.g. PACT] is or is not effective, when the model has been effective in other sites (Bond et al., 1988).

The developing methodology of process evaluation has tended to emphasize ways of improving the measurement of key aspects of implementation and service delivery (for example, Scheirer, 1995). But in addition to these quantitative measures, there is a need to improve the integration of qualitative and other descriptive process evaluation into quantitative and comparative evaluations. Narrative case studies and implementation analyses can be of some use, especially if conducted in accordance with reasonable standards, but leave the program information in a form difficult to integrate with other data. Logic models can help program developers and managers to articulate their program model, and are a powerful means of conveying program configuration information to others (Rush and Ogborne, 1991; Unrau, 1993). But the information contained in a logic model still needs to be abstracted and transformed into specific codes before it can be used in analyses of other data.

Finally, there is also a general need to accumulate a comparison base of program experiences across demonstrations, evaluations, and other studies. In the last section of this toolkit, we review some new developments in evaluation methodology that suggest ways project level databases can be used in constructing a comparison basis for single programs to use in evaluating their relative success.

C. Does a project-level database meet your needs?

The decision to develop a project-level database should follow consideration of questions like the following:

1.   Is there an interest in comparing programs or comparatively evaluating similar treatments in several sites?
2.   Are there many common theoretical element among the programs to be compared?
3.   Are there appropriate resources and experienced staff or appropriate training available for developing an electronic database using the in-house database software?
4.   Are there sufficient resources (time, staff, training) to support an extensive, iterative, process of coding, checking, and recoding project characteristics?

An answer of "no" to any of these questions, should lead to reconsidering the appropriateness of a project-level database. The possible future use of coded project descriptions as a basis of comparison for newly developed programs should also be taken into account.

Table of Contents | Previous | Next