Community Support

Integrating Process and Outcome Evaluations

IV. Coding Variables for a Project-Level Database

This section reviews the steps taken and the materials used in developing two PDBs and in planning several others. We make some suggestions and recommendations, but we do not regard the procedures we have developed as optimal. In fact, some lessons learned in the course of conducting these two evaluations suggest possible improvements over some of the procedures followed in developing these databases. Examples are derived primarily from the NIAAA and CMHS ACCESS homeless demonstrations, which featured comparative evaluation designs, multiple sites, implementation, services delivery, and outcome evaluations of case management, counseling, substance abuse, health, and housing service programs. Our suggested extensions of the methods to some less elaborate designs (not multi-site demonstrations) lead to discussion of some hypothetical cases and analogous work in meta-analysis.

In general, the approach we advocate for developing the PDB structure and code book, is the closest possible approximation of formal methods of qualitative data analysis (Miles and Huberman, 1994). Rather than viewing it as a routine preliminary data collection or research task ("coding") this view demands that the coding and rating of project characteristics be treated as analysis ("the categorization, ordering, manipulating, and summarizing of data to obtain answers to research questions...to reduce data to intelligible and interpretable form" (Kerlinger, 1973)). Recognizing this, ample time and research resources should be allocated for analyzing the wealth of program description and implementation data obtained at the project-level in a typical evaluation of a demonstration project.

In a multi-site demonstration, the demands of analyzing such data across sites can be especially burdensome and difficult to estimate. Although the careful development of a PDB can require considerable planning effort and evaluators' time, ultimately any evaluation or policy purpose requiring cross-site synthesis of the project experiences must depend on some version of such a database. In a number of evaluations of which we are aware, this has been neglected or left until the final stage of analyses, to the detriment of the validity, thoroughness, and integration of qualitative data into the overall analysis and interpretation of results. It is worth devoting time and effort, instead, to assembling and making use of a team of locally available consultants and researchers who can assist in planning and developing the database from the outset.

A. Identifying members of a project-level database team

A project-level database team should be selected to represent the breadth of the evaluation and program interests among the stakeholders in the programs being studied. The team will help refine the list of variables and provide insight into the coding schemes that are used. There should be at least one person on the team who understands the evaluation plan, the nuances of the program theory, the program's components, local community/environmental factors, and factors potentially influencing program implementation--as well as the practicality of identifying appropriate data sources. In planning for a PDB in a supportive housing demonstration, for example, we found that expert consultants in relevant program areas played a strong role in developing very specific variables that would not have been apparent to less expert nominators, and that were not represented in the sparse literature on the subject. Team members may include:

National and/or local evaluators
Federal/State/Local project officers
Project directors
Consultants/experts involved in this or similar program's development
Key stakeholders and potential users of the database

While a broad base of knowledge and experience is desired on the PDB team, it is also important that the team be "manageable" in size and structure, and also the availability of team members. While there is no magic number of people constituting the team, we recommend that no more than 10 people be on the team, since many will have numerous other commitments, and meetings or conference calls among even 5 or more team members can be difficult to arrange. This allows for adequate expertise and exchange of ideas among team members, while also providing for a manageable or controlled discussion. On the other hand having a very small group of team members, two or three, can bias the direction of the database in ways that are difficult to assess. There may be times when a small evaluation is being conducted and it is impractical to gather a large database team. In these cases, team members should be even more carefully selected with respect to the knowledge of the program and the stake or biases that they may have in the database and evaluation outcome. As with the development of all evaluation tools, objectivity is the desired outcome. Regardless of the size of the team, it is important for the evaluators developing the database to facilitate the group process. This will help to ensure that the variables and coding scheme selected is reasonable, in light of the evaluation design and the data sources available.

Ideally, team members would meet at least once face-to-face, but at a minimum, they should be available by telephone or e-mail. Team members should understand the importance of timeliness and deadlines in developing the database contents. Most database fields should be developed prior to the onset of data collection, so that data collection instruments include questions that will gather the desired information appropriately. Data elements that are suggested after data collection begins require retrospective data collection, which increases the variability of the response across sites, thus impacting the use of the variable in a cross-site analysis.

B. Initial identification of data elements and structures

Some data elements will be fairly easy to identify as being important, and some of these will even be fairly straightforward to code, without reliability or other issues. As shown in Figure 7, examples will include elements like the names of the projects, state, community; nominal program model; and target population demographic characteristics. Nonetheless care needs to be taken that all such elements are accurately and consistently coded, and all coders have access to comparable sources of information.

Figure 7. Easily coded data elements from the NIAAA database

Name of City/State (two character code)
Type of agency
1 - AOD treatment
2 - Health care
3 - Mental health
4 - Social service

Targeted Client Characteristics

Women
Women/Children
Men
African American
Native American
Hispanic
Veterans
Dual Diagnosis
Primary Alcohol
Primary Cocaine
CPI 0 - No 1 - Yes
0 - No 1 - Yes
0 - No 1 - Yes
0 - No 1 - Yes
0 - No 1 - Yes
0 - No 1 - Yes
0 - No 1 - Yes
0 - No 1 - Yes
0 - No 1 - Yes
0 - No 1 - Yes
0 - No 1 - Yes

Some aspects of program components will also be fairly straightforward to represent in standardized form. These might include items such as planned case management caseload and location of services. Some features of program implementation will also present few problems, such as dates of first intake, etc. Difficulties can begin to arise when attempts are made to capture features of a program's "theory" or "logic." Unless well-accepted taxonomies of program components exist--in published reviews that may include meta-analyses of outcomes or in sets of reports of completed multi-site demonstrations of similar programs--it will be necessary to employ all available expertise in making judgments about which aspects of programs to code, and how they can be most tellingly coded or rated. In the NIAAA demonstration, for example, several reviews of the case management research literature, including one completed by members of the team designing the database, provided some guidance on features that could be expected to make a difference in the effectiveness of service delivery or outcomes of case management programs. The literature failed to reveal a sufficient body of research or a consensus on the features of housing and supportive services that might be most important to capture. It was decided therefore to hold a conference call that allowed housing consultants and national evaluation team members who had previously reviewed lists of possible housing and supportive service items to discuss efficient ways of capturing key features of the housing and supportive service characteristics represented by the array of programs in the NIAAA demonstration. Although such consensus panels can be arduous--even when they are focused on the form of items rather than the coding of a particular program--we have found that they work well enough as a means of "item reduction" to recommend them for similar situations.

Some additional data sources can be used to prepare and support the development of an initial list of proposed data elements and tentative coding alternatives. We have found it especially useful to consider the following:

The selection of variables to consider can build on "principles" of the program area found in review articles, editorials, RFAs, and annotated bibliographies in the relevant research and program areas (for example: the Glossary of service activities for alcohol and other drug abuse treatment of homeless persons (NIAAA, 1991) or reviews of case management research such as Ridgely and Willenbring (1992).

Both conventional wisdom and evaluation-focused--possibly meta-analytic--literature reviews related to program content should play a role. The results and conclusions of literature reviews should be analyzed and extracted in ways that can be presented to other team members.

The limited body of general program implementation theory (e.g. Scheirer, 1987, advocating building on the Pressman and Wildavsky foundation) may provide some guidance, if it can be articulated clearly and applied to expected implementation experiences.

Again, because similar issues arise in coding characteristics of studies in a meta-analysis, a body of literature is available to formulate guidance on how to identify "potentially interesting variables" and connect these to analysis plans (Lipsey, 1994). The Lipsey and Wilson (1996) Toolkit provides a useful list of "Study Descriptors" that refer to domains that may be worth coding over a wide range of programs. The list is divided into "Substantive Issues" (source of clients, type of treatment, theoretical orientation, etc.) "Methods and Procedures" (research design, random assignment, sampling, attrition, etc.) and "Extrinsic Variables" (date a place of publication, funding source, etc.). Some of this guidance may be helpful in developing a PDB for a multi-site demonstration, but there may be less interest in methods and extrinsic variables, while there is greater interest in features of program setting or community context (cultural, geographical, economic, or temporal setting). In addition, since descriptions of programs may be more limited and subject to change, it may be more difficult to examine and manipulate them for planning purposes, than the characteristics of completed and published studies.

Lipsey and Wilson (1996) emphasize that there are practical conflicts between the desirability of coding a broad range of program characteristics for descriptive purposes and the fact that many of these features will have limited value in understanding outcomes, and we have found this to be so. In the development of the NIAAA database, for example, considerable effort was expended to negotiate and maintain adherence to a criterion for a "critical event" that referred to the potential impact of the event on client outcomes. The number of such events was thus kept to a manageable minimum. More generally, their warning about developing overly ambitious coding schemes is especially relevant in working with a PDB for an ongoing or planned demonstration project. This is because there is a natural temptation to take full advantage of the superior access to program information enjoyed by a PDB developer, in contrast to a meta-analyst. Not being limited by uncoordinated published reports, the PDB developer can plan to collect standardized information about key aspect of a program in advance, or fill in some missing items through follow-up data collection (e.g. in telephone contacts or subsequent site visits). Thus it is especially important to try to anticipate which aspects of the programs will be most interesting and useful to code, so that the total number of items is not overly burdensome.

C. Additional nominations of data elements are solicited

Regularly scheduled, interactive sessions with as many team members as possible should be held, at project meetings or in conference calls. Team members should be asked to discuss the criteria to be used in nominating variables for inclusion and guided towards achieving a common vocabulary. Whether or not preliminary discussion is possible, an open-ended request for a list of variables should be sent to all identified participants in the process, with a cover memo from a high-level program evaluation official.

Other issues should also be addressed in the initial phases, such as:

Whether to settle for a single "snapshot" of a program at a selected point in time, with some coding of implementation milestones and measures, or attempt multiple program portraits over time

Whether to focus on program components, intervention or demonstration groups, or overall programs as the basic units described in database records

D. Developing and refining appropriate instruments

At this point, a number of practical considerations must be addressed. It may be impossible to assemble all members of the team in one location for a brainstorming meeting or consensus discussion. Some form of mailed/e-mailed contact will then be necessary to elicit opinions from the desired range of respondents. We have tested at least two distinguishable approaches to this necessary step which differ in formality. (Variations on these approaches are of course possible.)

In the first method:

STEP 1: A complete proposed list of variables is incorporated into a questionnaire format;

STEP 2: Potential data sources, and the availability or accessibility of the data are listed for each potential item;

STEP 3: Respondents are asked to give each variable a priority rating based on both its potential importance to the process evaluation and the difficulty of obtaining and coding data;

STEP 4: These ratings are summarized in a report form redistributed to the respondents and other team members; and

STEP 5: A modified list of variables to pursue further is selected in a meeting or conference call.

A second approach which we used in one recent evaluation was less formal and sequential, more iterative and open-ended. It also extended over a longer period of the demonstration program's implementation. In this approach an initial list of variables was generated by selected members of the process evaluation team who were charged with developing the database, but who were not substantive experts in systems integration, the focus of the demonstration. After the initial development effort was introduced at a meeting of the national evaluation team, the first variable list was mailed to all other evaluators, technical assistance providers, and project monitors for comment and nomination of additional variables. An interim report and conference presentation then provided an opportunity to code and use selected "baseline" variables descriptive of the projects at the time of program start-up. A technical assistance conference and workshop for grantees was held during the second year of the program. These activities focused on developing strategic plans for systems integration. They provided an opportunity to identify and solicit from participants suggestions for a large number of potential factors that could influence the implementation of service systems integration. These were winnowed and refined by reference to the existing data sources: applications, progress reports, sites visits, telephone interviews, and the strategic plans produced in response to the technical assistance and workshops. This process led to a coding form for the second wave of items. Codes were developed, not only based on previous research, but on examining the range and specific content of the plans and implementation experiences of the projects in the demonstration program, by way of short summaries or notes recorded in "Comment" fields. Still further items were planned for development as later implementation issues arose.

This illustrates the adaptability of the PDB development process to different rapidly changing program contexts. Because data is being collect at the program level, rather than the client level, it is possible to rapidly assess site characteristics identified only in the course of an evaluation as potentially important, in a manner that would be impossible in client-level evaluations or service research.

E. Mailings, meetings, discussions to finalize coding schemes

Using either approach, it is necessary to plan for several iterations between the formulation of variable lists and coding forms on the one hand, and group discussion or systematic feedback and comment, on the other. Variables that are initially just names of domains or dimensions that respondents think are important must be operationalized as sets of codes or scales in relation to data sources, and then set up in final for coding in a series of panel-judgment and consensus discussions. These may be interspersed with preliminary coding and rating efforts that will reveal ambiguities, overlaps, and incomplete coverage in code sets. In general, all the principles of good questionnaire and coding instrument design are potentially relevant (GAO, 1986; Miles and Huberman, 1994, Ch. 4; Sudman and Bradburn, 1982). In the recent meta-analysis handbook, Stock (1994) reviews some of these issues as they have been applied to the coding of study characteristics.

Several objectives should be considered during this process:

Assuring content validity (coverage of important issues) within practical limits determined by resource availability and data access problems

Establishing common definitions of terms among team members and analysts from different disciplines

Developing clear and self-explanatory coding instructions whenever possible

Resolving both explicit and implicit disagreements on coding tactics and program implementation theories through discussions among coders and evaluators

In the development of closed sets of codes for any characteristics that are not numbers, dates, or similar items, we join Stock (1994) in recommending that analysts rely on widely used conventions wherever possible. These may come from published meta-analyses or reports of cross-site analyses in other multi-site demonstrations, or in some cases from standard questionnaire development handbooks (Sudman and Bradburn, 1982). One goal is to specify an appropriate range of clearly delimited options in advance, and minimize the use of write-in "other" categories. The standardized data that result can be assessed for reliability and entered directly into statistical analyses. The generally smaller number of cases involved in a demonstration or set of demonstrations, in contrast to the hundreds of studies possible in a meta-analysis may permit initial review and analysis of documents and other sources to help specify an appropriate set of codes before final coding, but the qualitative coding process should remain open to revisions and additions of categories that help to clarify contrasts among projects, sites, or treatments. Guzetti, Snyder and Glass (1991) have also presented one way to plan for the orderly additions to coding schemes during the process of coding.

(Appendix B) provides an example of a completed coding form, one which was used in constructing the NIAAA project-level database.

F. Managing the coding process

Developing an acceptable coding form is only the beginning of the data analysis that populates a PDB. A number of other key steps must be taken, and issues addressed.

Assigning coders
Standardizing coding instructions in a "codebook" or coding manual
Training coders
Assessing inter-coder reliability (agreement)

Unlike the coding of studies in meta-analyses, which is frequently accomplished by graduate students or research assistants, it is useful to be able to employ coders who have some--preferably deep--involvement with the mass of detailed process information that accumulates. Close acquaintance with data sources and the overall history of a project greatly facilitates both access of appropriate data and appropriate use of contextual information interpreting isolated statements about project characteristics. Fortunately, multi-site demonstrations often have assigned project monitors or continuing contacts who are able to perform or assist in the coding role.

Methods for assessing inter-coder reliability have also been reviewed in the meta-analysis context (Orwin, 1994). Assessing coder agreement constitutes a supplementary or side study, but is best undertaken as early as possible in the coding process. Typically, though, since there are fewer "cases" (projects or groups) to be coded in a multi-site demonstration than studies in a meta-analysis or other coding task, there is more leeway for a limited return to the coding task if alternatives or improvements results from the reliability assessment. Evaluators might take advantage of this circumstance to return to the coding task if it will provide:

Adjustments in coding forms or coding instructions that will substantially improve reliability;

Additional quality assurance, spot checking, consensus discussion of issues;

Provisions for emergent rules, re-categorization, and possible recoding;

Efforts to deal with missing data, data source problems, and unexpected ambiguities.

As with other qualitative analysis, coding project characteristics can be very time consuming. (The authors of the Case-Study Toolkit in this series, for example, warn evaluators to allow more than ample time and resources for all aspects of systematic case study work and, above all, to "set up realistic timelines.") It is necessary to plan for this in the PDB coding process. Since PDB coding builds on or proceeds in parallel to case study-like data collection, and needs to be iterative, with reliability and other data-quality checking, it too requires ample time. Fortunately, the entire course of demonstration program or other multi-year evaluation may be available for developing, coding, checking and revising the information in a PDB, and the total amount of time available may be greater than for some primary case study evaluations. Unfortunately, substantial evaluation resources are likely to be reserved for other aspects of the evaluation, and the qualitative data are often being collected for other purposes, most notably project monitoring. It is wise, therefore, to begin planning and implementation of the database from the beginning of a program, and expect that it will be completed and in its final form only in time for the final analysis and reporting. This is especially important if the PDB is planned to contain substantial data on service delivery or the full implementation history of the projects.

Table of Contents | Previous | Next