Community Support

Integrating Process and Outcome Evaluations

III. Selecting and Structuring Variables for a Project-Level Database

For researchers who have decided to capture some aspects of a multi-site demonstration or any other set of programs and develop a project-level database, the first step involves determining what information will be stored in it and how to select and structure data elements appropriate to the purposes. This section indicates some potential variable domains that can be developed, illustrating these categorical variables with examples from the NIAAA project-level database and references to additional sources that may be useful in identifying dimensions and categories to capture in a project-level database. There is also brief discussion of potential data sources.

A. Information at the program or "project level"

The only thing the various types of information in a PDB need to have in common is having been collected at, collapsed to, or aggregated to the same level--project, site, or treatment group, (depending on the choice made in setting up the database). This means that the data can be of several different types and come from very different sources. Examples include:

Characteristics of programs, their components, activities, and their immediate contexts. These can be derived from all the usual documentary and other sources for program description. Examples would include: length of case management; caseload size; presence and extent of outreach; parent agency focus; and presence of a local mental health authority.

Averages or summary statistics that characterize consumers or other program participants. These may be available from baseline (intake) or outcome assessments or client information systems. Examples include: target group characteristics such as gender (proportion male or female); average age; racial or ethnic breakdowns; diagnosis; client referral sources; and eligibility criteria.

Summaries of services delivered to participants. These may be part of planned process data collected at the individual level and averaged, or more qualitative characterizations of service patterns. Examples include: types of services; average length of service participation; on-site vs off-site service delivery; caseload size; and type of case management.

Characteristics of local evaluation designs or data collection procedures (sampling, design, etc.) These may include specific information about where and how samples were drawn and whether assignment to conditions was random or followed specifiable procedures.

Measures of program and design implementation. As discussed in more detail later, many aspects of a project's implementation history can also be coded: the type of service facility; length of time to set-up program; presence of NIMBY ("Not in My Back Yard") resistance to program siting; type of staff; staff turnover; impact of staff turnover; presence of program changes.

For the NIAAA demonstration we developed a standardized checklist of implementation problems that was used in analyzing cross-site implementation success. Some examples of items from the NIAAA database are shown in Figure 3. Project-level implementation measures may also be derived from individual-level measures, including summary statistics on service-delivery measures. For example, among other indicators of whether project components, activities, and designs were put in place as planned, it would be possible to include summary statistics on the observed degree of attrition from treatment and the extent to which a specific innovative services program was "contaminated" by services received from other providers.

Figure 3. Sample project level data elements from the NIAAA database

1. Type of agency (primary focus):
1. AOD treatment
2. Health Care
3. Mental health
4. Social service/multi service

2. Client Characteristics (Yes or No)
1. Women
2. Women and Children
3. Men
4. African American
5. Hispanic
6. Veterans
7. Dual Diagnosis

3. Extent of staff turnover (Clinical director and supervisors):
0 = Low (none)
1 = Medium (some)
2 = High (more than two cycles)

4. Impact of staff turnover
0 = Low (no impact on operations)
1 = Medium (some impact)
2 = High (severe impact)

B. Community and organizational context

In addition to characteristics of program components and participants, program environments are increasingly recognized as determinants of both service delivery and, indirectly, of outcomes. Although information on these characteristics will be available from the some of the same sources used to characterize programs, a detailed characterization of community context is likely to require additional data collection from a variety of local, state, and federal sources. It is also likely to require considerable processing and analysis before being suitable for entry into a PDB. Examples of community and organizational context variables include:

Economic, social, and services environments. These include statistics such as overall population, housing vacancy rates, availability of Section 8 vouchers, median income, unemployment figures, and education levels, as well as ratings by local experts of, for example, the community's attitudes toward homeless or mentally ill persons.

The number and characteristics of other service agencies in the community's service system. These could include, in addition to the numbers and types of service delivery agencies, the sectoral composition the service system, and the extent to which a service or intervention included in the project of interest was also provided by other community agencies.

Measures of collaboration, linkage, system integration, etc. These can range from informal coding or ratings of overall system linkage to statistics obtained through formal interorganizational network studies (CMHS, 1994; Morrissey, 1992).

Program resources, costs, etc. This could include overall and per capita mental health expenditures in the program area as well as overall and per capita Medicare and Medicaid expenditures.

Figure 4 shows a sample characterization of community-context data elements.

Does the city have a local mental health authority?
0=no 1= yes

Is the project in the local mental health authority?
0=no 1= yes

Is a substance abuse program part of the sponsoring organization?
0=no 1= yes

Amount of City-level per capita funds for mental health services
$_________

Is sponsor linked to housing development corporation?
0=no 1= yes

Figure 4. Community/organizational level data elements from the ACCESS project-level database

C. Program logic, component structure, or program theory

These are the kinds of characteristics of program structure and operation that are typically included in "program logic models" (Tessler and Goldman, 1982; Wholey, 1979). Logic models are essentially flowcharts that depict the intended operation of programs and the connections among their inputs, components, and outputs. The components and configurations of graphic logic models must be coded before it is possible to understand or specify exactly how two programs with different models differ, or to use the resulting information in analyses.

Examples of program-logic or program-theory elements exist in the literature for the following categories, among others:

Treatment innovation or systems change plans
Program philosophy, orientation, or theory
Case management models, team structure
Degree of consumer involvement, direction, etc.
Measures of planned program intensity; caseload, hours, availability, etc.
Connections between program components, sequencing, provisions for supportive services, etc.

Figure 5 shows examples of program logic or program-theory data elements that were coded in the NIAAA database. They were derived from expert-panel discussions and represent items included in the logic models.

Figure 5. Sample program logic/theory data elements from the NIAAA database

Philosophical Approach
1. Clinical (medical)
2. Social Model
3. Mixed

Orientation (focus on case management or combination with AOD or mental health counseling)
1. Single focus: CM
2. CM + AOD
3. CM + MH counseling
4. CM + AOD + MH

AOD treatment intensity:
a. Counselor client ratio
b. Frequency of contact (per week)
c. Number of required group meetings
d. Number of self-help meetings

D. Program implementation and changes over time: measures, key events, dates

One key to understanding why a program is or is not effective is its implementation history. Despite the program planning and program theory that initially defined the program, it is important to know how those plans were put into place and what factors may have influenced them. A PDB allows for the coding of such events and influencing factors. The values of key implementation variables can be identified by existing, relevant implementation theory, as well as characteristics identified in principles of implementation "tradecraft" for a specific program area. Baker (1991), for example, presents "Principles of Service Coordination" derived from a review of the systems integration/services coordination literature. Categories of implementation variables that might be included are:

Facilitators of implementation

Barriers to implementation and steps taken to overcome them

Dates of a small set of "critical events" in a project's history, both as
indicators of stages in implementation and as milestones for use in
analysis of outcomes or service delivery (before and after which these might be
expected to vary)

Changes in program logic, service components, key personnel etc.,
the course of implementation

Measures of implementation success, completeness

Figure 6 contains a sample of implementation data elements in the NIAAA database.

These were derived from both expert panel nominations and analysts' systematic review of implementation and formative evaluation reports.

1. Was NIMBY-type problem encountered?
0. None
1. Minor
2. Moderate
3. Severe
2. Did it delay implementation?
0. No 1. Yes
3. What was done to resolve this problem?
0. Went ahead with siting
1. Negotiated a resolution
2. Took legal action
3. Gave up; found new site
4. Did we provide T.A. on this?
0. No 1. Yes

Figure 6. Sample implementation data elements from the NIAAA database

E. Other guidance: "Program Templates"

The need for systematic characterization of similar programs is recognized in the recent development of guidance for developing program templates as tools for implementation assessment and formative evaluation (Scheirer, 1996). A program template is a matrix format for specifying the components of an intervention and comparing the elements intended to be implemented at a particular site with those actually implemented. Templates were intended to make it easier for evaluators and program managers to specify "best practice" or other external standards for programs, compare the components of a particular program with these standards, and then summarize what is learned in monitoring about the process and extent of program implementation. Although the method as practiced tends to keep the assessments at the level of qualitative summaries in a table, Lamberti and Katzenmeyer (1996) address ways of transforming qualitative data from templates into quantitative codes to describe multiple programs within a large-scale study. Thus, though templates were originally developed as formative evaluation tools, they are beginning to be used in cross-site comparisons and other comparative analyses. The volume edited by Scheirer includes templates for staff development and building new organizations, and also guidance for deriving best practice components for a particular program domain from literature reviews. It may thus provide additional guidance for identifying key dimensions and developing codes to structure a project-level database.

Beyond locating prior efforts to structure or code a program domain systematically, it will often be necessary to develop appropriate coding schemes for new program types through an iterative process. Initial outlines of codes that seem appropriate are based on detailed substantive knowledge of a program area's characteristics, and the contextual factors that may influence program operations, but these must be modified as attempts are made to code actual arrangements and new categories are discovered. Ridgely (1997) for example, is leading an effort to develop a detailed template to characterize contractual and operational arrangements between managed behavioral health care organizations and state Medicaid organizations, for a study of the impact of managed care on vulnerable populations sponsored by the Substance Abuse and Mental Health Services Administration (SAMHSA). The features and categories of managed care arrangements included in this template have been derived by repeated, detailed examination of documents and interviews with appropriate members of the organizations involved, over the course of many months, and reliance on a body of literature describing other managed care arrangements. When it is completed, this template will guide the collection of data that will form the core of a project-level database for this ongoing multisite cooperative agreement.

F. Appropriate data sources

One major difference between developing a project-level database for multi-site evaluations and conducting a meta-analysis, is the integration of appropriate data from multiple sources. Meta-analysts work from the information available in published articles, supplemented by their judgments and ratings or research quality and other methodological characteristics. Developers of project-level databases, in contrast, must collect or make use of data from multiple descriptive, monitoring, implementation and archival sources--some qualitative, some numerical.

In addition to the grant applications, site visit reports, interviews, focus group records, meeting notes, and other documents that accumulate over the course of a multi-year demonstration evaluation, the concern with capturing community context or a project's setting means that additional, archival data sources can and should be utilized. These could include, for example, census data and other generally available data sets such as the results of periodic national surveys, including, for example, the household and school surveys sponsored by NIDA on drug abuse. We have also made use of data from the Inventory of Mental Health Organizations on the city-level funding available for mental health services in a project site. When combined with census data, these figures can be used to estimate per capita funding available for mental health services.

Because of differences in community structure, researchers should use care in deciding whether to use data at a city, county, or state level. Dependent on the program purpose and the catchment are from which a program draws its clientele, the use of different data levels may be warranted.

Table of Contents | Previous | Next