Community Support

Integrating Process and Outcome Evaluations

Appendix A

Description of the NIAAA Cooperative Agreement Program

In fiscal year 1990 the National Institute on Alcohol Abuse and Alcoholism and the National Institute on Drug Abuse awarded funds to 14 individual research demonstration projects to provide innovative multi-component service programs to homeless persons. The interventions combined outreach, shelter, alcohol and other drug treatment, case management, mental health services, and alcohol and drug-free housing in a variety of approaches. The effectiveness of these projects was measured by their ability to reduce participants' alcohol and other drug consumption, increase the quality of their shelter and residential stability, and enhance their employment or economic status.

This demonstration program was a cooperative agreement that required each project to implement a comparative research design that could rigor-ously test the efficacy of the proposed interventions, to collect client-level data on all participants' service utili-zation and outcomes using standardized data collection instruments, and submit the data in standard formats to a national evaluation.

R.O.W. Sciences, Inc., and its subcontractor, Vanderbilt University, conducted the national evaluation of this multi-site research demonstration project. The national evaluation provided programmatic and evaluation technical assistance to the grantees; managed and analyzed all evaluation data submitted by the grantees; and analyzed, reported and disseminated information on the activities and results of the projects.

The specific tasks and activities of the national evaluation effort included the following:

A detailed National Evaluation Plan was prepared for implementing the evaluation design, and convened a panel of distinguished experts in program evaluation who reviewed and approved the plan.

Programmatic technical assistance was provided to all project grantees through direct onsite and telephone consultation on program policy, development, and implementation. Ongoing evaluation technical assistance was also provided to project grantees on all aspects of the design and implementation of their process and outcome evaluations and on their data collection, data management, data quality control, and analysis strategies and activities. Onsite training was provided to each project in the use of the core set of standardized data collection instruments previously tailored to the needs of the demonstration program. Follow-up telephone and onsite technical assistance was provided to grantee staff in collecting, coding, and managing the data from these standardized instruments.

A panel was convened to refine and update a comprehensive taxonomy of services for homeless people with alcohol and other drug problems. This taxonomy was published by NIAAA and has been adopted for other evaluation purposes. It formed the basis for an instrument used by service providers to record and report data on all services delivered to program participants.

A comprehensive technical assistance plan and an individualized technical assistance plan for each project. The national evaluation also:

assessed the early phases of implementation of each project and its local evaluation using narrative reports, site visits, and interviews;
analyzed and described each project's interventions and evaluation designs in the context of the state of the art reflected in the treatment services evaluation literature. This information was used to generate detailed graphic logic models of the program, both as it was intended and as it was actually implemented;
identified technical assistance needs during regularly scheduled periodic telephone interviews with each project's program and evaluation staff;
monitored narrative quarterly reports and quantitative data submitted by projects to the national database for problems requiring technical assistance, and;
conducted a 2-day site visit each year to each project to observe and review program operations, interview program and research staff, community and other representatives and clients.

A final report on evaluation and programmatic technical assistance summarized and analyzed the delivery and effectiveness of this technical assistance effort.

A database management and data quality assurance plan was produced, under which the company collected, compiled, aggregated, and analyzed all data required for the national evaluation. It:
designed templates with standardized names and formats, name and label templates, and appropriate data file formats for all outcome and services utilization instruments;
prepared a comprehensive schedule of periodic data submissions that built the national database as the demonstrations proceeded. Both the core outcome and service utilization data were compiled and aggregated.
acceptance inspection of all data files submitted by grantees for compliance with national database format requirements and identifier consistency;
logging and tracking all data submissions, resubmissions, and quality control procedures;
range and consistency checks; data entry verification on random samples of client interviews; feedback and technical assistance to site data managers on systematic data problems; and
conversion and recoding of site-specific data formats for integration into the national database.

Interim evaluation reports were based on the data submitted by grantees. These reports contain comparative summaries of participant characteristics and project activities, and report on the implementation history, the flow of participants through service components, and the services delivered to all participants in each of the projects.

Two 3-day working group meetings were conducted per year, during which lessons learned about the design and implementation of programs were shared by grantee's program and evaluation management and line staff. Participants in the working groups planned collaborative analyses and publication of results, and staff monitored grantee technical assistance needs. A special data analysis workshop was conducted in the final grant year to assist grantees in analyzing and reporting their results.

A 4-volume final national evaluation report was based on a systematic integration and analysis of data from all sources for the national database:

Volume I: Cross-site Synthesis of Intervention Designs and Implementation Analyses. This volume presents a systematic overview of cross-site variation in intervention designs and participant characteristics, categorizes and describes the various startup and implementation problems experienced by the projects, and analyzes the potential effects of these on the integrity of the interventions and their associated evaluations. It also synthesizes the participant-level services data with respect to: (a) specific service categories, (b) global services scales (strength, fidelity, and leakage), and (c) changes in services over the life of the projects. Combining the qualitative and quantitative evidence, it assesses the degree to which the Cooperative Agreement interventions were implemented as intended.

Volume II: Cross-site Synthesis of Retention Analyses. As concluded following the implementation analysis in Volume I, the Cooperative Agreement interventions were implemented essentially as planned, but the intended group distinctions in "dose" were severely diluted by participant retention problems in most intervention groups. Volume II presents the retention rates and length of stay in each group, uses graphic and statistical techniques (including discriminant analysis and survival analysis) to analyze why and when participants left, and draws programmatic implications for reducing premature exit in future programs. It synthesizes the midstream efforts to improve retention that were attempted by the projects and assesses the effects of these efforts in improving retention. Finally, it presents a number of lessons learned from these analyses about retaining homeless persons with alcohol and other drug problems in treatment programs.

Volume III: Cross-site Synthesis of Outcome Analyses. Volume III first presents initial assessments of both the magnitude of change over time and the relative effects of interventions, as assessed by standardized differences between groups in the amount of change that occurred in each of the key outcome domains. Next, the search for a better understanding of these results is carried forward--first by postulating a simple relationship between outcomes (both change over time and relative effects) and the strength of the implementation process. Additional analyses then focused on detecting and accounting for factors leading to departures from this postulated relationship, including: (1) measurement issues (regression toward the mean, measurement sensitivity, ceiling and floor effects, and measurement reliability); (2) design integrity (the adequacy of statistical adjustments for baseline nonequivalence, the effects of post-assignment nonequivalence caused by differential attrition from measurement); (3) project-level implementation problems and participant-level variations in service receipt (including changes in implementation across the project's timeline); and (4) the specific program theory underlying each project (including the nature of its target populations and its expectations for participant change).

Volume IV: Site-level Implementation and Retention Analyses. Volume IV presents a detailed project description and implementation analysis for each of the 14 Cooperative Agreement projects, and a participant retention analysis where applicable. As such, it provides the source material for the cross-site syntheses of implementation and retention presented in Volumes I and II, respectively. Key features include the establishment of a chronology of "key events" for each project (thereby providing a basis for estimating the potential impact of those events on participant retention and outcomes), and the development of "as proposed" and "as implemented" logic models.

R.O.W. Sciences developed and implemented a plan to disseminate the knowledge of alcohol and drug treatment programs for homeless persons that is being developed through the NIAAA demonstration. The strategy was intended to reach a broad audience, including researchers, service providers, and policymakers. The company is currently disseminating results through a number of different avenues, including presentations at professional meetings (American Psychological Association, American Evaluation Association, American Public Health Association), publication of specialized monographs in collaboration with grantees, and articles in professional journals (one article has been published in New Directions in Program Evaluation; nine others are under review or in preparation).

Table of Contents | Previous | Next