This Web site is a component of the SAMHSA Health Information Network

This Site Mental Health at HHS All Mental Health

About the Program Program Summaries Child and Adolescent     MentalHealth Programs Caring for Every Child's     Mental Health Systems of Care Resources For Children and Adolescents Resources For Parents and Families Related Topics Featured Publications In The News Related Links Child, Adolescent & Family Homepage  CMHS Programs:   Go to... Children's Campaign Community Support Emergency Services Financing HIV/AIDS Homelessness Managing Anxiety Mental Health Statistics Protection & Advocacy Special Programs State Planning Traumatic Events Suicide Prevention Programs Homepage  CMHS Activities:   Go to... 15+ School Violence Stigma Suicide Prevention Surgeon General

printer friendly page e-mail this page bookmark this page shopping cart  current or new account

This Web site is a component of the SAMHSA Health Information Network.

National Evaluation Team for the Comprehensive Community Mental Health Services Program for Children and Their Families

(ORC Macro)

Background

ORC Macro is a research, management consulting, and information technology firm supporting business and governments worldwide. It provides survey and market research, policy analysis and evaluation, performance improvement, training, and technology support.

In addition to providing funding to communities, through grants and cooperative agreements for children’s mental health services, the Center for Mental Health Services (CMHS) contracted with ORC Macro to conduct a national multi-site evaluation of the implementation of systems of care in these communities.

The evaluation has several goals:

• Describe the children and families served by the CMHS-funded systems of care.
• Determine the nature and extent of clinical and functional outcomes for children and families served in systems of care.
• Examine how children and families experience the service system and how they use services and supports (i.e., utilization patterns).
• Estimate the cost of serving children in systems of care.
• Examine the development of systems of care as they move toward offering integrated and comprehensive services.
• Assess the effectiveness of the system-of-care approach as compared to usual service delivery approaches.
• Assess the effectiveness of evidence-based treatments within systems of care.
• Support technical assistance activities to help CMHS best meet program goals.

Description of Activities

Study Evaluation

The budget of each funded community includes positions for two full-time local evaluators. These evaluators collect data describing children and families, services these families receive, and the outcomes of those services. The data are collected as specified by the guidelines of the National Evaluation.

As the National Evaluation Team, ORC Macro is responsible for training local evaluators in national evaluation data collection and submission procedures; analyzing the data; interpreting the results; producing annual reports to Congress and other written reports and products on the progress, results, and lessons learned from the implementation of system-of-care programs. In addition, continuous updates are given to CMHS on the status of the project, and funded communities are given individual reports with analyses of their data. Phase I of the evaluation began in 1994 and included the 22 communities that began their grant funding in 1993 and 1994. Phase II of the evaluation began in 1997 and included 9 communities that began their funding in 1997 and the 14 communities that received initial grant funding in 1998. Phase III of the evaluation includes the 22 communities that began their grant funding in 1999–2000. Phase IV of the evaluation includes the 18 communities that were awarded cooperative agreements in 2002, 7 communities that were awarded cooperative agreements in 2003, and four communities that were awarded cooperative agreements in 2004.

In addition, the National Evaluation Team collects system-level data that includes information on the development of interagency infrastructure, strength-based services, family involvement, and culturally competent services in each funded community. National Evaluation staff work with local communities to obtain service and cost data from billing and management information systems to examine expenditures for services and shifts in service costs. The Staff also conducts a number of special studies to assess thing such as the sustainability of systems of care, effectiveness of evidence-based practices, and the role primary care physicians play in systems of care.

Evaluation Design

The core evaluation design includes five components:

Descriptive Data Collection

The descriptive data profile children and families who enter systems of care. Descriptive data are collected on all children and include demographic characteristics, social/functional characteristics, and mental health diagnoses. The demographic data include characteristics such as age, ethnicity, sex, family income, and family living situation. The social/functional data include information such as educational performance, juvenile justice history, previous psychiatric problems, and abuse history. The distinctions and similarities among children based on their descriptive data may help explain variations in the treatment experiences of children and families.

Outcome Data Collection

The outcome data trace changes in children’s mental health across time. Outcome data are collected on a sample of children in each funded community at intake into the system of care and every 6 months thereafter. Outcome data include assessments of children’s emotional and behavioral status, strengths, educational performance, children’s living environments, families’ involvement in their children’s treatment, and child and family satisfaction with the services.

Cost and Service Data Collection

Cost and service data describe the types of services used by children and families in the system of care and the costs associated with these services. The evaluation collects individual data from community management information systems on children’s use of services and their costs in the system of care. The analyses of cost and service data highlight the mixture of services delivered by systems of care and the estimated average costs of treating a child.

System-of-Care Assessment Data Collection

System-of-care assessment data describe the organization of the child-serving system, how it delivers services, and how it affects the mental health outcomes of children and families. Every 18 months, site visits are conducted to each funded community to determine whether any developmental changes have taken place at the organizational and service delivery levels. A system-of-care community’s level of development is determined by measuring system attributes during every site visit through record reviews, interviews, and observations. Examples of the attributes include service availability, cultural competence of services, family involvement, and interagency structure.

Comparison and Treatment Effectiveness Studies

A significant aspect of the national evaluation is that it includes comparisons between those children and families served by CMHS-funded systems of care and those served by usual service delivery systems. The comparison study component of the national evaluation was initiated to address a critical question: “Can greater improvements in children’s behavior and functioning be attributed to the system-of-care approach?” To answer this question, three CMHS-funded system-of-care communities were matched in Phase I with three non-CMHS-funded communities that appeared to have made less progress toward adopting a system-of-care approach. Two additional CMHS-funded system-of-care communities from Phase II of the evaluation are being compared to matched communities without CMHS funding. Children are followed for up to 3 years, both during and after services are completed. Data are collected from caregivers, youth, providers, and service agencies.

A more recent addition to the evaluation is treatment effectiveness studies, which are designed to evaluate the effectiveness of evidence-based interventions within systems of care. In two CMHS-funded system-of-care communities from Phase II of the evaluation, some children receive Parent–Child Interaction Therapy, and some receive system-of-care services as usual. Children and families are followed for up to 18 months, both during and after services are completed. In two Phase III communities, some caregivers attend 6-week Common Sense Parenting® training, and children and caregivers are followed for up to 2 years. The treatment effectiveness study to be conducted in Phase IV communities is currently being designed and will be implemented over the next year.

The National Evaluation Team also coordinates or supports the implementation of the Family-driven Study, the Sustainability Study, the Culturally Competent Practices Study, the Primary Care Study, and an evaluation training curriculum for family members.

Partners in the National Evaluation

ORC Macro works closely with several partners to conduct the national evaluation of the CMHS-funded system-of-care programs. The Research and Training Center for Children’s Mental Health at the Florida Mental Health Institute, University of South Florida, collaborates actively by conducting System of Care Practice Review in selected communities, providing logic modeling supports to the program, analyzing national evaluation data, and consulting on other program evaluation activities. The Federation of Families for Children’s Mental Health is also an active partner who provides input on the process and outcomes of the program evaluation with a specific emphasis on how the evaluation results can be utilized at the individual community level to strengthen families. In addition, the Federation and ORC Macro are collaborating on the implementation of the Family-driven Study. The Research and Training Center on Family Support and Children’s Mental Health at Portland State University will evaluate and disseminate the Family-Evaluator Collaboration Training, a curriculum developed by the Federation of Families to help evaluators and family members improve their collaboration.  Faculty members from Duke University are assisting with implementation of the treatment effectiveness study that will be conducted in Phase IV communities. A Services Evaluation Committee composed of experts in children’s mental health services helps to guide the overall process of the evaluation by addressing key components within the context of regularly scheduled consultation meetings.

Evaluation Products

The national evaluation team is continuously involved in the dissemination of information regarding the program and its associated outcomes. Regular evaluation products include an annual report to Congress; monthly System-of-Care Evaluation Briefs on selected national evaluation topics; presentations at conferences and briefings to key policy-making audiences; journal articles and book chapters; and data reports to funded system-of-care communities that can be used for local-level dissemination and program development.

A liaison from the national team acts as the primary resource and point of contact for all system-of-care communities funded since 1997.  When specialized expertise in one of the components of the national evaluation is needed, however, liaisons work with the core staff of the component to provide technical assistance to the communities.