This Web site is a component of the SAMHSA Health Information Network
|
This Web site is a component of the SAMHSA Health Information Network |
| | | | | | |  |
||||||||||
 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This Web site is a component of the SAMHSA Health Information Network. |
|
2000 Annual Report to Congress on the Evaluation
|
|
CHAPTER SUMMARY In summary, descriptive and outcome data from the national evaluation of grant communities funded in 1993 and 1994 indicate that
Outcomes data to 2 years from the national evaluation of grant communities funded in 1993-94 indicate that:
Outcomes data from the national evaluation of grant communities funded in 1997-98 indicate that:
|
Descriptive and outcomes data collected on children and families who participated in services in grant communities funded in 1993 and 1994, and in 1997 and 1998 are presented in this chapter. Aggregate descriptive data for children and families served in communities funded from 1993 to 1998 are presented first. Then, 2-year outcomes for children and families served in communities funded in 1993 and 1994 and preliminary 6-month outcomes for children and families served in communities funded in 1997 and 1998 are presented. The descriptive data on children and families receiving services in the system-of-care programs supported by the Comprehensive Community Mental Health Services for Children and Their Families Program provide an overview of the characteristics of children and families served, including such information as demographics, diagnoses, risk factors, and behavioral, emotional and functional status. Assessing behavioral and functional outcomes contributes important information to understanding the effectiveness of services provided within a system of care for children with mental, emotional, and behavioral disorders. These and other outcomes data on school performance, involvement with the legal system, and stability of living arrangements are presented below. In addition, outcomes data on youth strengths and youth substance use, as well as family resources, family functioning, and caregiver strain are presented for children and families served in communities funded in 1997 and 1998.
The intended target population for the Center for Mental Health Services (CMHS) initiative is children aged birth to 21 years with serious emotional disturbance. Approximately 20 percent of all children across the nation may be diagnosed with a mental disorder using the criteria specified by the psychiatric community in the DSM-IV. However, it is generally recognized that a smaller subgroup, 6-8 percent of all children, present with a serious emotional disturbance (Friedman, Kutash, & Duchnowski, 1996) and display significant functional impairment, as demonstrated by a decreased ability to meet the challenges presented by day-to-day interactions in the school, the home, and the community. Children with serious emotional disorders are also at risk for out-of-home placement and involvement with multiple child-serving agencies.
In both the evaluation of communities funded in 1993 and 1994 (Phase I) and communities funded in 1997 and 1998 (Phase II), the primary purpose of the descriptive study is to inform about the children and families served across all system-of-care grant communities. All 22 communities funded in 1993-94 and 20 of the 23 communities funded in 1997-98 contributed data for the descriptive analyses presented in this chapter. As can be seen on the map in Chapter I ("Introduction") and also described in Chapter II ("System-of-Care Implementation"), communities varied considerably in geographic area. Some communities were localized in specific urban communities or counties, while others extended across a State or several counties within a State. Participation in the descriptive study in grant communities funded in 1993-94 was completed in 1999. Grant communities funded in 1997 began descriptive data collection in 1998; those funded in 1998 began descriptive data collection in either 1999 or spring 2000. Only data reported as of March 2000 are included in these analyses; consequently, some grant communities funded in 1998 did not have data to contribute, while others could contribute only limited data at that time.
Due to community-level differences, the number of children and families participating in the descriptive study sample varied across grant communities. Differences in the number of children and families served across system-of-care programs were influenced by community-level variations in definitions of target populations and eligibility for system-of-care services. Variability in the definition of target populations occurred across communities as different subgroups of children were identified to receive care, and different agencies took the lead in the development of programs. For example, some grant communities had a strong focus on children who were referred through the juvenile justice system, while others were primarily oriented toward enhancing mental health-school liaison relationships. Target populations also changed significantly across time in some grant communities. Services were developed throughout the grant period, with relationships between agencies continually cultivated and new service delivery options implemented as systems of care matured. Criteria for enrollment in services for children also became more clearly defined over time within each grant community.
Among grant communities funded in 1993-94, local decisionmaking affected collection and reporting of descriptive data, resulting in the absence of reporting of household composition, income, referral sources, and risk factors for children and families in several grant communities. The grant communities funded in 1997-98 are more consistent in the collection of the above data. Across all grant communities, the most complete descriptive information was obtained for child age, child gender, race or ethnicity, and diagnosis. Data completion rates for these four variables ranged from 94 percent to nearly 100 percent of the total population of children for whom data were provided to the national evaluation; therefore, findings from the descriptive sample for these variables are most representative of all children participating in the CMHS-funded system-of-care programs.
The primary purpose of the longitudinal outcomes study is to assess changes over time among children and families participating in system-of-care services. Child and family outcomes data collected in the evaluation of communities funded in 1993-94 and of those funded in 1997-98 include the child's clinical and social functioning, educational performance, restrictiveness of child's living arrangements, and child and family satisfaction. In the evaluation of communities funded in 1997-98, youth 11 years or older also reported on their delinquent behaviors and their history of substance use. Assessments of children's strengths, family functioning, family resources, and caregiver strain were also obtained. Standardized and non-standardized instruments common to the field of children's mental health services were used to collect these data.
In the evaluation of communities funded in 1993-94, outcomes were examined through a simple pretest/post-test replacement design that requested grant communities to evaluate more fully a selected group of children at intake, at 6 months, at 1 year, and annually thereafter for as long as they remained in services. When children and families exited the evaluation or were lost to follow-up, they were replaced with a new family who had just entered system-of-care services. Grant communities were also requested to collect outcomes data from children and families when they exited services. This design allowed for following children over time and examining outcomes of those enrolled in the evaluation who continued to participate in services, although it did not provide a controlled test of the system-of-care approach versus other approaches or no services at all.
Enrollment in the outcomes study sample varied across grant communities. In total, 21 of the 22 grant communities funded in 1993-94 provided data for the outcomes study. According to enrollment protocols, grant communities serving fewer than 400 children were to enroll all interested families in the outcomes study. If more families were served by the system of care, families were to be randomly selected for participation in the outcomes study. For those grant communities serving between 400 and 4,000 children, a sample of 400 children was drawn. For those grant communities serving over 4,000 children, a 10 percent sample was drawn.
In the evaluation of grant communities funded in 1997-98, the design was changed so that children are tracked every 6 months, regardless of whether they are still in the services. This allows comparison of children's clinical and functional outcomes for all children regardless of whether they continue in system-of-care programs. Each grant community is to enroll approximately 284 families in the evaluation during Years 2 through 4 of funding, although this figure may vary slightly for communities funded to serve smaller numbers of children (e.g., funding in some communities may be directed primarily toward infrastructure development, or the number of children meeting service criteria for serious emotional disturbance may be lower). While in most grant communities all willing families need to be recruited into the outcomes study, in some larger communities sampling strategies may need to be employed to randomly select a sufficient number of families from the pool of children who enter the system of care. At these sites, a systematic sampling approach will be implemented.
In addition to change in the evaluation design, new instruments were added in the evaluation of grant communities funded in 1997-98 to capture information on family outcomes, child strengths, and youth self-report of delinquency and substance use. These new pieces of information provide a more comprehensive understanding of the functioning and outcomes of children served in system-of-care programs.
Data Collected in the Evaluation of Grant Communities
|
Information presented in this chapter regarding longitudinal outcomes should be interpreted with caution due to the methodological challenges encountered in evaluating grant communities funded in 1993-94. Because children were followed only as long as they remained in services, little is known about outcomes for children who exited services prior to follow-up data collection points. Given this limitation in the evaluation design and lower-than-expected completion rates for longitudinal data collection across grant communities, generalizability of outcomes analyses from the evaluation of these communities may be limited to a select group of children with serious emotional disorders who remained in services in systems of care for relatively long periods of time. However, it is assumed that the samples of children who remained in services for 6 months, 1 year, and 2 years are relatively representative of the population of children who received system-of-care services for extended periods. Analyses of these children and families should provide information about the contribution that systems of care make to the well-being of children with serious emotional disturbance and their families.
Factors influencing outcomes study enrollment and completion rates at follow-up were in part characteristic of the communities in which services were delivered and the difficult family situations of the children and families served. Grant communities may have had lower numbers of children in the outcomes study due to geographical characteristics (e.g., serving large rural areas) and high mobility rates of families. Other factors influencing enrollment in the outcomes study and variable data completion rates included families who declined participation and differential allocation of evaluation personnel in each grant community to collect outcomes information.
To address methodological limitations in the outcome study conducted among grant communities funded in 1993-94, the design of the evaluation was changed in the evaluation of grant communities funded in 1997-98 to follow children whether or not they remained in services. In addition, comparison studies added to both the evaluation of grant communities funded in 1993-94 and communities funded in 1997 introduced a control group to the program evaluation design. These comparison studies in three grant communities funded in 1993-94 and two funded in 1997 and their matched comparison communities are yielding useful comparison information (see Chapter IV, "Longitudinal Comparison and Evidence-Based Practices in Systems of Care Studies").
For reporting and analysis purposes, descriptive and outcomes data collected at intake were aggregated across all grant communities. These data provide a reasonably accurate description of the characteristics of children and families entering federally funded systems of care since their inception in 1993. Analyses of longitudinal outcomes data are presented separately for the children and families enrolled in the evaluation in communities funded in 1993-94 and those in the evaluation of communities funded in 1997-98 because these communities are at different points in their follow-up data collection. Grant communities funded in 1997-98 are just beginning to collect follow-up data. Consequently, data from these communities are relatively limited, constraining outcomes analyses to changes from intake to 6 months. In addition to the preliminary aggregate outcomes analysis conducted for communities funded in 1997-98, some outcomes were also selected to illustrate site-level differences and similarities.
Extensive information collected at intake, including data on clinical and functional status, living arrangements, school performance, and juvenile justice involvement, was received from a sample of 19,677 children. Smaller samples were available for analyses for subsequent outcomes at 6 months (8,790), 1 year (6,593), and 2 years (2,983). These sample sizes at follow-up reflect cases for which at least one measure administered at baseline was also obtained at the subsequent data collection point. Because the evaluation design only included those children who remained in services at follow-up, it was expected that as families left services the sample size would decline. Because families could enter services and consequently the evaluation at any point during the years when services were delivered through the grant program in the community, families entering services at a later point in time could only be followed for a correspondingly shorter follow-up period. In addition, the sampling with replacement design meant that as families left services, new families were added, meaning that more families were always added at baseline. The advantages to using a sampling with replacement design for this study were (a) information about children remaining in services for longer periods of time could be maximized, and (b) given the limited resources available to communities for evaluation, the evaluation was circumscribed by the restriction to children in services. (See Appendix for detailed information about enrollment and follow-up in each grant community.) Overall, the demographic characteristics of children and families enrolled in the longitudinal outcomes study at 2 years were similar to those included in the larger descriptive sample, with slightly more males participating in the outcomes study at intake (70 percent) and at 2 years (71 percent) than in the larger descriptive study (62 percent). The average age was lower in the outcomes study sample and decreased across time from 12 years at intake to 11 years at 2 years. A relatively higher proportion of African-American children was involved in the outcomes study at intake (22 percent) than in the descriptive study (15 percent), but the percentage decreased to 15 percent at 2 years. There was a corresponding lower percentage of Hispanic children in the outcomes study sample at intake (14 percent) and at 2 years (16 percent) than in the descriptive study sample (25 percent).
Extensive information collected at intake, including data on clinical and functional status, living arrangements, school performance, and juvenile justice involvement, was received from a sample of 775 children. Smaller samples were available for analyses for subsequent outcomes at 6 months (323). Similar to the evaluation of grant communities funded in 1993-94, the sample size at 6 months reflected cases for which at least one measure administered at baseline was also obtained at the subsequent data collection point. Demographic characteristics of children and families participating in the outcomes study at intake and 6 months are largely similar to those enrolled in the descriptive study. Males comprised approximately 69 percent of the sample at intake and 74 percent at 6 months, in comparison to 67 percent in the descriptive study sample. The average age of the children is slightly higher in the outcomes study sample at intake (12.4 years) than in the descriptive study sample (12.2 years) and decreases at 6 months to 12.2 years. A somewhat smaller proportion of African-American children was involved in the outcomes study at intake (18 percent) and at 6 months (15 percent) than in the descriptive study sample (30 percent); conversely, there was a higher percentage of Hispanic children at intake (15 percent) and at 6 months (17 percent) than in the descriptive study sample (3 percent).
As of March 2000, a total of 42,221 children have been enrolled in the national evaluation of the system-of-care programs across the 45 grant communities funded from 1993 to 1998. Given that evaluation data collection is now complete in grant communities funded in 1993-94, the vast majority (96.8 percent) of these data are from children enrolled in the evaluation in these grant communities. Although additional questions are addressed in the evaluation of communities funded in 1997-98, information about demographic and baseline clinical and functional characteristics of the children served by grant communities' system-of-care programs is collected for all evaluation participants.
Table 5 presents the demographic characteristics of the children and families participating in the descriptive and outcomes studies of the national evaluation. Information is presented in aggregate from the 42 funded programs and separately for Phase I and Phase II. The discussion of demographic characteristics that follows focuses on data collected in the 42 grant communities funded since 1993.
Demographic characteristics of the children in the descriptive sample were consistent with proportions reported in children's mental health literature (Berndt et al., 1995; Cullinan et al., 1992; Landrum et al., 1995; Rosenblatt & Attkisson, 1993; Santiago et al., 1997; Silver et al., 1993).
Across all grant communities, the age of the children ranged from less than 1 year to 23 years of age The mean age of the children was 12.1 years and did not differ among children served in communities funded in 1993-94 (12.1 years) and those funded in 1997-98 (12.2 years). Slightly less than two-thirds of the children were male in the aggregated sample; however, more males are represented in the sample of children served in communities funded in 1997-98 (69.7 percent) than in the sample from communities funded in 1993-94 (61.9 percent). Among all children, slightly more than half were White, about 24 percent were Hispanic, and about 14 percent were African-American, as was the case among communities funded in 1993-94. Among communities funded in 1997-98, more African-American (22.6 percent) and American Indian (15.6 percent) children and fewer Hispanic (9.3 percent) and Asian (0.4 percent) children were represented. These changes, particularly in the percentage of Hispanic and American Indian children represented, reflect differences in the populations of the communities funded in 1997-98. Custody status of the children did not differ greatly across these two sets of grant communities, although a slightly higher percentage of children in the evaluation of communities funded in 1997-98 (12.8 percent) are in the custody of a guardian than in the evaluation of communities funded in 1993-94 (7.1 percent). Family income was lower among families served in communities funded in 1993-94; 60.4 percent of families had incomes below $15,000, compared to only 49.1 percent of families in communities funded in 1997-98. This difference may reflect, in part, economic changes from 1994 (when enrollment into the evaluation began in grant communities funded in 1993-94) to 1998 (when enrollment into the evaluation began in communities funded in 1997-98). From 1994 to 1998, the poverty threshold for a family of four with two children increased from $15,029 to $16,530 (U.S. Census Bureau, 1999a, 1999e).
According to the 2000 U.S. Department of Health and Human Services poverty guidelines, a family of four with two children is living in poverty if their income is below $17,050. National figures indicate that 15.1 percent of all families with children under age 18 lived in poverty in 1998 (U.S. Census Bureau, 1999b) and 18.9 percent of all children under age 18 lived in poverty in 1998 (U.S. Census Bureau, 1999d). After accounting for family size, 67.9 percent of the families in the descriptive sample reported gross annual incomes below the year 2000 poverty threshold ($17,050). This higher figure of families living in poverty in the descriptive sample reflects, in part, a higher percentage of mother-maintained households. Upon entry into system-of-care programs, about 34 percent of children lived with their mother only. By comparison, only about 24 percent of children nationwide lived in single-parent homes headed by their mothers (OJJDP, 1999).
Children under age 18 in mother-maintained households are more likely to live in poverty (38.7 percent) than children in households with married parents (6.9 percent; U.S. Census Bureau, 1999b). This relationship between mother-maintained households and poverty emerged in the descriptive sample. Figure 7 illustrates the poverty levels for two-parent, mother-maintained, and father-maintained households among the grant communities funded in 1993-94 and 1997. Mother-maintained households in the descriptive sample were about twice as likely to fall below the poverty threshold than the national average.
Among families in which a single mother maintained custody over the children, 76 percent earned annual incomes below the poverty level. In comparison, among families in which both biological parents maintained custody over the children, only 56 percent earned annual incomes below the poverty level (χ2 = 299.328, df = 4, p < .001). These figures closely reflect the national figure in 1998 that indicated that 53.3 percent of all poor families were maintained by female householders. (U.S. Census Bureau, 1999c). Employment opportunities for single-parent households that are challenged by a child's serious emotional disturbance are likely to be limited (Brennan & Poertner, 1997) and may extend the duration of poverty, thus adding to overall family stress.
National figures indicate that income and race or ethnicity are related. In 1998, for example, 9.1 percent of White families with children lived in poverty. By comparison, 30.5 percent of African-American families with children and 28.6 percent of Hispanic families with children lived in poverty (U.S. Census Bureau, 1999b). This relationship between race or ethnicity and income levels emerged in the descriptive sample at a greater magnitude. For example, while 61.3 percent of White families earned annual incomes below the poverty level, 85.5 percent of Native Hawaiian families, 82.4 percent of Hispanic families, 80.0 percent of African-American families, and 80.1 percent of American Indian families earned annual incomes below the poverty level (χ2 = 359.697, df = 12, p < .001). Moreover, a similar relationship between race or ethnicity and income remains when family structure is held constant (see Figure 8). Among White children in two-parent homes, 46.7 percent lived in families that earned gross annual incomes below the poverty level. In comparison, 67.7 percent of African-American children in two-parent households and 83.5 percent of Hispanic children in two-parent households lived in families that earned gross annual incomes below the poverty level. Consequently, even after taking family structure into account, families of minority background experienced more poverty than their White counterparts. While the system-of-care approach may be an effective mental health service model for children of diverse backgrounds, continued economic hardship for these families may constrain national mental health outcomes. Because such a large percentage of families receiving publicly supported mental health services are poor, the impact of economic circumstances on outcomes and on group differences is a particularly important concern.
Minority children were no more likely to be referred to services from external sources such as schools and mental health agencies, even though they were overrepresented among those receiving services when compared to population statistics. |
Children entered systems of care through a variety of avenues. Mental health agencies (21.9 percent) and schools (20.3 percent) provided the largest proportion of referrals. Slightly more than 14 percent of the children were directed to systems of care through social service agencies, and another 14 percent were referred by the courts and correctional institutions. In addition, 16.2 percent of the children were referred to system-of-care services by their parents or themselves. Variability in referral sources across grant communities may be explained by procedures specific to each grant community that dictate the routes necessary for access to services. For example, in the Milwaukee grant community funded in 1994 (Wraparound Milwaukee), a shift in emphasis to youth involved with the juvenile justice system yielded a substantially higher proportion of referrals from the courts and corrections. Figure 9 illustrates the proportion of children referred for services by each source among grant communities funded from 1993-98.
There was a significant difference in the racial or ethnic characteristics of the children referred by their parents or the youth themselves (i.e., self-referral) and those referred by external sources such as schools or mental health agencies (i.e., external referral; χ2 = 441.087, df = 6, p < .001). In the self-referral group, 69.5 percent of the children were White, 13.9 percent were African-American, and 6.8 percent were Hispanic. In contrast, 52.9 percent of the children in the external referral group were White, 25.3 percent were Hispanic, and 14.3 percent were African-American. Although external referrals were in proportion to the ethnic distribution of children in the overall sample, White families were over-represented and Hispanic families were under-represented among those who referred themselves to services. Even though minority children were over-represented among those receiving services when compared to population statistics (U.S. Census Bureau, 2000), they were no more likely to be referred to services from external sources such as schools, mental health, social service, and juvenile justice agencies. This contrasts with the literature which indicates that children and youth of color are more likely to be referred for services by external agencies than are their White counterparts (Takeuchi, Bui, & Kim, 1993).
Certain child and family risk factors are often associated with serious emotional disturbance. Childhood exposure to violence, sexual abuse, physical abuse and neglect, and substance abuse are linked with emotional and psychological distress in the literature (McLoyd, 1991; McLoyd & Steinberg, 1998; Patterson, 1988). Family characteristics such as a history of mental health disorders, family violence, and felony convictions are also believed to be associated with child and adolescent emotional disturbance (Friedman et al., 1996).
Sixty-seven percent of the children in the descriptive sample were described by their caregivers as having experienced one or more child risk factors, including physical abuse, sexual abuse, previous psychiatric hospitalization, sexual abusiveness, suicide attempt, drug and alcohol use, and a history of running away. Figure 10 presents the percentage of children with each of these risk factors at entry into system-of-care services. Figure 11 illustrates the proportion of children who experienced multiple child risk factors; 19.3 percent had experienced two such risk factors, and another 21.2 percent had experienced three or more risk factors.
Overall, among communities funded from 1993-98, about 84 percent of families reported the presence of one or more family risk factors, including family history of mental illness, previous psychiatric hospitalization, felony convictions, family history of violence, and family history of substance abuse. As displayed in Figure 12, family history of substance abuse, violence, and mental illness were the most frequently reported family risk factors. As with child risk factors, many caregivers reported multiple family risk factors: 21.1 percent reported two family risk factors and 42.2 percent reported three or more family risk factors, indicating high rates of multiple family risk factors in children's histories (see Figure 13).
These high proportions of multiple child and family risk factors reflect the highly stressed and burdened environments of children and families participating in services through systems of care.
Diagnostic information was collected at each child's entry into the system of care based on criteria from the DSM-IV. Because no standard procedures were established at local grant communities to collect diagnostic information, it can be assumed that the majority of the diagnoses were obtained based on clinicians' knowledge of the children being served in the programs and extracted from review of children's clinical records.
Of the 35,625 children in grant communities funded from 1993 to 1998 for whom a primary diagnosis was reported, 43.2 percent displayed a disruptive behavior disorder (29.4 percent were conduct-related disorders and 13.8 percent were attention-deficit/hyperactivity disorder, or ADHD). Over 25 percent of children were diagnosed with depression, 7.9 percent with an anxiety disorder, and 6.3 percent with an adjustment disorder. The remaining primary diagnostic categories (assigned to approximately 16.8 percent of the sample) included, but were not limited to, substance abuse, eating, somatic, and speech disorders, as well as enuresis, encopresis, abuse/neglect, personality disorders, and learning disabilities (see Figure 14).
Of the children with a valid primary diagnosis, that is, a diagnosis as specified on the DSM-IV and not the same as the secondary diagnosis, 29.2 percent also had a secondary diagnosis. Among these children, 18.2 percent had a secondary diagnosis of a conduct-related disorder and 16.8 percent had a secondary diagnosis of substance abuse disorder (see Figure 15). These findings are consistent with recent research among clinical samples that indicates that adolescents with behavioral and substance abuse disorders tend to have other psychiatric disorders as well (Grilo, Becker, Fehon, Edell, & McGlashan, 1996; Grilo, Walker, Becker, Edell, & McGlashan 1997; Wilens, Biederman, Abrantes, & Spencer, 1997). In total, children with a secondary diagnosis of substance abuse disorders represented about 5 percent of all children with a primary diagnosis in this sample. Assignment of a secondary diagnosis of any type was lower for children aged 5 or younger (16.6 percent) than for children aged 6-11 years (28.9 percent), aged 12-15 years (29.9 percent), and aged 16 years or older (32.7 percent; χ2 = 258.024, df = 3, p < .001).
The majority of the children stayed in one living arrangement in the 6 months prior to entry into system-of-care services. |
One way to gauge children's social well-being is to examine the stability of their living arrangements over time. Living arrangements were assessed for the year prior to intake into services in the evaluation of grant communities funded in 1993-94, and for the 6 months prior to intake in the evaluation of grant communities funded in 1997-98. As shown in Figure 16, among children enrolled in the outcomes study in communities funded in 1993-94, 57.5 percent of the children were reported to have lived in only one living arrangement, 25.5 percent had lived in two living arrangements, and 17 percent had lived in three or more living arrangements in the 12 months prior to entering system-of-care services. In grant communities funded in 1997-98, at intake, 56 percent of the children had lived in just one living arrangement in the previous 6 months, 18.9 percent were reported to have lived in two living arrangements, and 25.7 percent were reported to have lived in three or more living arrangements. Among all children enrolled in the outcomes study from communities funded from 1993 to 1998, at intake 40.9 percent of the children lived only with their biological mother, 4.6 percent lived only with their biological father, and 26.9 percent lived with both biological parents. Another 9.4 percent of the children lived with a relative other than a biological parent, and the remaining 18.2 percent lived with someone other than a biological relative (including arrangements such as foster care, group homes, residential treatment or drug/alcohol rehabilitation centers, adoptive homes, and youth correctional facilities).
Among grant communities funded in 1993-94 and 1997-98, almost all children in the outcomes sample at intake were of school age. Approximately 40 percent of children were in elementary school (grades 1 to 6) at intake, 25.3 percent were in middle school (grades 7 and 8) and 35.2 percent were in high school. About 72 percent of the children were reported to attend school on a regular basis (more than 75 percent of the time), while 16.2 percent attend infrequently (less than 50 percent of the time). Although 54 percent of the children were reported to earn average (37.6 percent) or above average grades (16.4 percent), about 30 percent were earning below average grades and almost 16 percent were reportedly failing in school. School performance was highest among those who attended school regularly (χ2 = 1309.995, df = 9, p < .001). At intake, 42.3 percent of the children received their education in regular classrooms with no special education supports. Nearly 17 percent did, however, have assistance from a classroom aide in addition to the teacher and close to 20 percent were placed in special education classrooms.
Difficulties that children experience in other areas of their lives can have a strong impact on their success in school. School performance was highest among those children who had experienced the fewest child risk factors. As shown in Figure 17, those children who had three or more child risk factors at intake were more than one and one-half times as likely to earn failing grades than those children with no child risk factors. No differences in school performance by number of family risk factors were found.
At intake, 27 percent of the children in grant communities funded in 1997-98 received their education in regular classrooms with no special education supports, 23 percent were placed in some type of special education class within a regular school, and an additional 9 percent received other kinds of special education services (e.g., HeadStart, homework club, home teaching by public school teachers). Of the children who were placed in special education classrooms at intake, within 6 months of entering a system of care, 20.5 percent were transferred either to regular classrooms or to regular classrooms with special education supports.
Forty-eight percent of the children enrolled in the Phase II outcomes study had an IEP at intake into system-of-care services. Children in grades five through eight (middle school) were more likely to have an IEP than any other group in the sample. Fifty-three percent of the children had IEPs that reflected behavioral or emotional disturbance. Thirty-five percent of the children had IEPs for challenges due to learning disabilities, and the remaining 12 percent had IEPs for physical disabilities and various other reasons. In addition, about one-third of the children with an IEP attended a special education class for up to 99 percent of their school day. An additional 13.4 percent received special education consultation, and 24.3 percent were placed in alternative or residential schools (see Figure 18). Other children with an IEP were placed in a variety of educational settings such as hospitals, home schools, and special public schools.
In the evaluation of grant communities funded in 1997-98, youth 11 years or older reported on their own delinquent behaviors and contacts with police or law enforcement. Figure 19 displays the nine most frequently reported delinquent activities in which youth engaged at least once in the 6 months prior to intake. Preliminary analyses reveal that there are relationships among delinquent behaviors. For example, youth who had carried a weapon in the prior 6 months were over three times as likely to have been in a gang than youth who had not carried a weapon before their enrollment in a system of care (31 percent vs. 10 percent, χ2 = 25.102, df = 1, p = .000).
Prior to their entry into system-of-care programs, 62 percent of 342 youth aged 11 years or older reported that they had been accused by the police of breaking the law and 50 percent of 338 youth reported that they had been arrested for offenses such as assault, driving under the influence of substances (DUI), criminal mischief, and vandalism. Fifty-four percent of 332 youth reported that they had been on probation. Forty-three percent of 328 youth reported having been in a detention center or jail, 70 percent of whom were in high school, and 84 percent reported a history of substance abuse.
Youth 11 years or older in grant communities funded in 1997-98 also reported on their own substance use. Fifty-eight percent of 345 youth had used alcohol, 67 percent had smoked cigarettes, and 50 percent had used marijuana prior to intake. Of the 190 youth (56 percent) who reported using at least one substance prior to intake, 42 percent had used two to four different substances, and 8 percent had used five or more substances in their lifetime.
When asked about alcohol and substance use in the 30 days prior to entry into system-of-care programs, 41 percent of 114 youth between the ages of 12 and 16 reported consumption of alcohol and 65 percent of 55 youth reported use of marijuana. These percentages are substantially higher than those reported for the same age group in the 1997 National Longitudinal Survey of Youth (Bureau of Labor Statistics, 1998; see Figure 20).
Although there were no significant differences between boys and girls in their history of marijuana and alcohol use, the proportion of youth who had used marijuana and alcohol in their lifetime was dramatically larger by age. Youth who were 16 years or older were three times as likely to have used marijuana or alcohol than those between the ages of 11 and 13 (see Figure 21).
Fifty-seven percent of 148 youth 11 years or older indicated that they had at least once used alcohol or drugs at least once on a school day or before work, and 32 percent of the youth reported that they had missed school or work as a result of alcohol or drug use. However, there was no significant difference in school performance (e.g., failing, grade average C, grade average B) between those who had ever used alcohol and those who had not (χ2 = 5.943, df = 6, p = .430).
Analysis of youth self-reported delinquent behaviors and substance use indicated that there is an association between drug use or sale and other problem behaviors such as alcohol consumption and weapon possession. Youth who had used marijuana in the 6 months prior to intake were more likely to have used alcohol, carried a weapon, or sold drugs or helped others sell drugs than youth who had not used marijuana in the same period (see Figure 22). Interrelationships of problem behaviors such as these among adolescents are well documented in the literature (Jessor, 1998).
In summary, a high percentage of children participating in services in systems of care presented with disruptive behavior disorders, although a reasonably large proportion were depressed or anxious. Multiple living arrangements and individual family risk factors characterized children's histories prior to entering systems of care, and child experiences of risk were related to low school performance at intake. Youth self-reports of delinquency and substance use indicated that many of these children had troubles with the law and various substance abuse problems prior to entry into systems of care.
Behavior, emotions, and level of functioning were assessed at intake using the Child and Adolescent Functional Assessment Scale (CAFAS; Hodges, 1990b), the Child Behavior Checklist (CBCL; Achenbach, 1991), and the Youth Self-Report (YSR, completed by children 11 years of age and older; Achenbach & Edelbrock, 1987). Repeated assessments at 6 months, 1 year, and 2 years provided an opportunity to evaluate the degree of change in functional impairment, behavior, and emotions for children in system-of-care programs. It should be noted that analyses of change across time are based on fewer children due to both completion of services and dropout. For each of these measures, higher scores indicate greater impairment or greater emotional and behavioral problems. Declining scores over time represent positive change. The analyses of clinical change to 2 years focus on groups of children and families from the outcomes study sample who had complete data at four evaluation points: intake, 6 months, 1 year, and 2 years.
The evaluation of grant communities funded in 1997-98 added the Behavioral and Emotional Rating Scale (BERS), a standardized, norm-referenced scale, to assess the behavioral and emotional strengths of children. The developers of the scale, Epstein and Sharma (1998), described strengths-based assessment as "the measurement of those emotional and behavioral skills, competencies, and characteristics that create a sense of personal accomplishment; contribute to satisfying relationships with family members, peers, and adults; enhance one's ability to deal with adversity and stress; and promote one's personal, social, and academic development."
Functional impairment refers to the degree of disruption in a youth's current functioning without regard to past history, known causes, or prognoses related to the youth's substance use or mental health problems (Hodges, 1990a). In the study of children's mental health services, functional impairment has become an important concept, particularly because diagnostic rates may vary due to a wide range of factors, and emotional and behavioral symptoms may not truly reflect the extent to which a serious emotional disturbance affects a child's life. Measuring functional impairment allows assessment of the extent to which functioning in multiple life domains is affected. Changes in functional impairment across time as a result of participation in a system of care can also be evaluated.
The CAFAS, designed for use with children ranging in age from 7 to 17.5, assesses the degree of a child's or adolescent's psychosocial functioning across five domains: (a) role performance in the community, school/work, and home; (b) thinking; (c) behavior toward others; (d) moods and emotions and self-harmful behavior; and (e) substance abuse. In most cases, a clinician who obtained information from multiple informants such as the child, the caregivers, schools, and official records completed the CAFAS. In a small number of communities, information to rate the child's functioning was obtained through a structured interview with a caregiver.
In previous reports from the national evaluation (CMHS, 1998, 1999), the CAFAS total score was calculated based on the five CAFAS domains. More recently, the scale's author recommended calculating a total score by summing all eight subscale scores, resulting in total scores ranging from 0 to 240 (Hodges, 1997). CAFAS total scores of 40 or below indicate minimal impairment, scores from 50 to 90 indicate moderate impairment, scores from 100 to 130 indicate marked impairment, and those 140 or higher indicate severe impairment.
Children entered system-of-care services with higher levels of impairment in school role, behavior toward others, and home role than in other areas such as community role and substance use. |
For children enrolled in the outcomes study, the average total CAFAS score at intake was 92.8, significantly higher than the clinical cutoff score. Figure 23 displays the percentage of children reported to have moderate or severe impairment in each domain at intake. Two-thirds of children received school/work role scores in the moderate or severe range. These children had the greatest difficulties at school as evidenced by being absent frequently from school, having low grade averages, being a threat or harmful to teachers or peers, or dropping out of school. Over 50 percent of children had moderate or severe scores for their functioning at home and in their behavior toward others. The home scale assesses whether a child had problems such as running away from home overnight, repeatedly behaving irresponsibly in the home, and failing to comply with rules. In recent investigations comparing this population to other populations of children with mental health needs, scores at intake were higher, indicating a more severely challenged population among the children served in the CMHS grant communities (Bickman et al., 1995; Bickman, Summerfelt, Firth, et al., 1997).
All CAFAS subscale scores at intake differed significantly according to a child's gender, age, history of risk factors, and diagnoses. For example, boys exhibited more severe functional impairment in their role performance in each domain, particularly at school and in the community (see Figure 24).
Likewise, older children were rated as more functionally impaired in their role performance than were younger children (see Figure 25). Community role performance scores were considerably higher for older children; children over the age of 15 had mean community role performance scores that were three times higher than mean scores of children between the ages of 5 and 11 years. This difference is, however, expected since CAFAS scores are to some extent affected by the age of the child. The Community role scale assesses behaviors such as stealing, robbery, intentionally playing with fire, and damage to community property. Older children have greater access to the larger community in which these behaviors are enacted and therefore may be more likely to engage in these behaviors than younger children.
Children's diagnoses and experiences with risk factors were related to their average CAFAS scores at intake (see Figures 26 and 27). Those children with no child or family risk factors at intake had significantly lower average total CAFAS scores than those with three or more risk factors (see Figure 26). In addition, certain primary diagnoses were related to higher CAFAS scores at intake (see Figure 27). For example, children with a primary diagnosis of depression or conduct-related disorder at intake had average total CAFAS scores in the severe range. In contrast, those children with a primary diagnosis of ADHD and or anxiety had average total CAFAS scores in the moderate range.
The Child Behavior Checklist (CBCL) has been identified as the most reliable and valid parent report measure currently available for assessing children's emotional and behavioral problems (Reitman, Hummel, Franz, & Gross, 1998). The child's primary caregiver directly reports information concerning the child's behavior during the previous 6 months. The CBCL consists of 118 problem behavior items categorized as internalizing or externalizing behaviors. Internalizing behaviors include sadness, depression, and anxiety. Externalizing behaviors include oppositional behavior, aggression, and hyperactivity. In addition, 20 social competence items are used to measure school performance, functioning at home, peer relationships, and social involvement. Youth aged 11 or older assess their own emotional and behavioral problems and competencies with the YSR, which is similar in format to the CBCL. The YSR assesses the same behavioral and emotional domains as the CBCL, with 112 items, and has the same scoring criteria, range of values, and clinical cutoff scores. However, YSR scores tend to be somewhat lower than those yielded by the CBCL. Both CBCL and YSR have been nationally normed on a proportionally representative sample of children across income and racial-ethnic groups. Achenbach (1991) has reported high internal consistency, test-retest reliability, construct validity, and criterion-related validity for both the CBCL and the YSR.
Generally, children in the outcomes study sample were rated as exhibiting more externalizing behavior problems than internalizing behavior problems. At intake, 51.5 percent of the children scored within the clinical range for internalizing behaviors, 65.7 percent attained scores in the clinical range for externalizing behaviors, and 69.0 percent of the children attained scores in the clinical range on the CBCL Total Problems scale. With the exceptions of somatic complaints and withdrawal (both internalizing behaviors), average syndrome scale scores on the CBCL at intake were all above the clinical cutoff score of 63. Average syndrome scores for aggressive and delinquent behaviors (both externalizing behaviors) were the two highest at intake. As previously discussed, disruptive behavior disorders accounted for 43 percent of the primary diagnoses at intake for the descriptive study sample. These diagnoses appear to be supported by the CBCL intake data.
According to both caregiver reports (CBCL) and youth self-reports (YSR), girls consistently scored higher (i.e., had more problems) in all three scores of Internalizing, Externalizing, and Total Problems. All differences are statistically significant. Table 6 details the results.
Child and family risk factors were associated with clinical scores on the CBCL and YSR. Children with multiple child or family risk factors were rated as having more serious problems than children with one or no risk factors ( see Figures 28 and 29).
As displayed in Figure 30, average total CBCL scores differed by age groups at intake. Children between the ages of 6 and 11 were rated as exhibiting the most serious behavioral and emotional problems. Children's self-reported problems on the YSR generally paralleled those reported by their caregivers when examined by age group (see Figure 31); children younger than 15 reported more serious problems.
Overall, girls displayed slightly higher total competence scores. |
In addition to children's behavior problems, primary caregivers also reported on children's social competence. Social competence scores on the CBCL represent the child's ability to function successfully in various social arenas, namely, in sports or work activities, in social organizations, and at school. Boys exhibited greater competence in sports and work activities (t = 6.769, df = 11.079, p < .001), while girls obtained higher scores for social and school activities (Social: t = -6.244, df = 10,331, p < .001; School: t = -4.197, df = 9,352, p < .001). Overall, girls displayed slightly higher total competence scores, with an average of 35.3 compared to an average of 34.9 for boys. Both of these scores were below the clinical cutoff of 37 for the CBCL total competence score, indicating low levels of social competence.
Although traditional approaches to assessment have focused more on deficits and problems, the importance of a strengths-based approach has gained attention. A strengths-based approach "represents a significant departure in how children are viewed, assessed, and treated" (Epstein & Sharma, 1998). It is important to acknowledge that all children, including those with serious emotional disturbance, have strengths despite the problems and deficits they may have. Providers working with the children and their families should recognize these strengths to provide the most appropriate services.
The BERS is designed to be completed by caregivers or professionals (i.e., teachers) to rate the behaviors of children aged 5 to 18. It consists of 52 items that can be categorized into the following five domains: Interpersonal Strengths, Family Involvement, Intrapersonal Strengths, School Functioning, and Affective Strength. Standard scores for the five domains and an overall strength quotient can be derived to indicate the level of a child's strengths. For children with emotional and behavioral disorders, standard scores between 8 and 12 and overall strengths quotients between 90 and 110 are considered to be average for children with emotional and behavioral disorders.
Baseline analyses based on caregiver reports indicated that children entering the system-of-care programs had average strengths in all five domains. The baseline standard scores for the 432 children in the sample were 6.7 (school functioning), 6.8 (interpersonal strength), 7.5 (family involvement), 9.0 (affective strength), and 8.2 (intrapersonal strength). The highest scores in the intrapersonal and affective domains indicate that the children were rated highly in their ability to identify personal strengths, request support from peers and friends, and be enthusiastic about life. There were significant differences in the standard scores obtained by boys and girls in all five domains, suggesting that boys had a higher level of strength in all areas measured by the BERS (School functioning: 7.0 for boys vs. 5.9 for girls, Independent t = 3.947, df = 430, p = .000; Interpersonal strength: 7.2 for boys vs. 5.9 for girls, Independent t = 5.034, df = 430, p = .000; Family involvement: 7.8 for boys vs. 6.7 for girls, Independent t = 3.939, df = 430, p = .000; Affective strength: 9.5 for boys vs. 7.8 for girls, Independent t = 5.359, df = 430, p = .000; Intrapersonal strength: 8.6 for boys vs. 7.2 for girls, Independent t = 4.519, df = 430, p = .000 ). The overall strength quotient at baseline was 83.7, with boys scoring significantly higher (86.3 vs. 77.2 for girls, Independent t = 5.815, df = 431, p = .000).
In the evaluation of grant communities funded in 1997-98, three questionnaires are added to assess family resources, family functioning, and caregiver strain. These standardized questionnaires are administered at intake and every 6 months thereafter. Preliminary findings from these measures are presented below.
The 30-item Family Resources Survey (FRS; Dunst & Leet, 1987) measures the adequacy of a variety of resources needed by households with young children. Caregivers rate the adequacy of their personal resources in a number of areas, including time for personal growth and interpersonal relationships; income; food, shelter, clothing, utilities, and health care; and the availability of a telephone, dependable transportation, and childcare. The FRS total score is calculated by averaging responses (ranging from 1 = not at all adequate to 5 = almost always adequate) to each item.
Caregivers indicated that their family resources were "sometimes" or "usually" adequate. However, ratings on items such as vacations were lower than those for the basic necessities such as housing. |
At intake, the average FRS total score (3.61) indicated that generally caregivers considered their family resources "sometimes" or "usually" adequate, but ratings of individual resources varied. Although families indicated that available physical necessities and medical care were largely adequate, additional financial resources available for vacations, entertainment, and savings, as well as time to be by oneself or with other adults were in shorter supply. Over three-quarters of the caregivers indicated that they almost always had adequate housing, but 50 percent indicated that money for savings or vacations was not at all adequate. In addition, 33 percent of the caregivers reported inadequate childcare/day care and 20 percent reported inadequate time to be by themselves.
The Family Assessment Device (FAD; Epstein, Baldwin, & Bishop, 1983) is a measure of family health and functioning (i.e., how the family interacts, communicates, and works together) along six dimensions: Problem Solving, Communication, Family Roles, Affective Responsiveness (the extent to which family members experience appropriate affect with each other in a variety of situations), Affective Involvement (the degree to which family members are interested in and place value on each other's activities and concerns), and Behavior Control. In addition, a General Functioning scale assesses overall family functioning. Both caregivers and youth 11 years or older complete the FAD in grant communities funded in 1997-98. It is useful in assessing the extent to which the family as a whole has been impacted by participation in system-of-care services.
At intake, both caregivers and youth rated overall family functioning as positive, especially in the areas of behavior control (M = 3.08) and affective responsiveness (M = 2.85). Caregivers and youth generally agreed, then, that their family members experienced appropriate affect with each other and controlled their behaviors in a variety of situations. However, family functioning as reported by caregivers differed significantly by the custody status of the children for the following areas: family roles (F = 11.273, df = 3/400, p = .000), affective responsiveness (F = 5.333, df = 3/399, p = .001), problem solving (F = 4.282, df = 3/399, p = .005), and general functioning (F = 5.766, df = 3/400, p = .001). In addition, children in the custody of their guardians such as adoptive parent(s), foster parent(s), aunt and/or uncle, and grandparents were reported to have the highest level of family functioning across all domains, although not all the differences in scores were significant. Children's self-report of family functioning did not, however, indicate significant differences by custody status.
The Caregiver Strain Questionnaire (CGSQ; Brannan, Heflinger, & Bickman, 1998) assesses the extent to which caregivers experience additional difficulties, strains, and other negative effects as the result of their caregiving responsibilities. Respondents assess strain along four dimensions: the total impact on the family (global strain); observable negative events or consequences related to the child's disorder have been a problem for the family (objective strain); negative feelings about the child such as anger, resentment, or embarrassment (subjective externalizing strain); and negative personal feelings such as worry, guilt, and fatigue (subjective internalizing strain).
At intake, the average score for caregiver global strain (2.97) indicated that generally caregivers reported that they were "somewhat" strained by the caregiving of their children, but ratings varied for caregivers' objective strain, subjective strain-internalizing, and subjective strain-externalizing. Caregivers reported the highest level of strain in subjective strain-internalizing (M = 3.63), which includes worrying about the child and family's future and being tired as a result of the child's emotional and behavioral problems. The average score for caregivers' objective strain (e.g. interruption of personal time, missing work, and financial strain) was 2.78, and caregiver reported the lowest level of strain in subjective strain-externalizing (M = 2.49), which includes embarrassment about the child, anger and resentment towards the child because of his or her emotional and behavioral problems.
Repeated assessments at 6 months, 1 year, and 2 years of children and families enrolled in the longitudinal outcomes study in communities funded in 1993-94 provided an opportunity to evaluate changes in functioning, behaviors, and emotions of children receiving services in the system-of-care programs over an extended period of time. Children who remained in services were assessed at each follow-up data collection point. By design, these children are a small representative subset of all children served in system-of-care communities funded in 1993-94. The analyses of changes among children enrolled in system-of-care services to 2 years include only those groups of children and families from the outcomes study sample who had complete data at all four evaluation points (intake, 6 months, 1 year, and 2 years) for a particular analysis.
Changes in functional impairment, and emotional and behavioral problems were assessed across time using the CAFAS, the CBCL, and the YSR. In addition to clinical assessments, children's ability to function in the home, school, and community were also assessed by examining stability of the child's living arrangement, educational status, and contacts with the juvenile justice system across time.
Functional impairment, as measured by the CAFAS, declined at 6 months, and continued to decline at 1 year and 2 years. As depicted in Figure 32, average total CAFAS scores decreased substantially from 91.3 at intake to 73.9 at 2 years for children who remained enrolled in system-of-care services. Changes in functional impairment varied slightly by gender. While boys and girls improved similarly across time, boys displayed higher levels of functional impairment at all evaluation points. Although girls made slightly greater improvement in functioning during their first 6 months in services, both boys and girls improved significantly from intake to 2 years.
Average CBCL total competence scores improved from intake to 2 years among children enrolled in the system of care. Although social competence and school competence showed some increase from intake to 2 years, competence scores for sports and work activities remained fairly constant and above the clinical range.
On average, caregivers reported significant improvements in their children's internalizing, externalizing, and total behavior problems on the CBCL from intake to 6 months to 1 year and to 2 years (see Figure 33). Similar improvements were observed for youth self-report of problems, as indicated by a 5.4 point decrease in average total YSR scores (see Figure 34).
The reliable change index (RCI; Jacobson, Roberts, Berns, & McGlinchey, 1999; Jacobson & Truax, 1991; Speer & Greenbaum, 1995) was used to assess individual behavioral and emotional change over time. The RCI is particularly useful when there is no comparison group in a study's design. This statistic compares a child's scores at two different points in time and indicates whether a change in scores shows clinically significant improvement, stability, or deterioration.
Figure 35 displays the RCI results from intake to 6 months, intake to 1 year, and intake to 2 years for children's total problem behaviors. For total problem scores, from intake to 6 months, 32 percent of the children displayed clinically significant improvement, from intake to 1 year, this figure rose to 39.6 percent, and from intake to 2 years, this percentage increased to 46.1 percent.
As indicated in Figure 36, within the group of children who improved significantly from intake to 2 years, about 70 percent were rated by their caregivers below the clinical range after 2 years in the system of care.
Youth self-report of total problem behaviors also indicated improvement from intake to 2 years ( see Figures 37 and 38). About 28 percent of the youth who received services during their first 6 months in systems of care self-reported significant improvements in their behavioral and emotional problems, 35.6 percent of youth who received services to 1 year self-reported improvement, and 40 percent of youth who received services for up to 2 years self-reported improvement. Eighty-five percent of those whose problem behaviors improved obtained average total YSR scores below the clinical range after 2 years in system-of-care services.
One of the major principles of the system-of-care initiative is to provide services so that children can remain in their homes and in their communities. Stability of living arrangements is a reflection of how well a system of care is achieving goals related to that principle. Overall, the stability of living arrangements increased among children in the system of care from intake to 2 years. The majority of children reported only one living arrangement in the year prior to intake (64 percent); however, there was a decrease from intake to 2 years in the percentage of children with multiple living arrangements (see Figure 39).
School attendance and grade performance are two important indicators of adaptation to demands placed upon children by parents and other authority figures in the community. Positive school adaptation and high performance are important predictors of life success. It is expected that both attendance and performance would improve significantly for children receiving services in systems of care.
Among grant communities funded in 1993-94, school performance and attendance improved for both boys and girls from intake to 2 years (see Table 7).
At intake, among children receiving services in communities funded in 1993-94, 16.5 percent received their education in regular classrooms with special education services, and 19 percent received education in special education classrooms. In addition, 33 percent received assistance from a classroom aide other than the regular education teacher. Of the 1,543 children who were placed in special education classrooms at intake, 17.6 percent were transferred either to regular education classrooms or to regular education classrooms with special education support after 2 years of services in systems of care. Those children who were in special education at intake showed significant school performance improvements from intake to 6 months to 1 year and to 2 years (see Table 8).
An Individualized Education Plan (IEP) is an important component for providing effective services through the school to children with emotional disturbance. These plans clarify service provision and provide annual goals for measuring progress in behavioral and emotional as well as academic areas. Individualized Education Plans can facilitate interagency collaboration in meeting the needs of individual children and are consistent with the system-of-care philosophy.
Overall, the use of IEPs increased by 19 percent from intake to 2 years, indicating that participation in systems of care may help children to gain greater access to educational planning and support services. |
Increased use of IEPs was found for children in the outcomes study over time. As shown in Figure 40, at intake into services, 41.7 percent of the 726 children in the outcomes study who had complete data at all four data collection points did not have an IEP. The percentage of children who had an IEP increased from intake to 2 years. These statistics demonstrate an overall increase in the use of IEPs of 19 percent from intake to 2 years. Participation in systems of care, therefore, may help children to gain greater access to educational planning and support services.
School performance improved among all children served in the systems of care from intake to 2 years. |
School performance improved among all children served in the systems of care from intake to 2 years. This improvement was evident among both children who had an IEP throughout the 2 years and those who never had an IEP in the 2-year period (see Figure 41). However, the patterns of improvement in school performance were different across time for the two groups. While children with an IEP throughout the 2 years improved their school performance steadily from intake to 6 months to 1 year and to 2 years, school performance for children who never had an IEP in the same period improved from intake to 6 months but deteriorated from 6 months to 1 year. However, these same children were reported to improve dramatically from their 1-year assessment to their 2-year assessment.
Overall, contacts with law enforcement decreased among children in the system of care from intake to 2 years. |
Decreased involvement with the legal system is also an indicator of positive change for children and adolescents who received system-of-care services. As noted previously, courts and juvenile correctional systems referred about 15 percent of the children into the system of care. Overall, contacts with law enforcement decreased among children in the system of care from intake to 2 years. Although most children had not had any contact with law enforcement in the year prior to intake into system-of-care services, 23.9 percent of the children had one or more contacts with law enforcement in the prior year as a result of one or more violations of the law (see Figure 42). Overall, for children with information on contact with law enforcement at all four data collection points (intake, 6 months, 1 year, and 2 years), there was a decrease in the number of contacts with law enforcement from intake to 2 years.
After a child had been enrolled in the system of care for about 6 months, family caregivers and youth aged 11 or older were asked their opinions about the services their children received and whether they were satisfied with their children's progress to date. Caregivers rated their satisfaction again at each subsequent data collection point.
After 2 years of enrollment, most children and families remained satisfied with the services they had received in the systems of care. As seen in Figure 43, at the 2-year follow-up, 68 percent of caregivers rated their children's services as either good or excellent, 71 percent were satisfied or very satisfied with their children's progress, 76 percent claimed that they usually or always had choice in the services their children received, 68 percent claimed they were usually or always asked for their ideas and opinions concerning their children's services, and 67 percent agreed that provision of care was unconditional. Satisfaction with system-of-care services at 2 years differed slightly from satisfaction at 6 months, yet caregivers reported higher satisfaction with their children's progress at 2 years.
These relatively high levels of satisfaction among caregivers are consistent with those found in the children's mental health literature (e.g., Heflinger, Sonnichen, & Brannan, 1996). High satisfaction may have been due to the systems of care involving families in service planning and treatment. The system-of-care philosophy calls for active engagement of families in the decisionmaking process, solicitation of family members' ideas and opinions, and provision of unconditional care.
As seen in Figure 44, youth were also generally satisfied with the services they had received while enrolled in systems of care. Though they did not generally feel that they had as much influence as their caregivers in determining the nature of their services, 66 percent rated their services to be good or excellent, 68 percent reported satisfaction with their own progress, and 65 percent agreed that provision of care was unconditional at 2 years.
It is not entirely surprising that satisfaction with services declined slightly but ratings of progress increased or remained stable among children and families enrolled in services for some time. Literature indicates that factors influencing initial consumer satisfaction with services differ from those that are important after services have been received for some time. The quality of service encounters is more important initially, while satisfaction with a product is more important later (Mittal, Kumar, & Tsiros, 1999). When applying this concept to system-of-care services for children with serious emotional disturbance, the child's progress could be considered the "product." Therefore, those children and families who were likely to remain in services over a 2-year period may be more likely to report improvement as a result of these services.
Children and adolescents participating in systems of care funded in 1993-94 experienced improved functioning and decreased behavioral and emotional symptoms across 2 years of receiving services. These positive changes were accompanied by improved school performance, decreased law enforcement contacts, and sustained satisfaction among caregivers and youth.
For reporting and analysis purposes, data were aggregated across grant communities funded in 1997-98. However, examples of outcomes compared across a subset of grant communities were also included to illustrate site-level differences and similarities. Information presented in the following section regarding longitudinal outcomes should be interpreted with caution due to the preliminary nature of the data. Given these limitations, findings from these outcomes analyses cannot be generalized and may only serve as snapshots of those children with serious emotional disturbance who have received services in these systems of care.
One of the goals of systems of care is to provide a broad array of services to meet the individual needs of children and their families. Information regarding the services children receive can provide valuable data to system stakeholders to help improve the implementation, refinement, and sustainability of services in systems of care.
After 6 months in systems of care, caregivers reported their utilization of services by using a tool developed for this purpose: The Multi-Sector Service Contacts (MSSC) form. The MSSC provides standard descriptions of types of services, with service names and settings customized for each grant community. Standard descriptions of services in the measure allow for cross-site comparisons. The use of local service and agency names familiar to caregivers is expected to improve the reporting of service contacts. In addition to the type of services, caregivers report the location where services were received, frequency of services received, and sequencing of services. Analysis of the MSSC data provides information about the diversity of the service array and potentially indicates how agencies collaborate or coordinate in a given community.
Preliminary analyses indicated that children and families used an average of six different services between intake and 6 months. Services most utilized were case management (78.8 percent) and individual therapy (73.9 percent), followed by assessment/evaluation (65 percent) and medication treatment/monitoring (58.7 percent). Case management services were primarily provided in a mental health setting (31 percent) and in the family's home (28 percent), as was the case with individual therapy (53.5 and 13.2 percent, respectively). Assessment services were primarily provided in a mental health setting (44 percent) or at school (23.5 percent). Sixty percent of all medication monitoring occurred in a mental health setting. On average, of those who had received these services in the past 6 months, children and families had 24 contacts with their case managers, and children had 15 individual therapy sessions, 2.5 assessment sessions, and 7 medication monitoring sessions. Figure 45 illustrates the distribution of traditional service use among the population.
The array and amount of service use in these areas is in keeping with higher levels of service use expected for children with serious emotional disturbance and their families. During their first 6 months in community-based, comprehensive services, only a limited number of children received more intensive services such as day treatment (6.5 percent) and inpatient hospitalization (13 percent). Even fewer children were provided services through a residential treatment center (4.3 percent), therapeutic group home (11 percent), or therapeutic foster care (6.5 percent) in the early months of participating in system-of-care services.
In addition to these traditional services, children and families also utilized nontraditional services such as recreational activities, family support, and transportation. One of the most popular services delivered in the first 6 months was recreation provided in the community through resources such as youth sports programs, club memberships and summer camps. Children engaged in these activities on average about 26 times in the first 6 months of services. Family support services were utilized by 32.6 percent of families. These services were provided in mental health settings (21.6 percent), community locations (25.5 percent), and homes (17.6 percent), and families used these services an average of 21 times in 6 months. Figure 46 illustrates the distribution of the full array of nontraditional services utilized. It should be noted that due to the age distribution of the sample, only a small percentage of children received independent living (7.4 percent) or transition (3.9 percent) services in the first 6 months of participating in system-of-care services.
In addition to potential benefits realized by youth, use of these services impacted caregivers' and other family members' ability to work. Of the 133 caregivers who worked outside of the home or reported on the ability of another family member to work outside of the home, 56 percent indicated that the mental health services their child or family received had increased their ability to do their jobs. This added benefit to families' economic and emotional well-being is an important collateral consequence of service delivery that cannot be understated.
After 6 months in the system-of-care programs, caregivers' ratings of children's strengths increased significantly in all areas except school functioning, which improved slightly. As illustrated in Figure 47, children made the greatest improvement in intrapersonal strength. Overall strengths also improved significantly from 83.4 at intake to 89.9 at 6 months (Paired-sample t = -4.709, df = 119, p = .000). There were no significant differences in the changes in standard scores and the overall strength quotient by gender or by age.
While there were no statistically significant changes from intake to 6 months in overall strengths for children with specific diagnoses (depression, ADHD, or conduct-related disorders), children with conduct-related disorders improved at a higher rate than those in the other two groups. Figure 48 illustrates that while all three groups of children had average strength scores at intake and improved from intake to 6 months, children with conduct-related disorders had an increase of 10 points in their strength quotient, compared to those with depression or ADHD, who had a 4-point increase.
As expected, children's behavioral and emotional strengths at intake were negatively correlated with their functional impairment as indicated on the CAFAS. Figure 49 illustrates how children's strengths decreased as their functional impairment increased. Children with minimal or mild impairment were rated with average strengths at intake as compared to children in the other three groups, who had below-average strength scores at intake.
Children's behavioral and emotional strengths improved over time. |
Initial analyses of evaluation data from grant communities funded in 1997-98 revealed that although children's overall strengths improved from intake to 6 months, the change patterns differed dramatically among grant communities. Figure 50 illustrates the changes in strength quotients for three grant community samples and the overall sample. These three communities were selected for comparison because they had the most BERS data available at intake and 6 months. Among these three communities, only the change in scores for children in central Nebraska was statistically significant, and this community contributed a majority of the data. While the trend toward improvement is mirrored in the aggregate data, site comparisons indicate the importance of interpreting these preliminary longitudinal outcomes within the context of site characteristics. These changes across time should be interpreted cautiously at this initial point in the data collection process.
Functional impairment, as measured by total mean scores on the CAFAS, declined from 94.7 at intake to 74.0 at 6 months (a 20.7 percent decrease). Changes in the percentage of children with moderate or severe impairment on the CAFAS role performance subscales from intake to 6 months revealed that, on average, children made discernable improvements in functioning at school, at home, and in the community.
The pattern of declining scores on the CAFAS from intake to 6 months differed slightly across grant communities. For analyses of community-specific CAFAS score changes, four grant communities that each contributed at least 10 percent of the data to the overall CAFAS sample were included. Score changes for these four grant communities and the overall sample are displayed in Figure 51. Only the change in scores in central Nebraska was statistically significant. Because this grant community contributed about 45 percent of the CAFAS data to the aggregate sample, it had a pronounced influence on the overall trend. Therefore, these preliminary analyses should be interpreted with caution until sample sizes increase across sites.
Overall, children in grant communities funded in 1997-98 made significant improvements from intake to 6 months in behavioral and emotional problems as rated by their caregivers on the CBCL. Children's competence scores on the CBCL also increased slightly from intake to 6 months.
Grant communities that each contributed at least 10 percent of CBCL data available at intake and 6 months are profiled in community-specific analyses (see Figure 52). Although caregivers from all three grant communities generally rated their children significantly improved after 6 months, mean scores still reside in the clinical range, indicating continued serious concerns. This pattern of change for the total problems scores held true for the Internalizing scores and Externalizing scores as well.
Figure 53 illustrates the changes in number of living arrangements after 6 months in the system of care for children who had lived in multiple living arrangements. Of these children, 53 percent stayed in just one living arrangement during the 6 months after intake into services. In addition, because 79 percent of the children at intake who had lived in one living arrangement remained in one arrangement after 6 months, the overall percentage of children in single living arrangements increased to 66 percent (an 11 percent increase from intake).
The majority of the children in grant communities funded in 1997-98 attended school on a regular basis at both intake and 6 months (73 percent). Still, after 6 months in the systems of care, there was a significant increase in the number of children who attended more than 50 percent of the time to 92 percent (χ2 = 14.462, df = 1, p < .000).
As expected, school attendance and school performance were strongly related at intake (r = .444, p < .000, n = 342) and at 6 months (r = .552, p < .000, n = 140). Children attending school more than 50 percent of the time were twice as likely to earn average or above average grades than those who attended less than 50 percent of the time. Of children with below average grades at 6 months, 79 percent entered the system of care with three or more family and child risk factors.
Of the 179 children with both intake and 6-month IEP information, the percentage of children with an IEP increased slightly from 43 percent at intake to 45 percent at 6 months.
The 1997 National Longitudinal Survey of Youth (NLSY), which interviewed 9,000 youth between the ages of 12 and 16, reported that 8 percent of the youth had been arrested in their life time (Bureau of Labor Statistics, 1998). This percentage is substantially lower than the 50 percent (of 267 youth) in the comparable age range who self-reported an arrest history among those enrolled in the evaluation grant communities funded in 1997-98. This higher rate of arrests among youth served by system-of-care programs is not unexpected, given that grant communities establish specific criteria such as prior involvement with the juvenile justice system for service eligibility. Of those youth who had been arrested prior to intake, 74 percent were boys and 26 percent were girls. Arrest history differed significantly by age, even after controlling for gender differences. As expected, both older boys and girls were more likely to have been arrested (see Figure 54).
Despite these high juvenile justice statistics and child and family risk factors at intake, delinquent behaviors among youth declined during the first 6 months in system-of-care services. Figure 55 illustrates the changes in juvenile justice involvement from intake to 6 months. Fewer children were convicted of crimes, on probation, or sent to jail/detention center at 6 months.
Of the 57 youth for whom both intake and 6-month data were available, the percentage of children who reported drinking in the past 6 months decreased from 67 percent at intake to 61 percent at 6 months.
Preliminary analysis from intake to 6 months indicated no significant changes in the FRS mean total scores, indicating that overall adequacy of family resources remained unchanged after participating in systems of care for 6 months.
According to both caregiver and youth reports, family functioning remained stable from intake to 6 months (see Figure 56). Generally, caregivers who reported at both intake and 6 months rated their families' behavior control as quite positive in their first 6 months of system-of-care services, and ratings on the other five dimensions remained steady.
Caregiver responses indicated that strain was reduced significantly across all dimensions from intake to 6 months, particularly in the domain of objective strain (see Figure 57). Decreased scores may indicate reduction in negative events such as disrupted family relationships, routines, social activities, and loss of personal time.
After being enrolled in systems of care for 6 months, most caregivers and youth were satisfied with the services they received (see Figures 58 and 59).The areas in which the highest percentage of caregivers expressed satisfaction included their involvement in service implementation meetings (87.9 percent) and their involvement in planning services (87.4 percent). The highest percentage of youth reported satisfaction with their providers' abilities to find services that acknowledge the positive aspects or strengths of their family's culture and traditions (77.9 percent), and their level of involvement in planning services (77.1 percent). These findings are consistent with the system-of-care philosophy which calls for active engagement of families in decisionmaking about their care, and they indicate that positive system reforms may have been implemented quickly in the grant communities.
The amount or variety of services, as measured by the MSSC, was not related to satisfaction levels. However, level of satisfaction did vary significantly for caregivers whose children received individual therapy (χ2 = 15.061, df = 4, p = .005). Sixty-nine percent of caregivers whose children received individual therapy indicated that they were "satisfied" or "very satisfied" with their child's progress in the past 6 months, compared to 59 percent of those whose children did not receive individual therapy. Furthermore, 19 percent of caregivers whose children who did not receive individual therapy reported that they were "very dissatisfied" with their child's progress, a sharp contrast to the only 1 percent of those whose children did receive the service and reported this level of dissatisfaction. Level of satisfaction also differed for those families who received caregiver or family support services versus those who did not. Families who received support services were more satisfied with both the number of times they were asked to participate in service implementation meetings and their providers' understanding of their family's cultural traditions (see Figure 60).
Data from both the evaluation of grant communities funded in 1993-94 and the evaluation of those funded in 1997-98 indicated that children participating in the Comprehensive Community Mental Health for Children and Their Families Program were challenged by serious emotional disturbance. Although grant communities differed somewhat in their target populations and the services they provided, the majority of children served had diagnoses of disruptive behavior disorders or depression and were challenged by serious behavioral problems and accompanying functional impairment. Among children enrolled in the evaluation of grant communities funded in 1993-94, their social functioning and behavioral and emotional symptoms improved significantly across 2 years of participation in systems of care based on parent, clinician, and self-reports. Children also improved in school performance and attendance, and they were more likely to remain in stable living arrangements. Because longitudinal outcomes presented here cannot be compared to outcomes for children in communities that do not have system-of-care grants, and because children in communities funded in 1993-94 were only followed while in services, changes across time should be interpreted cautiously.
Preliminary outcomes data on children participating in the evaluation of grant communities funded in 1997-98 indicate that these children faced similar challenges yet made similar improvements in their social functioning and behavioral and emotional problems. With the addition of a strengths-based instrument, data indicated that children had below average strengths upon entry into the system-of-care programs, although those with less impairment displayed higher levels of strengths. Improvements in children's strengths at 6 months were also noted. History of substance use was more frequently reported among children participating in system-of-care programs when compared to national averages for youth of similar age; however, preliminary findings show decreases in substance use over the first 6 months in services. Arrest rates among youth served by system-of-care programs were also higher than those reported nationally among youth; however, involvement with the juvenile justice system declined for youth in the first 6 months. Children and families used an average of six services in the first 6 months, receiving each service on multiple occasions. Caregiver measures of family resources, family functioning, and caregiver strain are providing new information from families receiving services in communities funded after 1994. Caregivers reported the need for additional personal and financial resources as they entered services. Six months after service entry, caregivers reported that the services that they had received improved their ability to work outside the home, and also indicated decreases in their strain associated with caring for a child with emotional or behavioral disturbance. Improvements were also noted in overall family functioning, and caregivers reported that the strain they experienced while caring for their child with serious emotional disturbance declined after just 6 months in a system of care.
Demographics