This Web site is a component of the SAMHSA Health Information Network
|
This Web site is a component of the SAMHSA Health Information Network |
| | | | | | |  |
||||||||||
 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This Web site is a component of the SAMHSA Health Information Network. |
|
Section II: Supporting Good Decision-MakingPDF
version Chapter 6. Sixteen-State Study on Mental Health Performance MeasuresTheodore Lutterman Vijay Ganju, Ph.D., Lucille Schacht, Ph.D., Robert Shaw, M.S., Kathleen Higgins Ray Bottger Molly Brunk, Ph.D., J. Randy Koch, Ph.D. Nancy Callahan, Ph.D. Craig Colton, Ph.D. Dennis Geertsen, Ph.D. Judy Hall, Ph.D. Debra Kupfer, M.H.H.S. Jocelyn Letourneau John McGrew, Ph.D. Sudha Mehta, M.P.H. John Pandiani, Ph.D. Bernadette Phelan, Ph.D. Mary Smith, Ph.D. Steve Onken, Ph.D. Donna C. Stimpson, MUP Ann E. Rock Jack Wackwitz, Ph.D. Marie Danforth, M.S.W. Olinda Gonzalez, Ph.D. Nainan Thomas, Ph.D., Ronald W. Manderscheid, Ph.D. Overview of Study and Key FindingsThe 16-State Study on Mental Health Performance Measures is a landmark joint State-Federal initiative to apply identical standardized definitions and obtain comparable performance and outcome indicators on public mental health systems from multiple States. This study presents results on 32 mental health performance indicators. Several indicators use more than one measure. Thus, a total of 49 different measures were included in this project (for example, the indicator on "seclusion" has two different measures: the percentage of consumers secluded and hours of seclusion). Data analysts, researchers, managers, planners, and consumers from 16 State mental health agencies participated in the project with the Substance Abuse and Mental Health Services Administration (SAMHSA) and its Center for Mental Health Services (CMHS) over a three-year period from 1998 to 2001. The 16 States were selected through a competitive grant application process. The States committed to work cooperatively with each other and SAMHSA to define, implement, and report on common mental health performance indicators based on the work of the Federally funded Mental Health Statistical Improvement Program (MHSIP), the National Association of State Mental Health Program Directors (NASMHPD) Framework of Mental Health Performance Indicators, and the original SAMHSA-funded Five-State Feasibility Study. Collectively, the 16 States have reported on community mental health services to almost 1.45 million individuals through public mental health systems that expended more than $7.3 billion annually to provide these services. Despite major differences in the organizational structure of the States, their mental health service system configurations, and in their information systems, this project has compiled data on many important performance indicators that represent the critical domains and measures identified in mental health development efforts by various national organizations. The project has demonstrated that the States can collaborate effectively to define and implement measures with appropriate Federal assistance and leadership. The performance measures selected for this study encompassed critical concerns identified in various performance measurement initiatives and for which at least some States could report data during the study period using standardized definitions. The selected measures include outcomes, appropriateness/quality and access measures (including clinical measures and measures from the consumer's perspective), and measures that apply to both inpatient and community settings. States reported each indicator for the State as a whole and for age, gender, racial/ethnic, and diagnostic groups. The 16-State Study has made substantial progress in operationalizing and compiling information on the indicators first tested by the Five-State Feasibility Study (Ganju and Lutterman, 1998) and the NASMHPD Framework of Performance Indicators (NASMHPD President's Task Force, 1998). In addition, many of the indicators in the NASMHPD Framework had their initial feasibility of multi- State compilation and comparability tested through this study. This project built on the work the States and SAMHSA completed on developing performance measures for public mental health systems. These earlier initiatives paved the way for the 16-State Study and enabled it to accomplish its goals. SAMHSA's CMHS has sponsored major efforts to help the mental health community define and implement performance and outcome measures. Through its support of the MHSIP Consumer-Oriented Mental Health Report Card published in 1996, CMHS helped develop a set of performance measures for systems to implement. The SAMHSA- funded MHSIP State Reform Grants enabled more than 40 States to implement components of the MHSIP Consumer-Oriented Report Card and provided needed resources and the impetus for States to implement these measures. The Five-State Feasibility Study tested the feasibility of a set of performance indicators with definitions and guidelines for State reporting that then became the basis for the 16- State Study. Each of the 16 States in this study received a SAMHSA grant of $100,000 per year to facilitate their participation in this project. Each of the States also devoted considerable State resources to this project. State resources included staff time, programming and computer resources, and funds to support consumer focus groups and other stakeholder participation The results of this study demonstrate the potential for developing standardized measures across States and confirmed that the realization of this potential will depend on enhancements of the data and performance measurement infrastructure. The results demonstrate that States are implementing mental health performance measurement systems and that some States can use these systems currently to report comparable information. The results also demonstrate that each State system has some performance measures that are unique. Considerable effort is required to assure the comparability of these measures across States and to support States as they produce improved measures of outcomes and consumer assessments of care. BackgroundThe 16-State Study was funded by CMHS, as a collaborative project. Sixteen States were awarded grants for a 3-year period (1999-2001) to implement performance indicators that were developed in the SAMHSA-funded Five-State Feasibility Study and the 1998 NASMHPD Framework of Mental Health Performance Indicators. The primary goal of the project was to (1) compile specific performance indicators that could be reported comparably across States for national reporting, and (2) facilitate planning, policy formulation, and decisionmaking at the State level. The grants also supported the involvement and participation of key stakeholders, including consumers and family members, at all stages of the grant process. The 16- State Study grantees were Arizona, Colorado, Connecticut, Illinois, Indiana, Missouri, New York, Oklahoma, Rhode Island, South Carolina, Texas, Utah, Vermont, Virginia, Washington, and the District of Columbia. Prior to its completion, the 16-State Study had a major impact on national performance indicator efforts in public mental health. Six of the 32 performance indicators in the 16-State Study are tied to the Federal Government Performance and Results Act (GPRA) as core measures for State reporting. Many of the indicators implemented by the 16-State Study have become the basis for the new SAMHSA State Mental Health Data Infrastructure Grants (DIGs) awarded in 2001 and 2002 to 49 States, the District of Columbia, and seven U.S. territories. Under these DIGs, States will report standardized data tables included in the new Uniform Reporting System (URS), a critical component of CMHS Block Grant reporting. Under the URS, the accomplishments of individual States can be aggregated meaningfully at the national level. Besides developing standardized definitions, the 16-State Study also provided an opportunity to develop measures related to recovery, consumer/ family involvement in policy development, quality assurance, planning, expenditures for mental health, and evidence-based practices. Public Sector Performance Measurement Initiatives The recent resurgence of mental health performance measurement has come about in response to consumer and family need, national and local funding demands for program quality and accountability, and emerging managed care initiatives. The increased consumer and family involvement in public mental health has led to an increased need to demonstrate that services are effective and have good outcomes. Payers of mental health services, such as legislatures, executive branches of government, and other funders, have demanded that public programs show results and outcomes from services. Managed care, as a mechanism to purchase mental health services, with its emphasis on cost management and possibilities of consumer choice, created new imperatives for public sector mental health managers. With the advent of managed care, managers needed to specify in measurable terms what was being purchased in contracts. In response to managed care's emphasis on cost, concern grew that savings were being accrued at the cost of quality. To counter managed care's focus on process measures, there emerged a new wave of activity to assess and monitor outcomes. In the last decade, regardless of whether a public sector entity was directly involved with managed care, a new, more business-like ethos began to permeate the public mental health sector. It included more sophisticated contractual arrangements, improved data and information systems, and an emphasis on performance and outcomes measurement. Figure 1 displays the interrelationships among the 16-State Study, a number of earlier mental health performance indicator initiatives, and future initiatives. Each of the initiatives is discussed below. Mental Health Statistics Improvement Program (MHSIP). Since its inception in 1976, the MHSIP Advisory Group and the new MHSIP Policy Group have worked to develop data standards for public mental health systems. The MHSIP Policy Group, which includes representatives of Federal, State, and local mental health agencies and consumers and family members, has advised on data analysis and reporting issues and the development of grant programs in these areas. Over the years, the MHSIP program has developed concept papers and provided guidance to State and local mental health systems in such areas as data standards, unique identifiers, consumer-centered information systems, performance measures, and report cards. Two specific MHSIP products resulted in major SAMHSA grant initiatives to States. The MHSIP document Data Standards for Mental Health Decision Support Systems (Leginski et al., 1989) defined core elements required for a mental health management information system in five areas-client, encounter, financial, human resources, and organizational. The MHSIP Consumer-Oriented Mental Health Report Card (Ganju et al., 1996) identified performance measures that reflected consumer concerns in the domains of access, quality/appropriateness, outcomes, and prevention to be used for assessing the effectiveness of mental health services. The MHSIP proposed a consumer survey as part of the report card to include a consumer assessment of indicators in each of these domains. The development of MHSIP minimum data standards resulted in two cycles of SAMHSA grants to States to incorporate and implement the standards in management information systems. Through these MHSIP grants, every State received at least one grant to facilitate its implementation of common data definitions and standards. This common set of data standards has led to the use of standardized data elements across the country. States have exhibited a range of infrastructural capacities in these areas. Not all States were able to implement the core MHSIP elements; nevertheless, the standards have helped the public sector move in the direction of a common, standardized data set. The 1996 MHSIP Consumer-Oriented Mental Health Report Card was developed by a MHSIP task force that included consumers; family members; researchers; advocates; and Federal, State, and local mental health agency representatives. SAMHSA promoted the adoption and use of the performance measures specified in the report card through the MHSIP Reform Grant program. These MHSIP reform grants assisted 45 States to implement different components of the report card. These grant-funded State activities have served as a key foundation for the subsequent work of States to produce comparable mental health performance indicators. The Five-State Feasibility Study. In 1997, SAMHSA funded five State mental health agencies (Colorado, Illinois, Massachusetts, South Carolina, and Texas) to identify and pilot performance indicators that would be feasible and meaningful to collect and could be compiled from existing data systems within the States in a comparable fashion. The Five-State Feasibility Study was an effort to assess the feasibility of States reporting data on standardized indicators and measures. Twenty-eight indicators were tested in the study. NASMHPD President's Task Force on Performance and Outcomes Measures. NASMHPD's President's Task Force was established to build on the work of the MHSIP Report Card so that it would have more of a management orientation and include measures that were responsive to the needs of State mental health commissioners. New performance measures were proposed in the domain of structure/ management and in the performance of State hospital systems. The NASMHPD performance measures also were intended to reflect public sector values and priorities, especially as other mental health sectors proposed alternative report cards. Using the MHSIP Report Card as a starting point, indicators from several national mental health performance measurement initiatives were reviewed, including those developed by the National Committee on Quality Assurance (NCQA), the SAMHSA Performance Partnership Grants (PPGs), the American Managed Behavioral Healthcare Association, The National Alliance on Mental Illness (NAMI), and the American College on Mental Health Administration. On the basis of this review and a survey of States designed to better understand priorities and issues of utility and burden, the task force proposed a standardized framework consisting of five domains (access, quality/appropriateness, outcomes, structure/plan management, and early intervention/prevention), providing States considerable flexibility in how these indicators were measured and reported. This standardized framework represents the consensual position of all State mental health commissioners/directors regarding the performance measures to be used in any comprehensive mental health service delivery system. It incorporated the results of the aforementioned Five-State Feasibility Study and the NASMHPD Research Institute (NRI Behavioral Health Performance Measurement System, used to report psychiatric hospital performance measures to the Joint Commission on Accreditation of Healthcare Organizations' (JCAHO) ORYX system. The GPRA, enacted by Congress in 1997, requires Federal agencies to identify a set of core performance indicators for which they will be held accountable. SAMHSA was required to identify and select performance indicators for GPRA in the late 1990s. On the basis of early work by the Five-State Feasibility Study and a set of regional stakeholder meetings, six measures were selected for GPRA reporting. These measures became the basis for voluntary reporting by States as part of their annual mental health block grant applications. However, the measures selected for GPRA have often been difficult for State Mental Health Authorities to report in a uniform fashion, and thus, to date, the GPRA measures have not been well reported. Block Grant Performance Measures from States: Uniform Reporting System. In 2001, SAMHSA published an announcement of its URS in the Federal Register. The new URS contains a set of "basic" and "developmental" performance indicator tables for States to report. Reporting of the URS basic tables began in 2002, with full reporting by all States expected by 2004. The developmental tables include indicators that either still need some operational definitions or will be more difficult for States to report comparably. The developmental indicators will be reported by States on a slower basis, with States working to define and test reporting them for the first time by 2004. The URS builds heavily on the work of the 16- State Study. The tables included in the basic set of the URS are indicators that the majority of States were able to report in the 16-State Study and for which good operational definitions exist. Measures in the developmental set of the URS are either those for which standardized definitions do not exist or for which data are not comparable. The work of the 16- State Study will inform the development of the standardized definitions that are needed. To accompany the URS, SAMHSA State Mental Health DIGs were made to 49 States, the District of Columbia, and seven U.S. territories to help them modify their information system infrastructure to report the performance indicators of the URS. Each of the DIG States and territories received a three- year grant of $100,000 per year to support their work to enhance their information systems to report URS data. In addition, the NRI received funding from SAMHSA to serve as a national coordinating center, to work with the States in their DIG and URS activities. As part of its role, the coordinating center facilitates the development of final standardized definitions, provides technical assistance to States, and will compile and report URS data from the States. 16-State Study Approach The 16-State Study project, initiated in 1998, was a three-year collaborative project between the State mental health agencies in 16 States and SAMHSA. In each of the participating States, representatives of both the data and management information systems offices and their mental health planning offices were part of the project. In addition, most States involved additional stakeholder groups, such as consumers, family members, and other advocates in their conduct of the project. The inclusion of consumers and other stakeholders was a consistent expectation of the project to ensure that study activities remained relevant to the persons served by State systems. The 16-State Study was conducted without a formal coordinating center. To facilitate and coordinate the 16-State Study, a model was adopted that emphasized extensive interstate communications through the use of Internet technologies, such as Web sites and a listserv. Throughout the study, extensive communications were maintained between the States and the Federal Government. All participants met face-to-face only twice during the project. Regular communications were facilitated by monthly conference calls, and an e-mail listserv was used to share data definitions, draft reports, and to both request and compile data. Key to the accomplishments of the 16-State Study was the development of a set of 19 different Indicator Workgroups. The Indicator Workgroups included volunteer groups of grant principal investigators and other grant participants who revised the Five-State Study definitions, compiled data from all States, and prepared reports on their sets of indicators. The 16-State Study Indicator Workgroups are listed in table 1. Each workgroup reviewed the experiences of the Five-State Study and developed updated recommended operational definitions. The recommended operational definitions were discussed with all 16 States on a conference call and the workgroup then issued a request for data from all participating States. The workgroup compiled data from the States into a draft report and sent the report to the States for their review and modification. After review by the States, the workgroup prepared a final report and then sent the data and report for incorporation into this final 16-State Study report. A Reporting Workgroup was established to develop a standardized set of reporting categories for all indicators. The workgroup based its efforts on the reporting categories of the Five-State Feasibility Study, and developed a standardized template used by all workgroups. The final set of recommended reporting categories included measures of consumers' age, gender, race/ethnicity, and diagnosis. For some indicators, data were also compiled for combinations of age and race/ethnicity. As a result of these common reporting categories, virtually all performance indicators compiled by the 16-State Study can be generated for either the total served population or various client subgroups. Thus, the 16-State Study indicators are able to demonstrate differences in subgroups of consumers to determine whether utilization rates, access to new generation medications, or employment status vary on the basis of the gender, race/ethnicity, age, or principal diagnosis of consumers. Some indicators, such as the use of new generation "atypical" anti-psychotic medications for persons with schizophrenia, were, by definition, limited in the consumer population categories compiled (i.e., only persons with a diagnosis of schizophrenia). The standardized reporting categories used by the 16-State Study are listed in table 2. Where individual indicators did not use the full array of reporting categories listed in this table, these exceptions are discussed in the narrative of individual indicators. Most States were able to report both indicator totals and the requested subpopulation categories. States had the greatest difficulty reporting on subpopulations by diagnoses. Several of the 19 workgroups were able to work much faster than others. This was usually because the assigned indicators were in areas that had already been developed by the Five-State Study and States already had the capacity to report. For example, the Utilization Rate Workgroup was able to compile data from all 16 States for all three years of the grant. Other workgroups spent considerable effort designing and piloting their indicators, and as a result, either only compiled data from a few States or never compiled indicator results from the States. Finally, several Indicator Workgroups were initiated near the end of the grant cycle and, thus, had insufficient time for States to compile and report needed data. Producing reports using standardized definitions took intensive work from multiple participants in each of the States. Project participants met twice to coordinate and ensure measure standardization. Many of the Indicator Workgroups met face- to-face, and all held multiple conference calls and extensive e-mail correspondence to operationalize their indicators, gather data, analyze and disseminate tables and graphics, and produce their final indicator reports. Most of the Indicator Workgroups compiled data for State Fiscal Year 2000. Some States were able only to report indicator data for fiscal years 1999 or 2001. Each of the Indicator Workgroups prepared a report on their activities. For most of the workgroups, these reports included results from many of the study States depicted by the core set of client characteristics. Other workgroup reports focused on the development of new instruments to be used in future performance indicator initiatives (e.g., Children's Survey Workgroup and the Evidence-Based Practices Workgroup). Still other workgroups identified additional developmental work that remains to be completed before a comparable performance indicator can be proposed and tested. Two ground rules were established for presenting data in this report. First, results are shown only when three or more States reported data for the indicator. Second, rates by a particular client characteristic are shown when there are at least 25 cases in a specific subpopulation group. For example, readmissions by the race/ethnicity subgroup Native Americans would not be shown if there were fewer than 25 Native Americans in the hospital discharge population for a given State. Key FindingsAbility of States to Report Standardized Measures The 16-State Study found that State Mental Health Authorities can report comparable system measures desired by consumers, families, funders, and others. Figure 2 and table 3 show that State Mental Health Authorities can implement and report on many comparable measures of performance of their mental health systems provided they have sufficient time and appropriate resources. Thirty-eight of the performance measures were developed to the point that standardized data were compiled from at least some of the 16 States. All 16 States were able to report data for four of the measures (10 percent of the measures). Overall, 28 measures (74 percent) were reported on by more than half the States. In addition, several States had begun implementation of some of these performance measures during this project, although they were not able to report indicator results in time for inclusion in this report. State Reporting Because of the large number of measures tested and that final operational definitions were not completed until late in Year three of the project for some measures, no State was able to report data for every measure. Figure 3 shows the number of measures that States were able to report. Some States reported measure results only by using definitions different from the standardized definitions developed for this study. It should be noted that many of the 16- State Study participants had "no cost" extensions that have allowed them to implement additional standardized performance measures after the completion. Participating States have also continued to develop and utilize additional different performance measures specifically relevant to their State's needs and priority concerns. Measures States Could Report As shown in table 3, measures of appropriateness relative to inpatient care, such as readmission rates within 30 days and 180 days; the use of new generation "atypical" antipsychotic medications in State psychiatric hospitals; seclusion and restraint rates in psychiatric hospitals; and contact in the community within seven days of hospital discharge were reported by more than half of the 16 States. Outcome indicators, such as living arrangements, percent homeless, employment status, and mortality rates, also were reported by at least half of the participating States. Access measures of utilization rates of community and inpatient services were reported by every participating State. The measures of outcomes of service-such as improvement in functioning, reduction in psychiatric symptoms, criminal justice involvement, and improvements in school behavior-were the measures with the least comparability. Many States have their own definitions for these measures that are used within the State for quality and accountability purposes. However, these measures, when used, are seldom the same across States. In general, measures that required State Mental Health Authorities to link their data set to other data sets, such as criminal justice records, were more difficult to implement because of technical and interagency issues. The ability of States to report measures may be related to the sources of the indicator data. Only four measures were reported by all 16 States: hospital utilization, community service utilization, and hospital readmissions within 30 days and 180 days. This suggests that States were more able to report indicators from administrative data sets that are included in information systems operated by all the State Mental Health Authorities. Other indicators that rely on consumer status information, such as living arrangements and employment status, were also reportable by many States once standardized reporting categories were developed. Additionally, consumer survey-generated indicators were reported by 13 of the States, demonstrating the broad application of the MHSIP Consumer Survey. Figure 4 shows the average number of States reporting data by various types of indicators, grouped on the sources of data used to generate the indicators. As discussed above, indicators based on single information systems maintained by State Mental Health Authorities generally had the highest level of reporting (hospital data sets, community utilization data, consumer surveys), whereas indicators that rely on matching data records with other information systems, such as criminal justice contacts, mortality rates, and follow-up in the community after hospital discharge, proved more difficult for many States to report. Finally, client assessment measures were particularly difficult for States to report because of differences in instruments and frequency of assessments. Consumer Survey Measures The MHSIP Adult Outpatient Consumer Survey, adapted from the MHSIP Consumer-Oriented Report Card published by SAMHSA in 1997, is being implemented in most States to assess consumers' perceptions of the outcomes, access, and appropriateness of public mental health services; consumers' participation in treatment planning; and contacts with physical health care. Thirteen States were able to report data using the MHSIP Consumer Survey. Each of these States uses similar instruments, and the workgroup was able to calculate scores for comparable indicators and domains from these surveys. However, States implemented the consumer survey using different survey methods, surveyed different client populations, and attained variable response rates, suggesting that caution should be used in comparing survey results across States. The MHSIP Consumer Survey revealed differences across domains that were consistently observed for individual States (see figure 5). More consumers were satisfied with their access to mental health care and with the appropriateness of the services; fewer consumers were satisfied with either the outcomes of treatment or with their level of participation in treatment planning. Nearly 82 percent of consumers surveyed by the States strongly agreed or agreed that they had access to needed services. More than 79 percent of consumers surveyed agreed appropriate services were delivered; and 69 percent reported improved outcomes as a result of these services. Groundwork for Reporting New Measures The 16-State Study participants found that several measures of great interest to States lacked standardized definitions, which limited the utility of reporting. In some cases, Indicator Workgroups completed important work to develop "fidelity" measures that will facilitate future reporting of standardized performance indicators. For example, despite widespread agreement that measures on access to evidence-based practices, such as Assertive Community Treatment (ACT) and Supported Employment, are important indicators to measure, earlier efforts to compile these indicators failed because of the lack of agreement about measuring these evidence-based services. The 16-State Study workgroup on evidence-based practices developed and successfully pilot-tested fidelity "checklist" instruments that can be used in future studies to determine which programs and services should be counted for indicators of ACT and Supported Employment. Similarly, 16-State Study workgroups on Criminal Justice Involvement, Children and Adolescent Perception of Care, Substance Abuse Impairment, Consumer Participation in Planning and Policy Development, Recovery, and Client Assessment Instruments conducted work required to operationalize consistent measures in their domains. As a result of these workgroups, an adolescent and parent consumer survey now exists and is being implemented in many States. This survey will be used by States to provide information for eight different important child indicators. States interested in indicators of criminal justice, substance abuse, and participation in policy and planning now have a solid base of developmental work from which to proceed. Other indicators, such as the use of client assessment instruments to measure the improvement in client functioning, reduction of symptoms, and reduction of substance abuse impairment, require additional development. The 16-State Study workgroups have documented that many States are routinely measuring these important indicators, but difficulties in reconciling different instruments being used by States was beyond the resources available to this project. An important future product of the 16-State Study may be the development of standardized measures of how the mental health system supports "recovery" from a consumer's perspective. The 16- State Study initiated a process and coordinated focus group meetings with consumers in eight States to define and operationalize measures of how mental health systems help and hinder consumers in their process of recovery. The work on recovery was, and continues to be, led by a group of consumer researchers who have defined the process and methodology for the development of this measure. This work on measures of recovery is ongoing; several new recommended measures should emerge later this year. Future work will be needed by States and others to test and implement these new measures of recovery. Ability of the States to Demonstrate the Impact of Mental Health Services The project also suggests that State mental health systems are delivering effective services to individuals with serious mental illnesses. State Mental Health Authorities provide mental health services for individuals most in need (i.e., persons with serious mental illnesses who lack adequate insurance to receive appropriate services from private providers). The 16-State Study documented that the consumers served by State systems frequently were unemployed (76.9 percent), sometimes homeless (2.6 percent), and disproportionately minority group members (31 percent). State Mental Health Authorities have been working actively to develop and implement meaningful measures of the performance of mental health programs for several years. The State Mental Health Authorities, the primary organizations responsible for funding, monitoring, and delivering public mental health services in the nation, have recognized the need to better understand the outcomes of the services they deliver. In this project, they have helped document services they deliver and the impact and outcomes of these services on the lives of individuals with mental illnesses. General Caveats and Cautions on Comparing Results Although the States have made significant progress in developing and reporting performance measures, substantial work remains to ensure that the measures are truly comparable across States. Major policy and regulatory differences among the States must be understood to be able to compare the results of performance measures correctly. For example, the indicator on employment found that persons with serious mental illnesses were less likely to be competitively employed than were persons with other diagnoses. It then becomes important to understand the extent to which an indicator of employment is being reported for comparable populations. A State reporting on a broad population with a small proportion with major mental illnesses is likely to show a higher employment rate than a State reporting only on persons with serious mental illnesses. Participants in the 16-State Study expended most of their time and effort in the development of standardized definitions and the compilation and reporting of information according to these definitions. Because the study was designed to determine the feasibility of States to report quickly a core set of mental health performance measures, the project did not have either the resources or the time to fully explore all the policy and programmatic differences that could explain differences in apparent performance on specific indicators. Because all the program, policy, and population differences inherent in State systems cannot be explained or accounted for, this report presents aggregate results from the study and discusses specific indicators including major cautions when discussing specific State results. For many of these measures, comparing the performance of an individual State across time is an appropriate use of the indicator. Although some States reported the indicator using identical indicator definitions, they may have such different policy mandates and consumer caseloads that the indicator's results may not be comparable across States. As additional work is completed to compile and report performance measures from the States, it will be essential to track not only the actual measure results but also the underlying policy and program differences that will help explain differences in performance indicator values. Considerable work remains to assess how States, consumers, family members, providers, and other advocates are using these performance indicators to improve care. The 16-State Study devoted its efforts to gathering these indicators, but there has not been sufficient time or resources to evaluate how these indicators are best used to improve mental health services. Due to differences among the States, one option would have been to present the data for each State individually. However, to assist the reader, State data are displayed side-by-side to consolidate State- specific results. Although underlying trends may appear similar across the States, the reader is cautioned against drawing such conclusions, because multidimensional differences among States may make such findings either inaccurate or spurious. Furthermore, although a reader may be tempted to extrapolate the 16 State results to the entire Nation, such an effort would be misplaced: the project cannot approximate national estimates using data from only 16 highly variable States and territories. Readers should realize that the range of data for the measures across the 16 States may not reflect the range of differences that might be observed if data and information from all 55 States and territories were included in an analysis. ConclusionsThe 16-State Study demonstrated that State Mental Health Authorities have the capacity and ability to implement and report on a standardized set of performance indicators. The work of the 16- State Study has developed operational definitions for indicators that have been implemented by many of the 16 participating States and that are ready to be used by other States and mental health providers. The 16-State Study workgroups developed operational definitions for such indicators as utilization, consumer perception of care, employment, living situation, use of medications, State hospital readmissions, follow-up in the community after hospital discharge, use of seclusion and restraint in psychiatric hospitals, elopements from hospitals, and injuries in hospitals that were reported by many of the 16 States. These indicators form a solid basis for the next generation of national performance indicator efforts. In addition to the indicators for which the 16- State Study developed and compiled indicator results, a substantial contribution was made to the field in the development of new measures. For example, the generation of fidelity checklist instruments for evidence-based services, such as ACT and Supported Employment, will enable future performance indicator initiatives to have a much better measure of these services. A listing of 16-State Study accomplishments related to performance measurement development includes the following:
Much work remains to assess how States, consumers, family members, providers, and other advocates are using these performance indicators to improve care. The 16-State Study devoted its efforts to gathering these indicators, but has not had sufficient time or resources to evaluate how these indicators are best used to improve mental health services. Implications for the Future This study represents a major advance in the standardization and comparability of performance measures across State mental health systems. This study shows that with appropriate financial and administrative support, States can report standardized performance measures on their mental health systems. Measures reported demonstrate that State mental health services help consumers recover and generate positive outcomes. However, major outstanding issues remain related to the development, implementation, and use of performance measures. These include issues related to definitions, comparability and use, and the significance and weight of various measures. Issues Related to Definition. Although standardized operational definitions were developed for the 16-State Study, they need further refinement and evaluation. At this stage, little empirical support exists for some of the definitions selected. For performance measures related to services, a major issue is the assurance that the critical components of the service are in place-that is, that there is fidelity to the model or operational conformance to some standard for the specific service. For performance measures related to symptoms and functioning, a major issue is the comparability-specific instruments. Many of the 16 States are implementing measures related to monitoring changes in level of functioning and reduction in symptom distress, but the instruments being used are different. Even when the instruments are the same, the durations between points at which measurements are taken may differ. The challenge is to develop consistency in both definitions and methodology and to develop reports at a higher level to incorporate operational variations. Issues Related to Comparability and Use. Some of the issues discussed under definitions clearly have an impact on the comparability of measures. However, even when the same standardized definitions are used, issues related to comparability remain. State Mental Health Authorities have different mandates and responsibilities related to populations covered and services provided. Some States provide services only to persons with serious mental illnesses and children with serious emotional disturbances (SED); others have a broader mandate. Some State Mental Health Authorities do not have responsibility for inpatient care. Some States can track only a segment of clients served by contracted providers. These differences make comparability and benchmarking across States a nontrivial issue. For measures that multiple States can report (e.g., penetration/utilization rates) the interpretation of results require including this additional contextual information. Issues Related to Significance and Weight of Various Measures. Measures reported in this study represent different concerns and aspects of care and recovery. No judgments or analyses have been conducted to assess whether one performance measure should be given more weight or significance than another. Even within a domain, how these performance measures should be weighted relative to each other is not defined. Recommendations for Future Developmental Efforts Reviewing the work in this study and looking ahead toward future efforts leads to the following recommendations:
REFERENCES Ganju, V., & Lutterman, T. (1998). The Five-State Feasibility Study: Implementing performance measures across State mental health systems. In R. W. Manderscheid and M. Henderson (Eds.), Mental Health, United States, 1998. DHHS Pub. No. (SMA) 99-3285. Washington, DC: U.S. Government Printing Office. Ganju, V., Beall, M. A., Callahan, N., Dailey, W., Dumont, J., Hornik, J., et al. (1996). The MHSIP Consumer-Oriented Mental Health Report Card. Rockville, MD: Center for Mental Health Services. Leginski, W. A., Croze, C., Driggers, J., Dumpman, S., Geertsen, D., Kamis-Gould, E., et al. (1989). Data standards for mental health decision support systems. Series FN No. 10. DHHS Pub. No. (ADM) 89-1589. Washington, DC: U.S. Government Printing Office. NASMHPD President's Task Force. 1998. Recommended Operational Definitions and Measures to Implement the NASMHPD Framework of Mental Health Performance Indicators. Alexandria, VA: National Association of State Mental Health Program Directors Research Institute, Inc. |
|
|
| Home | Contact Us | About Us | Awards | Accessibility | Privacy and Disclaimer Statement | Site Map |
