This Web site is a component of the SAMHSA Health Information Network
|
This Web site is a component of the SAMHSA Health Information Network |
| | | | | | |  |
||||||||||
 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This Web site is a component of the SAMHSA Health Information Network. |
|
National Strategy for Suicide Prevention:
|
||||||||||||||||||||||||||||||||||
| As not everything can be done, there must be a basis for deciding which things are worth doing. Enter evaluation. |
| M. Q. Patton (1997) |
Evaluation is the tool we use to ensure that programs, such as those that are designed to prevent suicide, accomplish what we intend. Evaluation may answer certain questions that have been taken for granted but that have not been scientifically tested, especially those related to proving the effectiveness of a program. Evaluation may also be used to improve the functioning of a program. Both types of evaluation–sometimes referred to as outcome evaluation and process evaluation–can help to ensure effective use of resources.
Weiss (1998) posits four defining elements of evaluation:
A process evaluation focuses on implementation. It describes how a program operates, how it delivers services, and how well it carries out its intended functions. By documenting a program's development and operation, a process evaluation can provide some understanding of the performance of the program and information for potential replication. The goal of a process evaluation may be to ensure that a project stays on course and is faithful to the initial model. It may also be designed to provide the opportunity to make midcourse corrections, to modify aspects of the program that are not working as originally intended, or to identify problems or gaps that need attention. Process evaluation can help a project ensure accountability by comparing its actual performance with expectations and explaining reasons for any differences. Such information can help program administrators understand why some activities were more useful than others, leading to improved services in the future.
An outcome evaluation employs a causal framework; that is, an intervention is assumed to cause a particular outcome. This type of evaluation is used to study the effectiveness of a program. It employs quantifiable data to determine whether or not a program had the desired effects. Examples might include a reduction in the suicide rate or in attempted suicides, changes in knowledge among primary care physicians of treatment resources, or changes in the number of depressed people taking antidepressants. While evaluation is often thought of in terms of measuring overall effectiveness, frequently less comprehensive questions can be asked. For example, an evaluation might address the ability of an outreach program to actually contact people at risk and it might assess the cost of doing so; another might examine the way in which health provider characteristics affect the ability and/or willingness of these individuals to effectively engage persons at risk of suicide.
Steps in Conducting an Evaluation
The key steps in evaluation are as follows:
Engaging Staff and Stakeholders: Involving staff and stakeholders in an evaluation ensures that their perspectives are understood. If they are not engaged, the evaluation might overlook important elements of the program. Stakeholders can also help to implement the evaluation. They can improve its credibility and help the project address any potential ethical concerns.
There are several ways to involve stakeholders in an evaluation. These include consulting with representatives from as many groups as possible; developing an evaluation task force and including representatives of the stakeholder groups; and providing timely feedback on the process of the evaluation. An advisory committee might be formed to function throughout the life of the project.
The provision of feedback to project staff and other relevant stakeholders on the ongoing progress of an evaluation is often overlooked, resulting in missed opportunities to improve the evaluation and ensure that its findings are ultimately used by the field. Examples of ways to provide feedback include weekly meetings with program staff; monthly discussions or roundtables with a larger group; newsletters; and/or biweekly memos from the evaluator(s) on insights and reflections for response and comment. Ongoing dialogue and frequent communication are essential elements in ensuring that providers remain engaged in the project; such communication may also assist the evaluation team to refine the design and interpretations of the study.
Focusing the Evaluation Design: The evaluation question(s) drive the study. There are many potential questions that can be asked in an evaluation. Patton (1997) identifies 57 alternative ways of focusing an evaluation, each type with a different purpose and associated question–and these, he states, are illustrative only. Examples of ways to focus an evaluation and the types of questions relevant to each are shown in Table 1.
|
TABLE 1 Focusing an Evaluation |
||||||||||||||||||||||||
|
||||||||||||||||||||||||
After defining one or more important questions, the program and evaluation team must then determine whether or not it is possible to answer them. Perhaps a question cannot be clearly stated or its elements adequately defined. Or perhaps there is not a methodology that can be used to answer the question. Or, while it may be theoretically possible to design an evaluation study to answer a particular question, it may be quite expensive to conduct the study and sufficient funds may be unavailable. Determining whether or not a question can be asked clearly, whether there is a way to study it, and whether there is sufficient money to undertake an appropriate study is sometimes referred to as an "evaluability assessment."
Many people now use a "logic model" as a way to identify evaluation questions. A logic model is simply a diagram (perhaps a flow chart or a table) that shows the relationships between program elements and presumed outcomes; it represents the theory of how and why the program is assumed to work. By developing such a diagram, program stakeholders can sometimes clarify areas of particular interest for evaluation. An example of a completed logic model is included at the end of this discussion.
Once the questions for the evaluation have been determined, the project team must design the methodology. Decisions are made on issues such as the specification of groups that will be studied, the means by which groups will be selected, time intervals for study, the kinds of comparisons that are planned, and the form in which data are to be collected. Either qualitative or quantitative data may be collected, sometimes both. An evaluation question that addresses proving effectiveness, for example, will usually require a formal research design that includes a control group and the development of quantitative measures, but a question that is concerned with understanding a project's responsiveness to cultural issues will most likely employ methods such as interviews and focus groups.
Gathering Evidence: As a part of the study design, the evaluation team will need to decide on the instruments for collecting it. Survey questionnaires, interview protocols, and coding forms are examples of instruments. In some cases, it is possible to use preexisting instruments; in other cases, the evaluator will need to develop a new instrument. An advantage of existing instruments is that they are often (but not always) standardized (i.e., scores on particular items have been rated as "normal" and "non-normal"), and they may have been established as valid and reliable (valid means the instrument measures what it is supposed to measure and reliable means that responses are consistent over time). The disadvantage of using existing instruments is that they may not be appropriate for the particular program being evaluated. For example, an instrument may refer to services not provided through the program, or it may be inappropriate for the cultural or ethnic groups that make up a community.
Justifying Conclusions: In the data analysis phase of evaluation, the information is interpreted and a judgment made about the meaning of the data that has been collected. What are the answers to the questions that have been posed and what do these answers mean?
Generally, some standard will be used to judge the meaning of the findings. For example, if one of the desired outcomes of a program is the institution of or improvement in outreach services, a number by itself will have little relevance in the absence of a standard. Is an outreach program successful that reaches 15 percent of the population? The answer depends on what the program and the community define as adequate and appropriate. When diverse stakeholders have different standards, they may disagree on the conclusions that may be drawn from the data analysis.
Ensuring Use and Sharing Lessons Learned: Evaluation is only worth doing if it leads to improvements in knowledge and program operations. There is both a local and a universal component to utilization of evaluation findings. Evaluation should be important first of all to the stake-holders of the particular program that was evaluated; evaluation findings should inform programmatic decision-making and address questions that are important to program staff and service recipients. Engaging stakeholders throughout the evaluation process helps to ensure an evaluation that is relevant to the program and that may lead to changes in procedures and policies, if necessary, or to enhanced support for the program.
The second audience for evaluation is outsiders with an interest in the issue. Findings may help to improve the functioning of related projects, convince policy makers of the importance of the program, and generate wider support for the program. Evaluation findings presented in the media can increase public understanding.
ConclusionThis discussion has provided a very brief overview of some issues related to evaluation. It is intended to provoke thought and to suggest the importance of evaluation for suicide prevention. More detailed information on evaluation may be found on the Web sites and in the books listed below.
Useful Web Siteshttp://www.cdc.gov/eval
This site, supported by the U.S. Department of Health and Human
Services, Centers for Disease Control and Prevention, provides a description of the public health approach to evaluation in a clear and straight-forward
manner. It includes links to other Web sites with additional
information on program evaluation, including numerous on-line publications that can be downloaded.
http://www.bja.evaluationwebsite.org
This site, supported by the U.S. Department of Justice, Bureau of
Justice Assistance, provides a primer on evaluation. While the examples
are oriented to projects of the Department of Justice, the text is generic
to evaluation of community-wide programs.
http://www.wkkf.org/
This site includes a downloadable version of the excellent evaluation
handbook developed by the W.K. Kellogg Foundation for its grantees. It
provides much useful information for evaluating projects that are com-munity-
based.
Afifi, A.A., Clark, V. (1990). Computer Aided Multivariate Analysis (2nd
ed.). New York: Chapman and Hall, 1990.
A text on linear models and regression analysis.
Centers for Disease Control and Prevention. (1999). Framework for program
evaluation in public health. Morbidity and Mortality Weekly Report,
48, 1-40.
A description of the elements of a good evaluation for public health
programs, including standards for effective evaluation.
Kvale, S. (1996). InterViews: An introduction to qualitative research interviewing.
Thousand Oaks, CA: Sage.
A comprehensive book written for practitioners but also useful for
researchers.
Lunnenberg, C. (1994). Modeling experimental and observational data.
Boston: PWS-Kent.
A text on linear models and regression analysis.
Maxwell, J.A. (1996). Qualitative research methods. Thousand Oaks, CA:
Sage.
A comprehensive look at qualitative research.
Patton, M.Q. (1997). Utilization-focused evaluation (3rd ed.). Thousand
Oaks, CA: Sage.
A clear and readable text providing good guidance on evaluation
strategies that can lead to useful–and used–evaluations.
Rossi, P.H., & Freeman, H.E. (1993). Evaluation: A systematic approach
(5th ed.). Thousand Oaks, CA: Sage.
A highly regarded text on evaluation.
Weiss, C. H. (1998). Evaluation methods for studying programs and policies
(2nd ed.). Upper Saddle River, NJ: Prentice Hall.
An easy-to-read and comprehensive text on evaluation.
Weitzman, E.A., & Miles, M.B. (1995). A software sourcebook: Computer
programs for qualitative data analysis. Thousand Oaks, CA: Sage.
A sourcebook addressing the relative advantages of different computer
programs for qualitative analysis.
NSSP Homepage | Table of Contents | Previous | Next
 Substance Abuse and Mental Health Services Administration |
 Centers for Disease Control and Prevention |
 National Institutes of Health |
 Health Resources and Services Administration |
|
| Home | Contact Us | About Us | Awards | Privacy and Disclaimer Statement | Site Map |
