Programs & Organizations > Chronic Disease Prevention > A Guide For Local Health Departments > A Participatory Evaluation Approach

Preventing Chronic Disease

A Participatory Evaluation Approach

This Guide might have ended with a chapter on evaluation, but in our experience, it's best to think about evaluation right from the beginning. So we provide a few words about incorporating it into the planning process from the start. We offer further suggestions throughout the Guide.

Traditionally, evaluation occurs in response to a funder's mandate and focuses exclusively on whether the project accomplished the goals, objectives and activities outlined in a program plan. The evaluation is usually an internal process: agency staff, perhaps working with consultants, determine what will be studied and what methods will be used. The approach is usually to collect quantitative information, such as how many people participated in a program or attended a training session. The information may be shared in a limited way with the broader community.

The evaluation philosophy described here is quite different. It is based on a commitment to empower communities by making the evaluation process an ongoing, sustainable part of community planning and action. Our approach provides residents with skills and opportunities to help determine how the evaluation is designed. They decide what they want to evaluate and how they want to gather needed information. The residents participate in collecting and reviewing the information and help analyze and present the findings to the community. The information is used to monitor progress toward the community's goals and to adapt and sustain the effort.

The evaluation needs to be designed with feedback from all key stakeholders, including the community affected by the program, paid staff, volunteers and funders. Each stakeholder will have its own questions for the evaluation. A funder will probably want to see how many people received services, or whether certain materials were produced. Community members are more likely to care about the accessibility and quality of programs and the extent of neighborhood improvements that result from the efforts.

A good evaluation will attempt to balance the needs and interests of the various stakeholders. The process generally takes place at three levels:

Process evaluationdocuments the degree to which you were able to complete planned activities, documents obstacles encountered and efforts to overcome them, and gauges the success of attempted solutions. This is the most basic form of evaluation.
Outcome evaluation looks at whether the activities led to the changes that were predicted in the stated objectives. Questions for a chronic disease prevention effort might include: Did more young people participate in physical activity programs during the project period? Did the activities of the program contribute directly to achieving this objective?
Impact evaluation examines your success in meeting the overall goals you set. For example, did you reduce the incidence of chronic disease or death in the identified population? Did the community identify and mobilize to reduce chronic disease risk factors? This level of evaluation requires substantial resources since it requires that you measure baseline conditions and then measure changes after the project intervention.
You will also want to demonstrate that the changes you measured are attributable to the project intervention, and not to some other cause.

Challenges in Evaluation

There are three major challenges we have found in evaluating chronic disease prevention programs developed with the community:

Demonstrating improved health outcomes.
Documenting the benefits of developing a partnership with communities.
Finding the resources for a full evaluation, which requires considerable time and skill.

Demonstrating improved health outcomes can be difficult to accomplish in a relatively short time frame because the results of a chronic disease prevention program are, by definition, long-term. While mortality data for chronic diseases may be available at the county, city or zip code level, data at the neighborhood level is difficult to obtain. Incidence rates for chronic diseases are even more scarce, particularly for small population groups and geographic areas. As a result, it may be more feasible to track interim changes like:

Increased community commitment to organizing around reducing risk factors as demonstrated, for example, by tallying the number of residents who participate in carrying out health improvement action plans.
Actions that promote the elimination of risk factors, such as removal of billboards advertising alcohol or fast food.
Modifications in community habits, shown by indicators such as growth in sales of healthy foods at local stores, or an increase in observed or reported numbers of residents walking in the neighborhood.

Assessing the impact of your efforts to partner with the community and build community capacity also presents a challenge. High turnover among participants over the course of the project may make it difficult to develop sustainable relationships. The priorities of the community may shift as well, particularly in neighborhoods facing other, more urgent concerns. Activities, objectives and even goals of the project may change, making it hard to document the impact of your efforts.

To address some of these obstacles, you will need to:

Develop ongoing mechanisms for orienting and building skills and capacity of new community leaders.
Create an evaluation design that is fluid and emphasizes documentation of changes to project goals and activities.
Rely on creative methods for gathering information.
Use multiple methods for capturing information such as meeting logs, participant stories, background interviews, and visual images, such as photos of the neighborhood before and after the project intervention.

The final challenge is finding the time and skill needed for a full evaluation. If you lack the staffing or funds for a complete evaluation, you can still do a simple one. First, consider limiting your evaluation to a few, primary objectives. Most funders are not looking for an in-depth evaluation of each aspect of the program.

Some strategies to consider if your resources are more limited include:

Select a small number of the most important objectives to look at thoroughly, rather than taking a superficial look at all of them.
Work with an evaluator from another health department, local university or community program. You may be able to borrow or adapt their instruments rather than starting from scratch.
Consider a time-limited contract with an evaluation consultant, preferably one with experience in participatory evaluation with community stake-holders. Even limited help may allow you to examine outcomes for your most critical objectives.

Content provided by the The Community Wellness & Prevention Program of Contra Costa Health Services.