Using Participatory Methods to Evaluate Cumulative Impact

More often than not, organizations coming to Metis with evaluation needs have a specific program that they would like to have evaluated. However, some organizations approach us with a much taller request: how to evaluate the cumulative impact of their organization as a whole, across all programs and services. 

While cumulative evaluation employs many of the same evaluation strategies and methods as a program-specific evaluation, there are important considerations to ensure a worthwhile effort. In this article, learn how we approach conducting agency-wide evaluations, and how a strategic and participatory process can help any organization interested in making the case for their cumulative impact.  

Bringing a Cumulative Impact Evaluation to Life with a Community-Based Organization

In spring 2024, Clinton Hill Community Action (CHCA), a community-based organization in Newark, NJ, approached us with just this challenge: how to measure the impact of their multi-pronged, resident power-building approach to improving the quality of life of residents of the Clinton Hill community in Newark’s South Ward, a neighborhood with an active network of community organizations that are working to break the neighborhood’s cycle of intergenerational poverty. CHCA not only wanted to understand the linear relationship between its activities and impacts; but rather, they wanted to go deeper to know how all these efforts work in concert to create meaningful community change. 

CHCA’s commitment to critical participatory action research [PDF] (CPAR) added further (welcome!) complexity to the evaluation planning, encouraging us to rethink traditional evaluation methods so we could incorporate more voices into various stages of the research process. CPAR is an evaluation approach that challenges the power dynamic in traditional research by prioritizing community members’ experiential knowledge. This approach ultimately enhanced the cross-agency focus of the evaluation.

The Logic Model: Creating the Evaluation Roadmap

Metis developed a year-one evaluation plan that included multiple approaches for looking at CHCA’s specific interventions across its five priority areas while being attentive to cross-organization goals. One foundational activity we facilitated was creating a logic model, which provides a graphic representation of the relationship between a program’s resources, activities, and desired outcomes. To develop CHCA’s logic model, we launched a series of collaborative logic model workshops for CHCA staff to create a visual that would act as a road map for the entire organization. Specifically, CHCA uses multiple, overlapping programs and strategies to create change across its five established priority areas:

1. Community building
2. Housing and environmental justice
3. Food security and economic justice
4. Citizen journalism and advocacy
5. Arts and creative place-keeping 

Drawing from CPAR’s emphasis on including the voices of those on the ground, CHCA staff members from across the organization, most of whom are also residents of Clinton Hill, enthusiastically participated in an opening workshop that introduced logic models and asked for them to share their in-depth knowledge of CHCA’s offerings. Subsequent workshops, working meetings, “office hours,” and homework assignments led us to craft a logic model that synthesizes dozens of CHCA events, programs, and initiatives across the five priority areas, connecting their combined impact to specific short-, middle-, and long-term outcomes. This process was iterative, with CHCA listing their activities and desired outcomes while we consolidated, revised, and returned drafts until everyone was satisfied.

The resulting logic model groups all of CHCA’s offerings—from one-time community workshops to ongoing neighborhood revitalization efforts—into one of five different strategy areas:

1. Resource connections
2. Direct services
3. Community power building
4. Participatory action
5. Advocacy

These strategic actions are intended to lead to multiple, interrelated short-, middle-, and long-term outcomes across the organization and community. For example, short- and middle-term outcomes seek to make residents feel more empowered to advocate for their community and become more supportive of community efforts. Long-term outcomes seek to change community conditions and improve quality of life in key priority areas; create a sustained, robust network that is accountable to the community and advocates on its behalf; and increase residents’ pride, joy, and desire to stay in their community. As summarized by CHCA’s newly articulated Theory of Change, which encompasses the logic model, in a single statement: “If we provide residents with local resources and participatory opportunities to build power and advocate for themselves, then Clinton Hill will be an empowered, thriving, and connected community.”

Cross-Program Surveys and Participatory Action Research

We also worked with CHCA to develop and administer a cross-program survey that captures some of the core elements that span the organization’s work. The survey not only explores the impact of one having attended a specific session but also gathers information on how this and other CHCA events might change residents’ perceptions of their role in their community and their hopes for Clinton Hill’s future. These survey results will be shared in an accessible manner with strategy groups focused on CHCA priority areas and populated by Clinton Hill residents. The findings will provide critical data for residents to make informed decisions about future CHCA services, programs, partnerships, and research directions. 

As the distribution of these surveys becomes institutionalized, we are optimistic that analyzing findings will provide insights into the cumulative impact of CHCA events while providing useful formative feedback that can enhance future offerings and inform residents’ strategic planning. Change does not happen overnight, but having a cross-organizational instrument based on a solid logic model helps ensure that progress is consistently documented across priority areas and over time. 

Cross-organizational survey data collection is critical but does not replace the need for efforts that drill down to get more nuanced reflections from community members about specific issues they face—which, in turn, may reshape the activities or desired outcomes on the logic model. For example, CHCA staff recently used their CPAR training to facilitate resident-led discussions that provided a venue to identify housing issues and beautification challenges they can address together. They will employ a similar strategy for an ongoing evaluation of CHCA’s food justice strategy, for which Metis will conduct a series of interviews and focus groups with food pantry managers and clients, Mutual Aid Grocery Store customers and staff, and members of the resident-driven Food Co-Op Steering Committee. 

Surveys may tell us whether change is happening. But these participatory qualitative methods will help to reveal why we are seeing impacts in key areas by showing us what is happening on the ground—in the words of those most impacted. This kind of information is essential if we are to have a clear understanding of why we are or are not achieving agency-wide desired outcomes.

The Value of an External Evaluator

Mission-driven organizations often expend significant effort to articulate their mission and vision for what they hope to achieve—after all, that’s why they exist. But many spend far less time considering how they will know what progress they are making towards achieving those agency-wide goals. An external evaluator can help translate your vision into specific, measurable, indicators that capture the complexity of all that your organization does.

As our work with CHCA shows, our focus on developing a collaborative, participatory logic model created a framework for assessing cumulative impact going forward. We will continue to work with CHCA to gather actionable insights using a mix of methods that reflect the complexity of their multi-pronged approach to community development and empowerment.

While CHCA’s new logic model is already proving to be a useful tool for communicating and guiding their organizational approach, it is also a working document. As research and evaluation reveal that community conditions and residents’ needs are changing, so too will CHCA’s roadmap for serving them and facilitating advocacy evolve. Our iterative and participatory approach presents a solid foundation for ensuring that CHCA’s work continues to resonate with the community and support the achievement of meaningful, shared outcomes. 

Talk to us about your agency-wide evaluation needs.

If you are interested in learning more about how a customized cross-agency evaluation could help your organization, please reach out so we can start a conversation.