What Is an Impact Evaluation and Is Your Program Ready?

Impact studies can provide program collaborators with key information about the effects of the work, which is crucial to the success and sustainability of a program. These studies also require considerable resources and don’t have a guaranteed outcome—you and your collaborators may not learn what you had hoped to learn. If your program team is thinking about engaging the program in an impact study, the question to answer is, “How do we know if we’re ready?”

What is an impact evaluation?

An impact evaluation or study uses a rigorous design that examines causal connections between program inputs and outcomes for the intended participants. Any one study will not “prove” causation—causal connections range from the lowest levels of confidence of causation to the highest levels.  

The more rigorous the evaluation design, the higher the level of confidence you and your program colleagues can have in causation. Rigorous designs include some type of “counterfactual,” that is, a control or closely-matched comparison group that would allow for examination of what would have happened if not for the program’s intervention.

Why do an impact evaluation?

Program teams are often interested in conducting impact studies for several reasons. Considerable time, effort, and other resources go into the community, education, and youth programs that federal grants fund. Naturally, the implementers expending those resources want to know if their program is working the way it was intended and to be able to draw causal connections between implementation and impacts, both short- and long-term, for participants. An impact evaluation can help provide those answers.

Additionally, funders often require evidence of impact, and programs are often beholden to provide evidence to maintain implementation. Without that evidence, many programs can’t continue to function. 

Programs are also often incentivized to become “evidence-based” so they can be part of best practice databases such as the What Works Clearinghouse (WWC) and the SAMHSA Evidence-Based List. Being part of these lists may increase the program’s recognition in the field, allow for greater access to funding, and increase opportunities for expansion and replication. 

3 Methodologies for examining impact

There are three major types of methodologies that can be used to conduct an impact study. Each of these methods allow for opportunities to examine the counterfactual.

1. Randomized control trial (RCT) design

The RCT design (also called an experimental design) is considered the gold standard in research. It involves the true randomization of units (e.g., individuals, classes, schools) into either a treatment or control group. The treatment group participates in the intervention, while the control group engages in “business as usual.” 

2. Quasi-experimental design (QED)

When it isn’t possible for logistical or ethical reasons to conduct an RCT, it is also possible to examine impact using a QED, which involves the use of matching procedures to select a comparison group that matches the treatment group, rather than randomizing units into treatment and control groups. While there are multiple ways to select a comparison group, it is important to ensure that the groups are well matched on all relevant variables prior to conducting the intervention.

3. Regression discontinuity design (RDD)

A third—and much less commonly used—methodology for examining impact is an RDD. An RDD can be used with programs that have a cut-off score for determining eligibility for participation. For example, students may be eligible to participate in a particular academic program only if they achieve below a certain score on an achievement test. 

Theoretically, if the program is effective, there should be a sharp upward jump in outcomes for students who participate in the program as compared to those who were near the cutoff point but were not eligible to participate.

Risks associated with doing an impact evaluation

While the advantages of an impact evaluation are clear, there are risks associated with doing one, especially if the program implementers aren’t ready yet. Here are some of the potential risks of doing an impact study:

• Inconclusive. Impact studies can be inconclusive or paint a misleading picture about the cause and effect relationship between a program and its impact if you and your colleagues don’t time the study correctly and/or don’t have the resources to do it right.
• Demanding. Impact evaluations can be logistically challenging and time-consuming for program staff members who often wear multiple hats and find themselves stretched thin. 
• Costly. You may spend time and money only to find no quantitative impacts or that the program design does not have the proper structures in place to even complete the study. 

For these reasons, ensuring that the program is at the right stage and that collaborators are ready to embark on an impact evaluation will increase the likelihood that everyone’s time and financial investment will result in helpful information for the program and, ultimately, contribute to its long-term success.

How to know if a program is ready for an impact evaluation

It is essential for program teams to consider the following criteria prior to embarking on an impact study. These are the criteria we look at to assess if you and your collaborators are ready to do this type of evaluation.

Does the program have a theory of change and/or logic model?

Before evaluators can measure impacts, the program should have an articulated theory of change or logic model and have it agreed upon by program collaborators. The theory of change or logic model should include clearly-defined expected outcomes for participants and theorized connections between specific program activities and inputs and participant outcomes. It is likewise vital that the program collaborators agree upon the expected outcomes and the theory of change. We have found that program teams may often make assumptions that everyone agrees on the underlying theory of implementation. However, if this is not formally articulated and discussed, differences in understanding may not come to light. 

Does the program have the tools to assess participant outcomes?

Once the program team articulates and agrees upon the expected outcomes and their connections to inputs, the next step is to ensure you have the appropriate instruments to measure the outcomes. The instruments should demonstrate adequate validity (evidence that they measure what they are intended to) and reliability (that they do so consistently) with the target populations for the program. Additionally—for an impact study to meet WWC standards—the instruments should have been tested within the program already.

Does the program have a well-established approach to implementation?

Prior to pursuing an impact study, the approach to program implementation should be well-established. Too often, program staff are pushed into examining impacts while they are still developing program components and determining what “fidelity” looks like for their program. Implementation studies answer questions like, “What are the key aspects of program implementation?”, “How does implementation vary within and across sites?”, and “What are the successes and challenges of implementation?” Results from these studies should lead to tweaks to program implementation and definitions of fidelity.

Does the program show promising results?

Before embarking on an expensive and time-consuming impact study, it is wise to conduct at least one outcome study. This study can look at outcomes for participants without examining the counterfactual—in other words, it can answer descriptive questions rather than causal ones.

Outcome studies can include questions such as, “On average, how do students score on the assessment following their participation in the intervention?” They can also look at whether differences exist between pre and post scores without addressing causation. An example of this type of question could be, “To what extent do students increase their scores on the assessment following their participation in the intervention?” 

Moreover, “dosage” questions can be examined, which will help define fidelity, as they address the question of “how much” intervention is needed to show promising outcomes. An example of this type of question may be, “To what extent do students who have greater program participation outperform those students with less participation?” and “What is the minimum number of hours that students need to participate in the program to show growth in the outcome?” 

If the outcome study shows promising results, it makes sense to undergo a more rigorous impact study to test for causal effects. If the outcome study does not result in promising findings, it may make sense to revisit implementation or expected outcomes before further pursuing an impact study. Moreover, the dosage analyses described above can inform the fidelity of implementation definitions and guidelines.

Does the program have the resources to complete an impact evaluation?

In addition to programmatic considerations, it is essential to keep logistics in mind as well, specifically:

• Finances. The cost of an impact study is an important factor to consider. The exact cost will depend on the specifics of the study. For example, it will be more costly if there are multiple school districts or entities involved, if the data are difficult to obtain or interpret, and if the study includes triangulation with qualitative data sources. At Metis, we have found that the cost of an impact study can range from 20-30% of the total cost of the project being evaluated.
• Staffing. Impact evaluations require dedicated staff time to the study. This includes time at the onset to agree on the study design, gathering data and interacting with program colleagues, reviewing instruments and other protocols, reviewing and interpreting study results, and convening ongoing meetings to discuss implementation updates and brainstorm through roadblocks.
• Strong partnerships. Non-profit organizations that partner with schools understandably want to conduct impact studies of the non-profit’s programs. However, the partner schools and districts must also be on board with the study. For this reason, non-profit organizations that are working with school districts should ensure that the districts are equally invested in the study before pursuing an impact study.. 

Partnering with a professional evaluation consultant

Despite their stringent readiness conditions and considerations, impact evaluations are valuable tools for understanding the effects of a program and improving your team’s decision-making processes. Partnering with a professional consulting firm, such as Metis Associates, can help guide you and your collaborators through the processes of preparing for and undergoing an impact study. 

We recommend reaching out to an evaluator before getting too far down the road in planning an impact study. Partnering with an evaluator during the readiness period will not only ensure that the program has the required components in place but will also allow the evaluator to get to know the ins and outs of the program and for both parties to develop a strong and trusting relationship.

Learn more about how Metis Associates conducts impact evaluations, or reach out to start a conversation.

Get In Touch