What Is an Outcome Evaluation? Benefits, Misconceptions, and How to Prepare

What is an outcome evaluation?

An outcome evaluation measures the extent to which a program or intervention is achieving its intended results, which could include shifts in knowledge, skills, attitudes, perceptions, and behaviors among participants or the community. 

There are different methodologies that can be used to measure outcomes, including surveys, interviews, observations, tests, and administrative or publicly available data. Types of analyses may include longitudinal analyses to track changes in outcome indicators over time, cross-sectional analyses to assess differences in subgroups at a given interval, and analyses to measure the association between intensity of programming (referred to as “dosage”) and outcomes, among others. 

Outcome evaluations can and often should be complemented with other types of evaluations. For example, a formative evaluation focuses on assessing an initiative or intervention’s implementation and documenting best practices, challenges, lessons learned, and recommendations for improvement. This type of evaluation can be particularly helpful to provide more context on the results you see from the outcome evaluation. An impact evaluation assesses causal links between a program or initiative and demonstrated outcomes. Impact evaluations are often conducted in the late stages of a program or intervention and typically involve a rigorous quasi-experimental or experimental design.  

In order for an outcome evaluation to be culturally relevant, key stakeholders must be involved in every stage of the evaluation process. Below are the most common critical junctures to involve your program stakeholders and examples of what that might look like:

• Creation of a learning agenda, research questions, and theory of change (ToC) or logic model (for example, through community stakeholder, program staff, participant, and/or funder meetings) 
• Development of data collection instruments (for example, through pilot testing)
• Collection and analysis of  data (for example with input/feedback from a Stakeholder Advisory Board)
• Interpretation of findings (for example through the use of data placemat activities)

Debunking 3 common misconceptions about outcome evaluations

Here are three common misconceptions regarding outcome evaluations you’ll want clarity on before tackling outcome evaluation for your initiative:

Misconception #1: There is a causal relationship between what we are doing and the results

Outcome evaluations may assess the relationship or correlation between program dosage or program services and observed outcomes, but they do not establish a direct causal relationship between an intervention and the outcome. In contrast, impact evaluations are designed to assess the causal relationship of the intervention through the use of rigorous designs including quasi-experimental and experimental studies. Still, outcome evaluations can yield critical insights, including the extent to which goals and objectives are being met, the changes that are taking place over time, and the differences in outcomes for specific groups of participants. 

Misconception #2: There is one yardstick per outcome

There is a growing field of evidence that not every outcome can be measured the same way for everyone involved in an intervention. Embedding an equity lens and understanding disparities in outcomes across various groups is of the utmost importance when planning an outcome evaluation. 

Consider standardized tests as an example. They’ve often been used to measure academic performance, but growing evidence suggests these tests are not a good measure for many types of students, including students of color or students from low-income households. Using course grades or portfolios may be better measures for these groups. 

Furthermore, outcome measures can be assessed in a number of ways. For example, you could frame your outcome indicators as a number or percentage of participants reaching a certain level or you could assess it in terms of growth (e.g., percentage of participants showing improvement). Understanding where your participants are at (baseline) before the initiative begins and how they may progress over time is key in order to determine how you will assess an outcome.

Taking the time to understand the community you serve, involving key stakeholders in the development of measures and data collection tools, and conducting reliability and validity analyses on the selected measures for all groups in your intervention, are all key steps you can take to conduct an equitable, culturally-relevant evaluation that will accurately depict and measure the success of your work. 

Misconception #3: If you work hard, you’ll see outcomes quickly

Organizations or programs are often eager to capture their success and want to measure key outcomes from the early stages of implementation. Yet, some outcomes may take several years to observe. Fixsen and colleagues (2005) indicate that it may take 2-4 years after a program has been fully implemented to start seeing positive outcomes. 

That’s why it’s important to identify short-term outcomes (e.g., changes in knowledge and attitudes) that are expected to lead to longer-term outcomes (e.g., changes in behaviors). Doing so will allow you to track progress in the earlier stages of a project’s cycle and ensure you are on the right track to observe the longer-term outcomes you expect to see. For example, a college readiness initiative working with high school freshmen may want to assess changes in participants’ knowledge of the college selection, application, and financing process first (1-2 years), followed by participants’ aspirations (2-3 years), and ultimately college enrollment patterns (3-5 years).

Why conduct an outcome evaluation? 4 benefits

There are many benefits to conducting an outcome evaluation.

1. Make data-informed decisions

Outcome evaluations help program leaders and managers track the short-term, mid-term, and long-term changes in key outcome areas the program is expected to impact, thus providing insightful information on who is benefiting, how they are benefiting, and under what conditions. 

Furthermore, by disaggregating the outcome data by participant subgroup, program leaders can better understand which groups of participants are faring better or improving faster, and which groups are not. You can use results throughout the cycle of a program or intervention to inform program development and improvement efforts, identify areas of concern, and make mid-course corrections.

2. Create an accountability tool

Outcome evaluations measure the extent to which a program or intervention is meeting its goals and objectives. You can communicate results to funders and program stakeholders, including the communities in which the program takes place, to show how the resources are being spent and progress towards meeting the goals and objectives.

3. Find unintended outcomes

Although it is best practice to have an a priori theory of change for the program or intervention that includes the short and long-term outcomes you want to measure, outcome evaluations can also lead to the discovery of unintended or unexpected outcomes. This information is critical to advance knowledge in the field. 

4. Get actionable data early on

There are often several different methodologies you can employ when conducting outcome evaluations, thus allowing for tailored analyses depending on the context for the intervention, data availability, and type of outcome measured. Outcome evaluations may also be conducted early on by tracking short-term outcomes, and build over time with longer-term outcomes, thus providing early, actionable findings and the opportunity to make mid-course corrections. 

How to prepare for an outcome evaluation

There are several important steps an organization or program can take to ensure a successful outcome evaluation.

Step 1: Create an a priori theory of change (ToC) or logic model to inform the evaluation

ToCs and logic models are visual representations of a program or intervention that promote a shared understanding of what a project aims to achieve and why. They play a critical role in informing evaluation efforts and ensuring those efforts reflect the assumptions and expectations of the project. These tools describe the following: 

• The context in which the program was conceived 
• The assumptions under which the program operates
• The resources, supports, and activities that have been or will be put in place to support implementation
• Short-, mid-, and long-term outputs and outcomes
• The relationships and connections among all these factors

Step 2: Develop SMART measures

An outcome can be measured through one or more measures or indicators; these should be Specific, Measurable, Achievable, Realistic, and Time-bound (SMART). Measures should also be valid (i.e., they accurately measure what they are supposed to measure) and reliable (i.e., they can measure consistently across time and/or groups), overall and for each group to be impacted by the program or intervention. 

Step 3: Assess your resources and internal capacity for evaluation

A question you’ll need to answer: Do you have the capacity to conduct the evaluation internally, or do you need to engage an external evaluator?

When assessing your internal capacity, you should determine whether your staff have the experience, expertise, time, and resources to conduct an outcome evaluation. You should also consider whether having an outside evaluator (often considered as an impartial expert) would be valuable. Some funders do encourage the use of external evaluations. And participants and program stakeholders may be more amenable to share their honest feedback with a third-party consultant who can ensure confidentiality.

Step 4: Create a comprehensive evaluation plan

Once your outputs and outcomes have been identified and operationalized, you will need to develop an evaluation plan. Your evaluation plan should include: 

• The context for the evaluation
• The learning agenda and research questions
• The ToC or logic model
• A matrix of outputs and outcomes, indicators, and the appropriate methods/data sources to measure these (If there are no available data sources, you will need to create the instruments or tools)
• Detailed processes and timeline for data collection, analysis, reporting, dissemination, and long-term planning 

Step 5: Involve key stakeholders in each stage of the evaluation

People who are immersed both in program design and on the delivery or receiving side of the program or intervention have intimate knowledge of program details and context that cannot be obtained anywhere else. Therefore, you should design evaluations in a participatory and culturally-relevant way. The evaluator should work collaboratively with project leadership, partners, and stakeholders (including participants receiving the services or intervention) in every step of the process, from the development of a learning agenda and theory of change to the creation and refinement of instruments and protocols, to the data analysis and findings interpretation phase.

Step 6: Refine the ToC or logic model and evaluation plan

If conducting a multi-year outcome evaluation, make sure to reflect on the learnings from each year, and discuss what worked and what did not work both in terms of program implementation and evaluation practices. Using these reflections, update the logic model or ToC and adjust your data collection, analysis, and reporting practices, as needed. Make sure to include any unintended outcomes you have discovered while conducting the evaluation.

Hiring an external evaluator

If you don’t have the internal capacity to conduct an outcome evaluation, hiring an external evaluator can help you access all of the benefits of this evaluation approach, including understanding whether the initiative or intervention is achieving results, for whom, how much, and in what timeframe.

If you’re finding yourself considering a partner to support your evaluation, please reach out. We’d love to start a conversation.

Get In Touch