Services
Impact Evaluation Consultant Services
Impact evaluation assesses the changes directly attributable to a particular intervention, program, policy, or model. It has the very specific job of determining whether or not the outcomes are happening as intended. If you want your initiative to inform policy, practice, or funding decisions, building or strengthening your evidence base is essential—the right impact evaluation design can help you get there.
Impact evaluations are often used to establish or build an evidence base for the target intervention. Our lengthy experience designing and conducting rigorous impact evaluations has earned us a national reputation for meeting nationally-recognized evidence standards, achieved through randomized control trials and quasi-experimental studies. We speak the often highly-technical language of impact evaluation and distill and clarify it for our clients and their stakeholders and funders.
Experts in Rigorous Impact Evaluations
Our team includes evaluation consultants trained by the What Works Clearinghouse (WWC), the source of scientific evidence on education programs, products, practices, and policies trusted by federal funders. We stay up to the minute on the WWC’s standards and use rigorous impact evaluation methods to help our clients build a body of evidence with statistically significant findings to quantify their program’s outcomes, and hopefully, success. Our expertise includes the most commonly required impact evaluation designs for federal grants, including randomized control trial and quasi-experimental designs, to produce evidence of promise as well as moderate and strong evidence of effectiveness.
Our Impact Evaluation Services
We offer a full range of impact evaluation consulting services, including:
- Determining whether an intervention or program is ready for a rigorous impact evaluation
- Identifying key impact evaluation questions based on the program logic model or theory of change
- Selecting the most appropriate impact evaluation design (e.g., randomized control trial design, comparison group design, short-interrupted time series, regression-discontinuity design, etc.)
- Conducting a priori power analyses to determine target sample sizes for the impact evaluation
- Identifying valid and reliable measures for target outcomes, including baseline, mid program, and post-program data
- Using qualitative data to inform the design and interpretation of quantitative data
- Selecting the statistical analysis methods for each research question based on units of assignment and analysis, analytical sample size, dimensions of data structure, covariates, confounding factors, and outcome measurement scales
- Creating high-quality technical reports for evidence bodies such as the WWC, Substance Abuse and Mental Health Services Administration, and Arts Education Partnership, as well as research articles for peer-reviewed journals
- Expertise in the programmatic areas of youth development programming, college persistence and degree attainment, arts education and magnet schools.
Achieve Impact With Experts
Our impact evaluation consultants assess if your programs are making the difference you envision. Start a conversation with us today to learn more.
Our Approach to Impact Evaluation
Consultation
We embark on every impact evaluation engagement with a collaborative assessment of your needs and capacity to conduct this type of evaluation. If we can’t implement an appropriate impact evaluation design, we can work with you on alternate evaluation methods that still meet your needs, including less rigorous comparative designs.
Impact study design
We identify the appropriate design to determine impact—including quasi-experimental, randomized control trial, and regression discontinuity designs—and identify an appropriate comparison pool for quasi-experimental designs. We design procedures for minimizing attrition and incentivizing comparison or control groups.
Evaluation planning
We conduct document reviews, literature reviews, and initial program leader interviews to immerse ourselves in the program design and implementation. Working with stakeholders, we finalize the evaluation and research questions and develop or refine your program’s logic model.
Data collection
We identify the most appropriate data sources to address your evaluation questions, building and testing data collection tools, and collecting credible evidence. We have vetted strategies to tackle one of the biggest challenges of impact studies—collecting data from those not receiving the intervention.
Data analysis
We develop and execute a customized data analysis plan based on the types, quality, and completeness of the data collected. We complete multivariate analyses, such as logistic or linear regression, when covariates are included. When data quality is optimal, we derive impact estimates that can speak to the causal relationship between a program’s interventions and outcomes.
Why Partner With Metis Associates
Ethical participation
We recognize the dilemmas posed when conducting experiments in the real world and work with you to find ways to make participation ethical. This can include providing incentives for participation, waitlisting individuals not receiving a service for later participation, or offering services to control group members immediately after the study period. We can also pivot to rigorous impact designs that do not have the ethical implications associated with a randomized control trial evaluation, such as quasi-experimental matched group designs.
Digestible data
We deliver data and insights in a variety of deliverables to maximize stakeholders’ understanding of the evaluation results. We break down the technical jargon and communicate complex, multivariate data analyses in accessible language.
Feedback loops
Impact studies are often a one-way street where data are collected from clients to satisfy the funder. Our model creates a constant feedback loop to provide you with information for program improvement, so that the data can do more than check a box.
Our Impact Evaluation Consultants
Michael Scuello
Michael directs research studies with impact or rigorous designs and serves as the quantitative lead for many of our program evaluations of educational interventions. He is also steeped in evidence-based approaches to social and educational research and completed WWC training in using evidence-based standards to support our work. Michael serves as the chair of Metis’s Institutional Review Board, and he is a member of the NYC Department of Education IRB.
Susanne Harnett
Susanne has nearly 25 years of experience with research design, sampling methodology, field research, qualitative and quantitative methodology, data maintenance, and the design and implementation of randomized and quasi-experimental evaluation designs. Examples of recent impact studies she has led include those for Single Stop USA, the Center for Educational Innovation, and Studio in a School, among others. She holds a Ph.D. in Educational Psychology from the University of Virginia.
Our Impact Evaluation Case Studies
Bard High School Early College (BHSEC)
Metis studies the efficacy of one of the nation’s premier high school/early college models, uncovering consistent college success outcomes.
Case Study: Determining the Impact of Going to College Early
Community College of Philadelphia
Community College of Philadelphia contracted Metis to conduct a rigorous quasi-experimental study to examine impacts of their Single Stop USA program. Metis also conducted an implementation study to offer the best implementation practices and recommendations for program changes.
Children’s Services Council of Palm Beach County, Florida
Concerned about the pregnant women and families of young children who were declining or dropping out of services, the Children’s Services Council commissioned Metis Associates to conduct a survey of disengaged families.
Case Study: Keeping Families Engaged in Healthy Beginnings Services
City University of New York (CUNY)
Metis and the City University of New York joined forces to rigorously evaluate the ACE program, the baccalaureate version of the highly successful Accelerated Study in Associate Programs, a comprehensive program that supports students in reaching their graduation goals.
Real Stories of Impact
Our impact evaluation consultants assess if your programs are making the difference you envision. Start a conversation with us today to learn more.
Frequently Asked Questions: Impact Evaluation
Impact evaluations are necessary to determine how, and with what magnitude, interventions affect outcomes. In particular, impact evaluations are necessary to establish causal evidence for interventions, demonstrating that any observed outcome changes are due to the intervention and not other factors. There are many reasons to carry out an impact evaluation, including establishing an evidence base for an intervention, contributing to the academic field, meeting grant requirements, and/or justifying funding decisions.
Impact evaluations can be costly to develop and execute with no guarantee of positive results. Evaluations conducted on immature models or interventions, or interventions implemented with low fidelity, may not accurately depict their actual impact(s). High attrition and overlap between treatment and potential comparisons can also affect quality of findings, so a great deal of planning and monitoring needs to be conducted prior to and during the evaluation period.
Impact evaluations include meticulous data collection, cleaning, and monitoring plans as well as sophisticated analyses. For RCTs, treatment and control conditions must be monitored regularly to determine the potential effects of cross-overs, attrition, and missing data on future analyses. These activities take more time and resources than a standard program evaluation, which typically makes them more costly to conduct.