Monitoring outcomes for program success

Evaluation can feel intimidating if you have not done it before, but it does not need to be. Treated as a routine part of program planning, it tells you what is working, shows funders the difference your program makes and helps you do more for the people your participants serve. This guide walks through the steps of planning an evaluation, a seven-level model for choosing outcomes, the data we recommend every ECHO program collect and ways to keep learning from what you gather.

Why evaluate?

There are three broad reasons to evaluate your ECHO program:

• Program improvement. Continuous improvement asks what is working and what needs attention. If you are not reaching the learners you set out to recruit, for example, your data can show where to pivot: new regions, new professions or new outreach channels.
• Outcomes and essential elements. Is your program moving the outcomes you care about? Which parts of the ECHO Model produce the impact: the didactics, the case presentations, the peer-to-peer discussion? And what does ECHO contribute when it is one piece of a larger intervention?
• Funding and sustainability. Evaluation is central to funding. It is almost impossible to secure or renew support without evidence that the program works.

Evaluation is also one of the 10 essential public health services, so building it in from the start puts your program on the same footing as the systems it aims to strengthen.

Follow a planning cycle

We use the CDC’s Framework for Program Evaluation in Public Health. It has been around for more than 20 years and was updated in 2024. It is simple and iterative, and every step matters. The rest of this article follows its six steps.

1. Assess context. Understand the program’s setting, who has a stake in it and whether it is ready to be evaluated.
2. Describe the program. Lay out what you invest, what you do, who you reach and what should change.
3. Focus the evaluation questions and design. Decide what you need to know and how rigorous the design should be.
4. Gather credible evidence. Collect data with methods that fit your questions and your capacity.
5. Generate and support conclusions. Analyze and interpret the results against clear standards.
6. Act on findings. Share what you learn and use it to improve the program.

At the center of the cycle are standards every evaluation should meet: relevance and utility, rigor, independence and objectivity, transparency and ethics. Around the outside are three practices that run through every step: engage collaboratively, advance equity and learn from and use insights.

Equity and cultural humility

Evaluation is not a neutral activity, particularly when you work with communities that have been under-resourced or have experienced trauma. In research settings we tend to assume we are objective, but evaluation and research are social processes, subject to the same biases as anything else. Who you engage, which questions you ask, what you measure and how you report results are all choices, and none of them is neutral.

Consider culture, race, ethnicity, power, trauma and voice at every point in the process: stakeholder engagement, program design, evaluation questions, measures, data collection and analysis, reporting and your choice of evaluation consultants. These resources are a good foundation:

• Equitable Evaluation Framework from the Equitable Evaluation Initiative
• Guiding questions for culturally responsive evaluation from MDRC
• Public statement on cultural competence in evaluation from the American Evaluation Association

Step 1: Assess context

Assessing context is the first step in the cycle, yet teams often engage their stakeholders last. Bring them in early. Even participants have useful views on which outcomes matter and which are feasible to measure, and they may know about data you could use.

Look at four components of context:

• Readiness for evaluation. An evaluability assessment asks whether the program is ready to be evaluated at all.
• People. Map the interest holders: who is affected by the program, who influences it, who wants to know the results.
• Place. Document the program’s environment and features.
• Evaluation capacity. Assess your team’s capacity to evaluate, and reflect honestly on your own readiness as an evaluator.

Stakeholders for an ECHO program typically include:

• Funders
• Leadership at your institution
• Partner organizations
• Program staff
• The hub team and presenters
• ECHO participants
• Professional groups
• “Consumer” groups: the patients, students or community members your participants serve

Step 2: Describe the program

Be clear about what you are doing. A logic model lays out the “logic” of your program: what you invest, what you do, who you reach and what changes as a result. Working through one is a great way to engage stakeholders. Even a simple program looks different from different perspectives, and surfacing those differences early is productive. Come back to the model over time as the program grows.

The template is deliberately broad. Make it specific to your program, especially if you will use it to communicate with external stakeholders. Its five columns are:

• Inputs: what you invest. Funding, the hub team, participants and partners, plus any best practices, guidelines or curriculum the program is built on.
• Activities: what you do. Recruitment, regular free virtual sessions, brief didactic presentations, case presentations and recommendations, facilitated two-way learning between peers and experts, and on-demand recordings and resources. If other interventions are connected to your ECHO, include them so the context is clear.
• Outputs: who you reach. Sessions held, unique participants, participant characteristics, attendance rates and satisfaction.
• Short-term outcomes: learning and action. Increased knowledge, skills, confidence and self-efficacy; improved use of evidence-informed practices; decreased professional isolation and improved job satisfaction.
• Long-term outcomes: change. Improved outcomes for end beneficiaries such as patients or students, improvements in the organizations and systems that support evidence-informed practice, better population measures and improved workforce retention.

Example: Improving Perinatal Health ECHO

This program’s model names the best practices it promotes (the AIM patient safety bundles) as an input, and it ties a specific, observable practice change, less stigma and less stigmatizing language toward pregnant and postpartum people who use substances, to the patient and community outcomes it hopes to see. That specificity is what makes the outcomes measurable.

Choose outcomes with Moore’s seven-level model

Deciding on outcomes is often the hardest part. We use Moore’s expanded outcomes framework, developed for continuing medical education to get beyond “smile sheets” to more meaningful results. The pyramid builds from the bottom up: each level depends on the ones below it.

Most ECHO programs target Levels 1 through 4 and rely on subjective measures, asking participants to rate their own change in knowledge and competence. Level 5, performance, is where ECHO moves into the real world, and it is the linchpin of the whole model: if participants do not change what they do, there is no path to end-beneficiary or population outcomes.

• If your program is established, try to measure practice change. The methods in Step 4 will help.
• If your program is new, think now about how your curriculum will influence the practice changes you want to see, and link sessions to those changes.

Step 3: Focus the evaluation design

With stakeholders engaged and a logic model in hand, decide how rigorous the design should be. Designs usually get more sophisticated as a program matures, or sooner if you already have evaluation experience or a dedicated evaluator on the team.

Step 4: Gather credible evidence

Once the design is in place, collect the data. These are the standardized methods the ECHO Institute has developed. Use the linked templates as they are or adapt them to your program.

A few notes on the less familiar methods:

• Hub team self-reflection asks whether your team is following ECHO best practices. Build in time to review the data you already have instead of waiting for a report. Many teams meet regularly to walk through their dashboards together.
• The follow-up case presentation survey goes to the presenter a month or two after their case, to learn whether the recommendations were useful and what changed for the patient, student or system.
• Mini-focus groups surface benefits, barriers and changes in practice and systems that a survey can miss.

Recommended data

At a minimum, we recommend every program measure satisfaction, knowledge and relevance after each session. The questions below come from the feedback survey templates, which are built into iECHO (see Conducting a session feedback survey) and are also available in PERL.

Measuring practice change

Start with one question: what do you want participants to do differently because of ECHO? Answer it broadly first, then break the answer into specific pieces. “Manage opioid use disorder in primary care” is a goal; “prescribe buprenorphine” and “screen every new patient with a validated tool” are practices you can measure.

If it feels too early to answer, that is fine. Programs are at different stages, and defining the practice change you want is part of program planning. Doing it well is progress toward evaluation. Ways to collect practice-change data include:

• Surveys, and focus groups or interviews
• Electronic health records
• Quality improvement metrics

Step 5: Generate and support conclusions

Analysis, qualitative or quantitative, is driven by the question at hand and the data you have. Interpret the results against standards you set in advance:

• Outcomes. Did you meet the bar you set for yourself, such as a significant increase in knowledge from before to after the program?
• Values. State them clearly. Which findings count as positive or negative given your program’s goals?
• Stakeholders. Do the findings mirror their experience and expectations? If not, why not?

Recommendations should be consistent with your conclusions and appropriate given the limits of your findings. Would your program have the same effect in a different context? Say what will matter for success, or failure, going forward.

Step 6: Use data for learning

It is easy to report only to funders. Getting data to your program team more often makes it far more useful. For each audience, decide on the content, format and frequency:

• Audience. Funders, leadership, the hub team, participants, partner organizations, the community.
• Content. What each audience needs to know to make a decision or feel the program’s value.
• Format. Dashboards, written reports, a slide at a team meeting, a note in a session.
• Frequency. Quarterly for a team review, annually for a funder, whatever keeps the data in use.

Close the loop by sharing what you learn with participants, their organizations and the communities you serve. They gave you the data; show them what it changed.

Dashboards in iECHO

The iECHO Analytics Center gives your hub team ready-made views to review together: attendance, participant insights and session feedback. Start with the Overview dashboard, then dig into the Participants and Forms dashboards. Putting one of these on the agenda of a regular team meeting is the simplest way to build a habit of learning from your data.

Building on evaluation: ECHO research

There is no hard line between evaluation and research. They use the same tools and often ask the same questions; research usually takes on more complexity and rigor. More than 900 peer-reviewed publications on ECHO are cataloged on the ECHO publications dashboard, and their distribution across Moore’s levels mirrors what most programs measure.

Patient outcomes in the literature

Research pushes toward objective measures: claims data, electronic health records, chart review, patient surveys and other programmatic or administrative data sets. Most patient-outcome studies so far come from hepatitis C, diabetes, HIV and pain management. The outcome measures they used may suggest what your own program could track.

Research opportunities

If you want to move from evaluation into research, a few things help:

• Build on your evaluation foundation. Rigorous studies depend on structures that already collect data reliably.
• Build partnerships. Most ECHO research involves more than one party: hubs working together, or a hub paired with a researcher who has content expertise. ECHO collaboratives are a good place to find programs working on similar questions. You do not have to figure it out alone.
• Seek research funding. Rigorous designs usually need support beyond the program budget.
• Prioritize gaps in the evidence base. Even with hundreds of studies, there is little on end-beneficiary outcomes, randomized controlled trials, education ECHOs, programs in low- and middle-income countries and the essential elements of the ECHO Model. Filling a gap improves your odds of funding and publication.

References and resources

• Centers for Disease Control and Prevention. (2024). Framework for program evaluation in public health. MMWR Recommendations and Reports, 73(RR-6), 1–37.
• Moore, D. E., Green, J. S., & Gallis, H. A. (2009). Achieving desired results and improved outcomes: Integrating planning and assessment throughout learning activities. Journal of Continuing Education in the Health Professions, 29(1), 1–15.
• New York Academy of Medicine. (2017). Project ECHO Evaluation 101: A practical guide for evaluating your program.
• PERL: the ECHO Institute’s library of evaluation templates, including the logic model and survey instruments linked above.
• ECHO publications dashboard: peer-reviewed research on ECHO, searchable by topic and outcome level.

Questions about evaluating your program? Contact the ECHO Institute’s research and evaluation team at jljones8@salud.unm.edu.