Four Time-Tested Strategies for Monitoring and Evaluation Success

Contributed by Jacob Allen, Senior Partner and Managing Director, Social Impact Practice and Callie Kennel, Engagement Manager at Cicero Group.

Monitoring and evaluation (M&E) plays a critical role in impact-driven organizations. Whether you are an executive director or an administrative assistant; if you distribute $50 a year or $50 million; if your work is international or domestic, your impact matters. The metrics matter. How you define progress and change matters. M&E influences how we look at the problems we face, and how we can truly partner with grantees to ensure we achieve our shared mission.

At Cicero Group, we’ve spent the last 20 years helping organizations develop, execute, and refine their M&E practices. Our objective in this post is to share the biggest lessons we’ve learned along the way, including: 

• How to recognize successful M&E

• Using our tried-and-true M&E framework

• Addressing four common M&E tensions funders and grantees face

How to recognize M&E Success

From working with nonprofits and foundations of all sizes, we’ve found that there are five main components to successful M&E. The first is mission. It needs to be aligned with your strategy so it can advance your purpose. It does that by being systematic. You must be deliberate and intentional with your approach so M&E can deliver the correct insights. Insights can be both quantitative and qualitative, but they need to help you understand your immediate, short-term, and long-term impact. Insights help prove and improve your impact. Proving your impact provides evidence that you are creating the intended change, while improving your impact ensures that the programs driving change are as effective and efficient as possible.

Our tried and true M&E Framework

Successful M&E strategy, design and implementation can be broken into four distinct pieces. 

1. Theory of Change 

This identifies the population you are striving to make a change for, what those changes are, and how you will pursue the intended impact. Some questions to consider when developing or refining a Theory of Change are: 

• Audience: Who are we serving? 

• Desired Change: What do we ultimately want to be better for this population? 

• Levers of Change: What are the programs and resources necessary to create this change?

2. Design

Deciding on the correct metrics and measurements will be instrumental in accurately proving and improving your program effectiveness. In order to do this, you must first define your objectives:

• What decisions will benefit from measurement data? 

With objectives in mind, you can begin to determine the metrics, also referred to as output and outcome indicators: 

• Outputs: What volume of (resources, activities, programming) will lead to the changes in the target population?

• Outcomes: What are the changes in the population? 

3. Systems Development

This is where things tend to get more complicated. Most organizations get to this third step and realize they have not taken the time to think about what data collection will look like. Systems development forces us to take a step back and develop the required roles, tools, and processes to gather, access, and learn from the data. Taking each of those in turn, this step requires answers to the following questions:

• Roles and responsibilities: What capabilities and resources are needed for effective M&E?

• Tools: What tools are necessary to facilitate the evaluation design?

• Processes: What processes facilitate the collection, analysis, reporting, and use of this data?

4. Execution

Execution ties Theory of Change, Design, and Systems Development together to answer a final question: 

• Is your M&E effective? 

The answer lies in two parts: 

• Effectiveness: Effectiveness considers if your levers of change have achieved the intended impact and what should be changed to improve your impact.

• Efficiency: Efficiency looks at how you can make M&E simpler, faster, and less costly. How can you increase the value we get from M&E? 

Don’t overlook execution. It takes a team with the right capacity to execute a strategy that drives impact.

Addressing four common M&E tensions funders and grantees face:

1. When is it appropriate to require grantees to report on ‘our’ outcomes using ‘our’ methodologies or metrics? If this is ever appropriate, how do we do this without creating undue burden, diminishing impact, or exacerbating power imbalances?

Organizations typically see this as a simple “hands-off” or “hands-on” approach. However, in our experience, we’ve noticed that a more nuanced approach that centers around consistency of the data and collaboration with grantees typically leads to greater impact.

There are three rules of thumb that you can follow when you’re thinking through whether to have grantees report on metrics, how collaborative you want to be, and how consistent you want your data:

• Avoid extremes. M&E is not an all-or-nothing approach. Rather, you need to determine what’s the rule and what’s the exception for M&E and work to avoid one becoming the other. 

• Ensure M&E benefits everyone. There are multiple stakeholders across your M&E spectrum. Every single person who has touched your M&E process should benefit in some way. 

• Invest in learning and improvement. M&E is a baby step system. Start from where you are, know where you want to go and take small steps to build up to your end goal. It does not need to be an overnight change!

2. How do we measure our impact across multiple outcome areas or aggregate our impact across all grantees?

Issues get complicated when connecting not just ‘smaller’ and ‘bigger’ things, but different things – aggregating apples and oranges. When you add in multiple areas of focus, many potential outcomes can arise.

To better conceptualize this, let’s think of a funder whose ultimate goal is health equity. Obviously, some metrics associated with health equity might include improved access and improved quality of care. But we also know that secondary factors like education, location, and income can play a major role in health equity. So, while one grantee might report on the number of clinics accessed, another grantee could report on increased 3^rd grade reading rates. 

When analyzing data, you will look at the collective impact the grantees had on the common metric or outcome to determine if there was improved access across the grants that address these specific metrics. Then you can determine if health equity or access was improved because of these grants. The challenge is to work through that complexity, to try to get to something that is even cleaner and simpler.

There are various keys to success when it comes to measuring across multiple areas:

• Focus on a small set of ‘big’ outcomes: this allows everything else to be seen as a contributing factor.

• Gain clarity about what the program areas are and set boundaries between them.

• Define the ‘equation’ that explains the impact relationship between elements.

• Track the impact elements that relate to the equation.

• Develop dynamic statements that enable reporting at different levels of the process.

However, consider the limitations that come with this. Measuring the ultimate outcome can be difficult or impossible so you must be comfortable with broad information. You may need to constantly clarify your tools, processes, and roles along the way. Sometimes you are drawing an M&E boundary when there is not a real-world boundary.

3. How do you measure a funder’s impact on long-term change or nebulous activities like advocacy or field building?

In a basic theory of change, things tend to get fuzzy if the activities, outcomes, and metrics involved are not clearly outlined and defined. To avoid the “then a miracle occurs” trap, you must start with a robust theory of change that extends all the way to your desired ultimate outcome. 

This requires you to be specific about required key areas for the long-term or systems-level change.

• Be mindful of the types of outcomes that exist: Nature of Change vs. Unit of Change.

• Be realistic about what you can directly influence and hold yourself accountable for.

• Measure direct impacts systematically and, when possible, quantitatively. Find ways to measure indirect or longer-term developments as best as possible. 

• Use existing research to bolster the connection between the two sets of outcomes that are beyond your ability to directly measure. This will help tie your outcomes together and paint a bigger, overarching picture that shows cause and effect. 

As always, assess, learn, adjust, and refine over time. Be data-driven and strategic as you assess the likelihood of change in and degree of your influence on your ultimate outcomes; adjust your M&E, activities, timeline, and resources accordingly. 

4. How do we make measurement happen in the real world—operationally, technologically, and culturally?

When determining scale to set operational expectations, start with the full picture, but be deliberate about building an MVP vs. ‘the sun, moon, and stars.’ Start with features that will enhance end users’ experience and deliver in small steps. 

Once the strategy is set, your next priority is information that will affect near-term management and impact decisions. Remember that software and technology are not solutions—they’re tools. Their usefulness depends on how fit they are for your purpose, how they are used and what goes in and how what comes out is used. These tools need context to be designed and implemented properly: objectives, user context, processes, roles, etc. As with everything new, effective implementation—including change management—matters 

M&E is a tool, it’s a means to your end goal. However, it is not the end in itself. I hope for those of you who are more on the operational side of things, that this will help drive you to better results. The systems you build, the integrations you have with GivingData, will be more thoughtful and nuanced so they do create the intended results. M&E is one contributor to our ability to advance our mission. It is one crucial element of driving change in this world. 

Cicero Group Is a management consulting firm dedicated to people and impact. We use a unique mix of research and data-driven recommendations to help organizations across the globe prove and improve their impact. If you have any questions or wish to learn more about successful M&E, please reach out to us or follow Cicero Group on LinkedIn.