The Art of Measuring Change
What are your associations with words such as “measurement”, “evaluation” or “indicator”? For many people these words sound annoying at best, and there are good reasons for it. Yet - given the fact that you clicked on this article - it’s also very likely that you are motivated to contribute to social change, and the main intention of this article is to share about the beauty and collaborative power of attempting to measure it. What I am not doing is offering quick solutions, the same way art is not a solution to a problem. Rather, it is an invitation to step back, look at the nuanced process of change with curiosity and an inquisitive mind, and maybe discover something new. The article is also available as a dynamical systems map.
In a nutshell
There are two ways of reading this article:
- Exploration mode: Use the systems map provided above and use the visual representation to navigate through the article more flexibly.
- Focussed mode: Stay here and follow the flow of the article.
Here already a quick overview of the main points and arguments:
- There is a good reason to be skeptical of impact measurement. If the approach taken is too simplistic and/or mainly serves to validate a perspective (e.g. of the donor), it’s almost impossible to measure what truly matters. Rather, such an approach further perpetuates existing power imbalances and puts beneficiaries at risk. (section: The dangers of linearity)
- The measurement of social phenomena has to pay justice to the intricacies and complexities of a project, a program, or any given action. This requires understanding, which puts empathy, listening and collaboration at the centre of an empowering approach to measuring change. (section: About potluck dinners and “power with”)
- Meaningful indicators serve as building blocks in the attempt to capture social change. An indicator is meaningful when it not only follows the SMART principles, but is also systemic and relevant, shared (understood and supported by all stakeholders) and inclusive (aware of power relationships). (section: Indicators that matter)
- Complexity science can provide an alternative scientific paradigm to understand and make sense of our world and, hence, to measure change. (section: Coming back to complexity)
- It’s time to equally distribute power, namely the ability to influence and shape “the rules of the game”, or potentially even the kind of game that is being played in the first place. A crucial step towards this is describing and defining what desired social change is and how we go about measuring it in a collaborative way. (section: Synthesis)
The starting point
I have been working in the field of impact evaluation and measuring social change for around 6 years now. I’ve interviewed cocoa farmers in Ghana, developed surveys and frameworks, crunched Excel tables of different sizes and qualities, mapped indicators, and held workshops on tracking change in networks. Doing all of this is way more than a job to me. I have met wonderful people on this path, have been part of impactful projects and really had the feeling of being able to contribute in a meaningful way.
To me, impact evaluation is an ambitious, creative and collaborative application of complexity science (more on that later). It can be a deeply empowering process that reveals hidden opportunities and structural challenges, creating empathy between groups of people.
And it can be harmful.
The dangers of linearity
Let me share a definition with you to explain what I mean:
“An impact evaluation analyses the (positive or negative, intended or unintended) impact of a project, programme, or policy on the target population, and quantifies how large that impact is.”*
I agree with that definition in general, yet when it comes to the details in language, my opinion differs significantly. For example:
- Nobody is or should be a target. We don’t shoot projects, programs or policies at people, we work with people.
- Quantification plays a role, but we should also qualify impact.
- It is not said who has the power to define “positive or negative”, and “intended or unintended”.
These are not simply trivial semantic differences. All too often I experienced situations where impact evaluation was used by organisations to prove and validate the effectiveness of their services, rather than to truly understand the perception and consequences of what they offer.
Here is an example of what I mean: A couple of years back I had the opportunity to visit cocoa farmers in Ghana for an impact evaluation for an NGO. We found out that, yes, the NGO’s activities are actually having an effect and are contributing positively to farmers’ income. But we did not capture in writing the fact that most farmers stopped growing subsistence crops for their own consumption, hence making themselves more vulnerable to global trends and market prices. A few months after I left, the market price for cocoa dropped significantly. We did not capture it as it was not part of our evaluation framework or our mental model about what counts as relevant information.*
This is what happens all too frequently: We have pre-defined and pre-conceived notions of what counts as a data point, and what doesn’t. This again is usually based on a linear model of thinking (aka “A” leads to “B”), not accounting for the complexities and intricacies of human interaction and social systems. I have talked to so many people in the field that were frustrated, stating something along the lines of “we don’t measure what is truly relevant”.
To put it another way: If empathy and understanding are not at the centre of impact evaluation, both as the foundation and the goal, then it might be (unintentionally!) used against the people we want to benefit. It’s like having a knife in your hand. You can use it to prepare a delicious meal, but also to hurt yourself and others.
About potluck dinners and “power with”
Let’s dig a bit deeper into that: We’ve learned that what we do in impact evaluation is to assess the change attributed to an intervention (a project, program, workshop etc.). You do something and then something else happens. Now, we want to understand the “somethings” and how they are connected. This is not as simple a task as it may seem. Reasons include:
- We humans love to make our own meaning. You (and your organisation) probably know what the intervention is, but others perceive it fromtheir own perspective and life reality.
- Change is like air: Ever-present and hard to grasp. The “mechanics” of change can’t be pinned down easily and require us to look at context & conditions. Similarly, we know how difficult predictions are. Should we trust the weather forecast? For the next 3 to 5 days probably, but beyond that?
- There is power in the game: Oftentimes, multiple stakeholders and their multiple opinions are involved in an evaluation. There is nothing wrong with that per se, yet it is important to acknowledge.
The art of impact evaluation, therefore, is not to publish fancy reports, but to apply it in such a way that it deepens the internal understanding of issues at hand and strengthens the collaboration between actors. When this is the case, It builds empathy and human connection, and enables stakeholders to jointly develop shared meaning and scenarios for social change and transformation.
In other words, impact evaluation is based on and deepens listening. Listening not only to people, but to their context and to groups of people, to underlying and invisible challenges and hopes.*
In a more recent project I worked on we called all stakeholders together from the very beginning. It was clear who “has the money”, but we also shared the ambition to collaborate on a level playing field. As a consequence, we co-developed the project’s vision and the evaluation framework. We made clear that the first round of data collection also serves the purpose to understand what is relevant; and that the project goals will be further developed based on the “reality on the ground” rather than the content of the funders’ strategy paper.
We created the space for a genuine dialogue, and the data collected further strengthened the mutual understanding and trust in the group and in the process. It revealed both further challenges and where the leverage points are for creating systemic impact. It also became clear that power is not only linked to money.
“What makes power dangerous is how it’s used. Power over is driven by fear. Daring and transformative leaders share power with, empower people to, and inspire people to develop power within.” (Brené Brown)*
What we did was to ask a different question than normal. The central question was not “how can one central stakeholder prove their impact?”, but rather “how can we jointly contribute to desired change for a matter that we all care about?”. Shifting the question means that stakeholders and their individual contributions are valued differently, and that the quality of interaction changes.
Speaking in the metaphor of our delicious meal: We did not go into a restaurant where you tell somebody what should be cooked for you. Rather, we sent out the invitation to join a potluck dinner. Organising a dinner in this way does not happen at random. It requires clarity on roles and contributions, as well as shared agreements (e.g. on when and where to meet). The fundamental difference to a restaurant visit, though, is that it values participation and what each stakeholder (dinner guest) brings to the table, which in turn requires a mindset of being curious and open to surprises.
So far so good, but how do we actually measure change?
Jannik Kaiser is co-founder of Unity Effect, where he is leading the area of Systemic Impact. His desire to co-create systemic social change led him down the rabbit holes of complexity science, human sense-making (e.g. phenomenology), asking big questions (just ask “why” often enough…) and personal healing. Having worked in the NGO sector, academia and now social entrepreneurship
Originally published at Unity Effect
featured photo by Adi Goldstein on Unsplash
