Measurement
We broadly distinguish two main reasons for measuring impact:
- Learning and steering – an impact measurement provides information on which effects have and have not been achieved, and thus offers valuable insights to learn from and adjust activities or approaches. With this information, you can implement improvements to realise even more impact.
- Accountability and communication - making the societal value of your work visible. To find out which impact you actually achieve, it is important to carry out impact measurements periodically. This gives you insight into the impact you make, enabling you to account for it internally and externally. At the same time, it is valuable to communicate about it.
Which steps can you take?
Determine where you want to focus your impact measurement, for example by choosing two or three effects from the Theory of Change (ToC). Consider which learning questions you actually have for your impact measurement. What do you want to find out? It is important to make choices and set priorities. Measuring everything at once requires a great deal of time. Read more: Measuring impact - five perspectives to make sharp choices
Assess what is already known: this may be information already available within your organisation, or knowledge and insights from the literature.
Next, consider how you want to measure these effects, i.e. which measurement instruments (such as interviews or questionnaires) you will use and with whom. Does the instrument fit the target group you want to study? It is important to consider whether you are mainly looking for quantitative data (counting) or qualitative data (storytelling).
Once the measurement instruments are ready, you move on to data collection. Will you do this yourself or have it done by an external party? Collecting data yourself may reduce objectivity. Determine how many responses you need using a sample size calculator. For questionnaires, you generally want more responses than for interviews. In general, the more responses, the better, as this provides a more stable basis for conclusions. For interviews, fewer respondents are needed, but you aim for diverse insights, and saturation will occur over time.
The final step in impact measurement is data analysis. This is best done by collecting all data in one (Excel) document. This makes it easier to see similarities and recognise patterns. Some survey providers, such as SurveyMonkey, automatically generate analyses, and nowadays AI tools can also quickly assist with analysis. It works best to carry out this data analysis with several people, so that you have multiple perspectives on the data. Also ask questions about the data analysis: what do you find striking? What had you already expected? Can reasons from the data explain certain results?
Which tools can you use?
- One of the tools that can help with impact measurement is a measurement plan. A measurement plan serves as a prioritisation and concretisation of which changes, derived from the ToC, will be monitored. It clearly shows what needs to be measured (outputs, effects, assumptions), which indicators are associated with this, which measurement instruments are used to collect the data, where data can be found, who is responsible and when measurement takes place.
- The most commonly used measurement instruments are questionnaires and interviews, but there are also other instruments that can be used. Examples include Most Significant Change, Rich Picture or formulating statements for which voting boxes can be used. These are creative ways to measure impact among target groups that require a different approach, such as children or people with limited language proficiency. In addition, there are various frameworks that can provide guidance when conducting an impact measurement, such as the Impact Norms of Impact Frontiers or the B4SI Framework for portfolio organisations. These frameworks are particularly suitable for organisations with a large number of projects, where it is not always possible to measure all impact. They offer guidance to analyse and shape your portfolio based on more concise impact data.
Tips
When designing your measurement instruments, for example when drafting an interview guide, take careful account of the effects you want to explore. Sometimes you become so absorbed in the instruments that you drift away from the actual effects you want to test. Regularly check: do these questions still align with the effect I want to investigate?
Different target groups require different measurement methodologies. Look beyond traditional instruments such as questionnaires or interview guides. It can be very valuable to work with more creative tools, such as a “rich picture” or a “pride tree”. Read how we measured impact among children, commissioned by the Elja Foundation. Learn more.
For organisations that work with partner organisations, it is not always possible to measure impact at the level of the end target group. In that case, it can be very useful to measure the effect on those partner organisations themselves, such as the effects of knowledge sharing and professionalisation.
Read more about measuring impact in the arts and culture sector:
Example: WOMEN Inc. acts as lead partner for two multi-year alliances: the Alliance for Tailored Healthcare (AGOM2) and Financially resilient through work (FSDW), aimed at reducing inequality at the intersection of gender and health and work. These alliances pursue change at multiple levels simultaneously, from individual behavioural change to policy influence and systemic shifts in sectors such as healthcare or the labour market.
If you want to bring about such major changes, where do you start with measurement? With that question, Impact House supported WOMEN Inc. and the other alliance partners over several years: from operationalising the measurement plans through to supporting monitoring and the interim and final evaluation. In concrete terms, this meant: sharpening measurement priorities together with the alliance members; developing practical tools and templates for data collection; and facilitating interviews, document analyses and survey processes.
One of the methodologies we applied is outcome harvesting, an approach that is particularly suitable for mapping changes that are difficult to measure directly, such as shifts in political agenda-setting or behavioural change within healthcare institutions. In outcome harvesting, alliance members themselves reflect on the changes they have observed and link these to their own work. This requires guidance, but yields meaningful insights that cannot be captured through standard surveys.
At the same time, we ensured that the monitoring system remained manageable. By making clear choices about what is and is not measured, and by developing templates and tools that align with the partners’ day-to-day practice, measurement becomes something teams actually do, rather than something that gets left behind.