April 25, 2022, by Digital Research
Approaches to tracking impact
In this blog, we consider approaches to tracking research impact, with a focus on digital tools.
We will outline reasons for gathering evidence of impact, what tools you might need to do this, and how digital approaches can help.
In short: how can you gather material to show the positive societal change that your work has enabled?
Why gather evidence for impact?
To demonstrate change: show to others that things are heading in the desired direction, satisfy stakeholders who are invested in your work, show that promises have been kept and increase your accountability, boost your own confidence.
To convince: build trust among potential partners, increase support for your work, make the case for scaling up, reduce scepticism.
To motivate: spark the curiosity of others, provoke questions and critical thought, inspire others to act, astound your audiences.
What do you need in your toolbox?
To demonstrate change: clear and credible baselines, testimonies from people affected by your work, user or access data, financial models and information, explorable data dashboards.
To convince: impressive comparisons (before and after your intervention), digestible data visualisations, a trusted messenger, a message tailored to your audience.
To motivate: inspiring stories, wide-reaching messages, equip your audience to take action.
How can digital approaches help (and the CART principles)?
Digital methods for gathering evidence of impact can help:
Speed: gather evidence quickly
Reach: gather evidence from a large group of stakeholders
Automation: gather evidence with less effort
Variety: gather evidence of different flavours
Cost: gather evidence cheaply
Digital tools can help you gather lots of evidence quickly, at low cost, and without much effort. We go into detail about some of these digital tools below, looking at their pros and cons.
However, just because we can do something, that does not mean we should. This is where the ‘CART principles‘ can help. CART stands for Credible, Actionable, Responsible, Transportable. In short:
Credible: privilege high-quality data over mass data collection; avoid bias in your data collection, for example by using randomised controlled trials; ensure that your counterfactuals (what would have happened if I had not intervened) are believable and supported with evidence.
Actionable: commit to collecting data you can actually use; collect data that will be useful also to others; make sure your data collection matches the needs of your stakeholders.
Responsible: ensure that the benefit of tracking impact outweighs the cost; ensure your data collection methods respect ethical guidelines.
Transportable: collect data that can generate knowledge for others; gather data that demonstrates why (or why not) your intervention could work in different contexts.
Following the CART principles when designing and carrying out your evidence collection will avoid waste and help maximise the value of your efforts.
Some specific approaches (and things to watch out for)
Build impact evidence-gathering into your project from the outset. In that way, you can plan the optimal time(s) for gathering evidence, establish ways to diversify the sources of your evidence, and even build the evidence-gathering process into your output (for example, ensuring from the very start of your project that you are able to see how many people download your toolkit, rather than waiting until after the toolkit is published to think about this).
The table below outlines some common tools for measuring impact, what they are good for, as well as some things to watch out for (you can open the image in a new tab for a larger view):
Take online surveys as an example. These are great for reaching a wide audience and collecting different types of information about the impact of your intervention. The cost is low (often free), surveys are flexible, and they help reduce biases that can occur in face-to-face settings. That said, surveys need careful designing to get the information you need, they often provide merely a snapshot of your impact at one moment of time (rather than regularly tracked impact), and they can be problematic in cases where your audience has low literacy levels.
Or how about tracking the number of people who have downloaded your resources? This is a great way of measuring the reach of your outputs over time. The process can be automated and it is free. However, this sort of tracking will not tell you how your resources are being used. So while it provides good quantitative data, it is less convincing when trying to construct a powerful narrative about the positive change you have enabled.
All the other tools in the table have their advantages, their challenges, and their optimal uses. The upshot is that one of the best approaches to impact evidence-gathering is to employ a combination of tools. This will allow you to gather a mix of quantitative and qualitative evidence and reduce the risk that key (i.e., persuasive) information is missed.
Last thoughts: planning you impact evidence-gathering
Planning your impact evidence-gathering will save you time in the long-run and set you up for demonstrable success.
Think early about what evidence you hope to gather (testimonies, cost savings, geographical reach, downloads of documents, etc.). Make sure you have good baselines, because you will want to be able to show that change has truly taken place and that this is linked to your intervention. Then think about the different audiences for your evidence, and what you might need to adapt to suit their needs – some audiences may prefer to see hard figures, whereas others will be persuaded by emotive stories. Finally, think about how you are going to collect the evidence (and how often), potentially using a mix of the tools outlined in the table above. If you can do all this at the start of your work, you will be in a fantastic place to demonstrate your impact to various audiences, when the time comes.
Further reading and inspiration
There is a wealth of resources on measuring and demonstrating impact effectively. Here is a selection:
A. Ebrahim (2019) Measuring Social Change. Performance and Accountability in a Complex World. Stanford – how to track performance toward worthy goals, with case studies from the non-profit sector
M.K. Gugerty & D. Karlan (2018) The Goldilocks Challenge. Right-Fit Evidence for the Social Sector. Oxford – using the CART principles to assess impact evidence-gathering and build better systems for tracking positive change
The Stanford Social Innovation Review (ssir.org) – for the latest thinking and innovations around impact
thinknpc.org (with various toolkits) – New Philanthropy Capital thinktank for the charity sector that aims at maximising positive social impact
Examples of socially conscious data visualisation and data dashboards (Periscopic)
If you have questions about any of the material discussed above, please don’t hesitate to contact a member of the team. We’d be happy to advise.
Sorry, comments are closed!