Impact measurement: A how-to guide

We are all natural-born researchers. All you need is a sense of curiosity and a dogged nature to continue to ask questions, test new approaches and meticulously measure any change that occurs.

Most of us have this curious and inquisitive nature. It is only natural to want to know the drivers behind, say, changes in human behaviour, or the difference a particular service or intervention has made in a person’s life. This is why I find it hard to believe that impact measurement can sometimes be met with confusion–or even trepidation. Perhaps it is because impact measurement is still shrouded in an air of technical difficulty. Even the term itself can be prone to differences in interpretation. It shouldn’t be like this.

When we use the term impact measurement, all we mean is the tracking (or monitoring, or evaluation or measurement, whatever word you prefer) of the difference your particular charity, organisation, intervention or approach is making. And it is not difficult. Start with your own natural curiosity to think about the types of questions you want to answer. Deciding on the right questions can be the most difficult part of the process. If you aren’t measuring the right things, the results you get are meaningless. This is where theories of change can come in useful, to help you map a logical, causal chain of events that need to happen for you to achieve your final goal or mission.

Once you’ve decided on the right things to measure (or outcomes as we call them at this stage), you need to decide what tools you are going to use to start tracking progress. These tools can be on a sliding scale from the more robust to less robust. On the more robust side, and more straightforward to measure, are hard outcomes: definite, tangible changes that have occurred. For example, gaining a qualification, completing a course or moving into employment. But many charities work with people to improve the softer skills they need for these hard outcomes to happen. These softer skills can be around improving confidence, changing attitudes, fostering a sense of purpose and ownership, improving communication skills or improving someone’s social networks, amongst others.

These soft outcomes can be trickier to measure robustly. Charities often rely on case studies and interviews to show how these softer changes have occurred. The issue with this type of evidence is that although it is rich in detail and incredibly useful (and necessary) to communicate individual stories, they are too prone to bias (because they rely on one person’s own subjective feedback) and there is a risk that they are not representative of a charity’s entire client base.

This is where standardised questionnaires and scales come in useful. Psychologists and sociologists who study these softer skills and emotions have developed (over decades of research) robust scales to measure exactly the types of soft skills mentioned above. If the right scale is chosen and is given before and after an intervention, they can provide extremely robust evidence that change has occurred. What’s more, dozens of these scales are freely available for use online.

Charities needn’t measure the world but starting with a wisely chosen outcome and scale can make a world of a difference.

Get started!

• CES’s guide to theory of change
• NPC’s training on theory of change
• NPC’s well-being measure (a validated questionnaire for children) free taster workshops.
• Forum for Youth Investment’s guide to measuring soft skills in youth programmes
• A list of the most common psychological scales and questionnaires
• Tools for out-of-school time programmes
• Research and evaluation centre for outdoor education programmes

If you’re still confused, NPC’s measurement team can always help!

Knowing the difference

‘We make a difference’. This is probably one of the most commonly heard statements in the charity world. But have you ever stopped to think about what it really means?

Charities and funders make a difference to people’s lives in countless ways. But just knowing you are probably making some kind of difference isn’t enough: you need to know what difference you are making, how much it is and how it happens.

Charities usually have a pretty good picture of the difference they are making. They often work with people on the ground, can see change with their own eyes and hear the personal stories of the people they work with. However, relying on this kind of anecdotal information means that charities cannot communicate the difference they make to those who do not have this first-hand contact. It also means that charities do not have a complete picture of all their impacts and cannot systematically track or aggregate the progress of the people they help.

This is a problem because it means that charities cannot argue their case for funding, cannot make changes to their services to increase their impact and might continue running programmes that are not as effective (or in some cases, create harm). It also means that the field as a whole misses out—it has no robust data to demonstrate its value and cannot learn what interventions work best for different people and problems. The solution is for charities to set up robust research and evaluation methods to track the difference they are making to the people they help. This process has various names: monitoring, evaluation, research, outcome monitoring, social impact analysis or impact measurement. But whatever terminology you use, it all boils down to the same thing—robustly and systematically tracking the difference you are making to the people you help.

This is easier said then done. From our experience of helping charities to measure their impact, there is a lot more to implementing measurement than simply finding good measures. How do you convince staff of the importance of monitoring when they are already bogged down with client work, how do you develop their skills so that they are able to monitor and evaluate? What software to you use to store your data? How do you convince funders of the need to pay for such work? These questions are examined in NPC’s new report looking at six organisations at the forefront of impact measurement. The report examines how they overcame such barriers to develop impact measurement systems and what benefits they have received as a result.

However, the report is not just about the stories; we also outline a step by step guide and a number of resources to get charities started in their own journeys to good impact measurement.

In the current economic climate and with the government driving through a payment-by-results agenda, charities need to ensure that they are as effective as they can possibly be. We hope the stories of these six inspirational organisations in ‘A journey to greater impact’ can help them take the first baby steps to being able to show clearly the difference they are making.

The challenges of calculating the cost of social problems

Riots. Homelessness. Abuse. Underage drinking. The list goes on. Its really hard at times to ignore all that is wrong in today’s society. But the more you are confronted by these issues, the more you understand how many different problems there are out there and how complex their various causes and consequences are. Which is a problem if you work at a place like NPC where we try to analyse social problems and their solutions in an empirical and rational way.

This is why our research for Barclays Wealth was both extremely interesting but extremely challenging. The final aim of this research was to prioritise a list of interventions that, when invested in, will save the highest amount of money for society. The obvious starting point therefore was to find out what society’s costliest  problems are.  I was tasked with phase one of this research where we started with the thirty biggest social problems in the UK and prioritised what we considered to be the six most serious, based on the numbers of people affected and their economic cost.

How did we to do this? The answer is—with difficulty. In an ideal world all costs would be comparable and it would be easy to see which were the most expensive.  But this is not the case—especially when it comes to social issues.  This meant that we had to overcome a number of methodological issues at the first stage of research:

• We wanted to look at both individual and government costs—it became apparent that it would not be feasible to find cost figures in such detail.

• It was necessary to prioritise based on the best and most comparable figures we had: scale and larger society costs.

• A lot of costs had been calculated using different methodologies and so true comparison was not possible.

• Costs tended to be skewed to existing government funding priorities.

• Most costs presented were current costs, although some were presented as life-time costs, so comparability mixed.

All this meant that we prioritised issues based both on costs and on how much we trusted the quality of the data. Admittedly, this was an imperfect process, but still allowed us to get an approximate idea of the most expensive problems.

Many of the major challenges we faced in this research were to do with the availability and quality of data, an issue which has come up again and again on this blog. But I think the most interesting lesson from this process is that taking an economic approach to understanding which problems to fund can be done -  even with these imperfections.

Growing your organisation in tough times

Recession and funding cuts. Probably two of the most commonly heard words this year. If you are a charity or indeed any organisation with a social mission, you are probably wondering along with the other 171,000 Voluntary Sector Organisation’s out there how you are going to weather this particular storm. In a research report that NPC presented last week at the annual NCVO Researching the Voluntary Sector conference, we examined whether social enterprise might offer a more resilient funding structure. This is based on an evaluation that we carried out for the School for Social Entrepreneurs last year. Our findings showed that the 153  social enterprises who responded to our survey were 20% more likely to survive for 5 years than the average UK business, had very high annual growth rates and reported growth even during the recession. Interestingly, the reasons behind this resiliency seems to lie, not in the fact that the organisations are social enterprises but in the way they combine commercial income with traditional charitable funding (public grants and contracts as well as funding from grant-making trusts). This diversity in funding seems to act as a stabiliser to organisations in tougher times. Although the research was with a small and select group, it offers some interesting lessons for all organisations with a social mission wishing to make their work more sustainable.

Read more about the research in today’s Guardian Social Enterprise Network blog or download the full report.

Getting the right evidence

Yesterday, I was discussing the topic of evidence with a colleague. An unsurprising choice of topic, given that one of the things we do at NPC is to look at charitable interventions and find out which one gives you most bang for your buck – or most impact for your money. We do this by trying to understand  ‘what works’ in an entirely rational, empirical and analytical way.

This may sound obvious: in science and business, finding solutions to complex problems usually involves empirically rigorous methods of investigation. The Greeks thought the atom was indivisible but in the 20^th century new theories and experiments opened it up (and helped create nuclear power). 18th and 19^th century medical analysis often required cutting open the body; today magnetic resonance imaging  provides the clearest and most accurate pictures yet inside the human body. In business, Henry Ford produced 1 million cars in 1920, all black, all Model Ts; today, consumer marketing firms like CapitalOne can run thousands of new product ‘experiments’ every month, testing what consumers like best.

These developments in business and science are the result of investing the time and money to investigate what works and what doesn’t. Imagine a world where we applied the same rigorous approach to solving social problems. How many lives would be irrevocably changed, how much misery would be averted? For this to happen, we need to get a lot more sophisticated about good data and robust analysis. We need to take the best bits from science and business and go about building similar structures in the charity world. We need to invest in analysing and researching interventions and unearth those with the best results.

Unfortunately, we are a long way off the rigour of the scientific or business world. Charities are organic developments—they often grow from within a community, and respond to specific needs in unique and creative ways. They do not (for the most part) develop out of years of empirical research. This is both their advantage and disadvantage. The lack of a body of research to assess their efficacy means that we have very little idea about the results that a lot of charities achieve. However, charities’ organic and home-grown nature means that they often come up with innovative and effective ways of tackling social problems, well-suited to their community of need.

But without reliable results how do we know which interventions to back? Where should we put our money?

This is why the idea of a “shared measurement” agenda is so important. Having comparable if necessarily simple measures would at least allow us to compare the results of different interventions if not be completely rigorous. This foundation could then be gradually built up to the level of empirical sophistication found in other disciplines.

We need to recognise where we are with evidence in the charity sector and aim our sights on what is achievable. And right now to simply compare different charitable interventions in the same sector would be a watershed.

Meaningful measurement

In recent years there has been a perceptible shift in the charity world in relation to results measurement and demonstrating value. The language we hear from charities talks more and more about outcomes, impact and value for money. This thinking is also making its way into government policy: payment by results is a big agenda of the coalition government and Barack Obama plans to spend $100m on “social impact bonds” in the US.

We are now seeing concerted efforts on the part of charities to measure and talk about their results. Working on various projects here at NPC, I have come across extremely dedicated individuals and organisations who are doing their very best to implement these systems. However, I have a concern for the future: today’s efforts are fragmented, one-off pushes to impact measurement from different organisations in different fields. This effort on evaluation is commendable but it only tells us how well an individual charity is doing. While it can give some indication of best practice or impact in a specific area, the lack of comparability means we are still left in the dark about what interventions work best and what should be scaled up.

The next step that now needs to be taken is a drive towards common evaluation systems and shared measurement. For different charities in similar fields, we need common goals and outcomes and a standardised way of measuring those outcomes. Not only will this allow us to see what works best in a sector but it will also demonstrate the value of that sector’s work on a wider scale. And, as my colleague John wrote yesterday, demonstrating this value is particularly pertinent right now as the Big Society plans of the coalition puts charities’ work in the limelight.

This is why 2011 will see NPC looking at different charity sectors and producing shared measurement frameworks. This is the next big leap that the sector needs to make. It is not only crucial to learning and improvement but essential to the people that charities help.

This is the first in a series of blogs about NPC’s shared measurement programme. We will be publishing shared measurement research pieces on youth offending, NEETS and mental health later in the year. So watch this space!

What Teach First can teach us

Last week, I attended a very interesting presentation—a discussion of findings from an independent evaluation of the charity, Teach First. Teach First is a charity whose aim is to address educational disadvantage by transforming exceptional graduates into effective, inspirational teachers and leaders in all fields.

The research, carried out over three years, looked at whether Teach First was achieving this goal. It was measured by looking at whether their teachers were effective and whether pupils’ outcomes were altered as a result.

The findings were notable for a number of reasons. Firstly, the research actually found statistically significant results, which as Professor Muijs who co-authored the report told us, is very rare in the field of educational research. It is even rarer to find in the charity sector. As discussed by my colleague Sarah Keen, in an earlier blog, it is very hard to find empirical research in the charity sector that actually attempts something as sophisticated as a regression, for example.

The second reason why the research is notable is that it highlights the role charities play in bringing innovative solutions to the most complex of social problems. The research was able to show that the idea behind Teach First does seem to work. The graduate teachers were found to be either as effective or more effective than more experienced teachers. The factors leading them to be effective were found to be a combination of enthusiasm, strong leadership and a belief that they could positively impact on their pupils’ outcomes. The research also found a correlation between Teach First schools and improved GCSE results. While correlation does not amount to causation, and I would have liked more detail on this aspect of the report and more probing of students’ individual journeys, this finding is a good start.

Understanding what makes teachers effective is a key question in understanding what improves outcomes for pupils. This report highlights the role charities play in pioneering new approaches that can turn out to be more effective than traditional methods. As another colleague, Esther Paterson, wrote last week, charities need a significant amount of creativity when it comes to thinking of new ways to tackle the most deeply entrenched issues in our society.

The work of Teach First is incredibly relevant right now with the government unveiling new plans last Wednesday to tackle declining school standards. This plan will seek to give more freedom and authority to teachers, as well as encouraging high achieving graduates into the profession. Even more interesting, a report published yesterday by the consulting firm, McKinsey, discusses how countries with the best faring students tend to have teachers who are enthusiastic and well-educated, which emulates the findings of this Teach First research.

Unfortunately, most charities have yet to carry out well-controlled research that captures the impact they are having. Charities, as a result, are losing out, through being unable to show that their innovations are effective. This is why evaluations like the one discussed are so important. They show that charities can be innovative in fields where it is needed most. In turn, this demonstrates just how important and valuable the sector is at a grass-root, as well as a wider policy level.

Why government measuring well-being is good for charities

Money is important. But unlike the popular lyric, money does not, in fact, ‘make the world go round’. Research since the seventies has shown that despite rising levels of GDP in most developed countries, happiness or a measure of it has remained mostly the same. How could this be?

Most of us know that there are things other than money that make us happy. That is just common sense. Government policy, however, has long been measured by how successful it is at increasing GDP. But, as we now know, that is not necessarily a  good measure of success.

That is why government plans to measure well-being across the population from next year should be welcomed.

The cynic in me thinks that at a time when everyone is very aware of the financial climate, it is not surprising for government to acknowledge that there is more to life than money. But it is still a very significant step. Almost 30 MPs have signed a House of Commons motion arguing that ‘promoting happiness and well-being is a legitimate and important goal of government’.

More intriguingly, well-being seems set to become a buzzword for evaluating public policy, and how individual organisations – including charities – prove their impact.

For charities, it is no leap to say that well-being is important. Well-being has always been an important outcome of charities’ work. We just haven’t been very good at measuring it. This is part of the reason why NPC has spent the last two years developing a tool for charities working with 11 to 16 year olds. This tool quantifies the impact these charities are having on children’s well-being.

 Charities should welcome the government’s announcement and seize the opportunity to prove that what they do has a significant impact on well-being.

It has long been that the ‘standard’ measures of success don’t work very well when working with the most vulnerable people. Cost benefit analysis, studies of exam grades and measures of health outcomes can be good indicators but often miss the point or are not sensitive enough to capture impact. For example, the help School-Home Support gives to vulnerable families often provides the ‘first step’ to a more successful educational career and positive family life, but it would be unreasonable to expect to see an immediate impact on a pupil’s academic attainment.

Many people will now be questioning how exactly you can go about measuring something as subjective as well-being and will be wondering whether it is, in fact, possible. Our experience with the well-being tool has told us that  not only is it possible, but crucial to do so.

A Dragon’s Den for Social Entrepreneurs?

The Dragon’s Den is a BBC venture capitalist pitch contest. The programme, first aired in 2005, sees hopeful would-be entrepreneurs pitching their ideas to a panel of former industry tycoons and business moguls. Since 2005, the panel has invested in 110 different companies.

While the investments have had varying degrees of success, (some phenomenal like Reggae Reggae sauce which has gone from being sold out of a stall at the Notting Hill carnival to being stocked in most major supermarkets in the UK) their true reach goes far beyond mere profit: the programme has brought the basic tenets of enterprise to the common person on the street. Getting the idea that anyone can be an entrepreneur or try their hand at business into the public consciousness is, in itself, a huge achievement.

All this makes you wonder if the same could be done with a Dragon’s Den for Social Entrepreneurs? Of course, this is not an entirely new idea. Social Venture funds have been around for a good few years now, and terms like Philanthrocapitalism, Social Investment, and SROI (see NPC’s position paper) are becoming more and more frequently heard. I have even come across Dragon’s Den Social Enterprise competitions which are already being run here, and organisations like the Funding Network provide forums for funders to connect with catalyst charities, but as yet none have made their way into the mass media.

The problem is that for the most part the wider public remain completely unaware of these new developments in the charitable sector. And we all know that for real social change to happen, we need to mobilise the masses.

Although, in many ways, Social Entrepreneurship is just an alternative description of things people have been doing for centuries, it is also something quite new and pioneering. First of all, using new terms and phrases reinvigorates the way we deal with and look at different problems. Secondly, the movement provides us with innovative tools and alternative approaches for solving social problems. But right now I feel like this new way of thinking is still contained to too small a circle of people. Social entrepreneurship needs to be brought to a much wider audience.

Could a programme like the Dragon’s Den be a vehicle for doing this? In the same way that the traditional Dragon’s Den is changing our thinking around who can be a successful business person, a Dragon’s Den for social entrepreneurs might be an essential ingredient in getting more people involved in good deeds, in more innovative ways. A snappy TV programme along these lines could create the publicity, mild amount of controversy and healthy debate that is needed to get more people thinking about and involved in social entrepreneurship. Ultimately culminating in a tipping point for social revolution…..Any opinions?

Opening up data

A well known quote tells us that to “improve something, first measure it”. Having just joined NPC, I was unsure what charity research would look like. A month on and that is the least of my worries. Now I am more concerned with how to go about doing that research in the first place.

The issue is; how exactly can we understand charity effectiveness when the data we base it on is notoriously inaccessible?

Last week, my colleague Camilla Nevill wrote about her frustrations over the lack of access to data on youth offending. This is an often repeated story at NPC and my first project here was no different. This involved trying to track down government figures on the needs of children and public spending across each region and country of the UK.  Three weeks of constantly coming up against closed doors made me realise how hard it is to take a quantitative approach in this sector. With such a lack of data, it is extremely challenging to do charity research. But if we can’t measure in the first place then how can we begin to improve?

In business and science, the ever increasing availability of vast amounts of data is bringing about an information revolution. It is now possible to understand trends and relationships in ways that were inconceivable even five years ago. Credit card companies can pinpoint from millions of transactions those which are likely to be fraudulent and the use of data online has revolutionised the world of advertising.

If the same amount of data was available for those working in the field of human welfare, we could begin to understand exactly what issues need to be addressed, where they should be addressed and how best to address them. Pinpointing resources in this way will allow charities to rapidly and effectively deal with problems and reduce wasteful spending.

One of the first things president Obama did when he got into office was to make as much government data available to the public as possible.

Similar efforts have been made over here. Websites like data.gov.uk and tso.co.uk are making good headway but are still frustratingly lacking.

In fact, as I write this, The Stationary Office are in the middle of the OpenUp challenge; a nationwide competition looking for the best idea on how to improve the availability and presentation of government data.

This OpenUp challenge is a great idea, but we still need more data to be made available.

Given the cuts that are looming for charities across the UK, anything that gives us more insight into which organisations and approaches are the most effective must be a top priority. So like Camilla, I add my voice to the call for more data. As a favourite quote of mine goes: “The point of open information is not merely to expose the world but to change it”.