Should we do a SROI?

It’s a question that the measurement and evaluation team here at NPC gets asked a lot. Like any impact measurement, Social Return on Investment (SROI) is an investment in itself—it takes time and a mix of skills to do one properly. In the current funding environment, many charities want to know whether that investment will pay off.

Perhaps unfairly, SROI is often regarded as an approach to impact measurement that is much more about proving value to funders than improving value for beneficiaries. Of course, this may have something to do with the headline grabber that sums up a SROI—usually expressed as ‘For every pound spent, charity X creates Y pounds of social value’.

These ratios are a neat summary but having a number to stick in your marketing materials should not be the only benefit from doing a SROI. In principle, SROI is all about stakeholder perspective and value. You involve your stakeholders in most stages of the SROI process, from deciding what to measure to deciding how to value what you measure.

So why not check with your stakeholders about whether you should do one?

• Check with your organisation whether it has the data and skills to do a SROI, or at least the capacity to develop these necessary ingredients. Thinking about this at the beginning will make it much lower cost for your organisation.
• Check with your funders about what they think about assigning financial values to outcomes. This should help inform whether it worth going that final, technical, and sometimes controversial step.
• And last but not least, check with your beneficiaries about whether SROI would be interesting and relevant to them. At the end of the day the results should reflect the change that they experience as a result of what you do.

If you are still not sure, then it may be worth taking a look at just the first few stages of SROI. Not actually unique to SROI, these stages can help you to develop a logic model for your intervention based on stakeholder perspectives. (If you want to go a step further, then you may want to read about theory of change.) Regardless of whether you then go the whole hog, they should be valuable in understanding what difference you make.

Is a number worth a thousand pictures?

Anyone who has played a musical instrument, painted a picture or performed in a play knows that the arts are not the soft option. The arts may be enjoyable, but they also have the ability to engage people, to develop their skills and sense of responsibility, and to foster better relationships.

For these reasons the arts have long been used to help rehabilitate offenders or those at risk of getting involved in crime. And yet arts charities have traditionally struggled to provide hard evidence of their effectiveness, particularly in achieving criminal justice system targets.

Today we launch a report that explores whether the value of the arts in criminal justice can be shown through economic analysis. Commissioned by the Arts Alliance, the report takes three arts charities—Clean Break, Only Connect and Unitas—and quantifies the costs and estimated benefits of their interventions.

The three charities that we look at provide savings to the public purse as well as provide savings to the public purse as well as improve the life chances of the people helped. Our best estimates shows that these three charities provide returns on investment of between £3 and £6 for every £1 invested.

However, our findings could have been more conclusive with better data. Economic analysis has the potential to be a powerful tool for valuing the arts in criminal justice,but charities, funders and the government need to prioritise better data collection and access so that economic analysis gives the numbers that fully and accurately capture the value of arts in criminal justice.

You can download the full report, executive summary or press release here.

How to turn anecdote into data

Members of a board of trustees plot their views about the difference their organisation makes onto a map of the geographical area that they cover.

‘I can tell these stories about every service but I’ve got no way of gathering them all up.’

Transforming anecdote into data is a common frustration for charities wanting to demonstrate their impact. But it is a particular challenge for multi-purpose community organisations.

Over the past two years we have collaborated with the Institute for Voluntary Action Research (IVAR) on an action research project assessing the impact of multi-purpose organisations. Our research found that leaping straight to impact was impossible. Every conversation about impact began with an illustration about what is special about an organisation—for example,‘it’s been around for 100 years’.

Only after exploring these characteristics did we begin to talk about the actual difference that the organisation makes. In our research we identified four ways in which multi-purpose community organisations make a difference—tackling deprivation, integrating services, offering stability, and building confidence.

We then trialled three approaches to capturing this difference:

• relationship mapping: using visual maps of organisations’ relationships to uncover ways that they make a difference to communities;
• local economic analysis: understanding local money flows and how organisations can practically improve their local economic impact; and
• stakeholder appraisal: using local opportunities to ask stakeholders a series of questions about the difference the organisation makes to the community.

These approaches each had their strengths and weaknesses. As a quantitative analyst, used to dealing with facts and figures, I was interested to hear that the qualitative approaches were more useful to organisations trying to get to grips with understanding their impact. The leap to straight statistics was too far—the organisations needed a systematic approach to eliciting impact from qualitative evidence.

Overall,we found that it is important for organisations to think carefully about their impact before selecting an approach to impact measurement. Our project demonstrated that although there may not be a one-size-fit-all solution,it is possible to develop approaches that are meaningful and relevant to different audiences.

You can read more about the research in the recently published final report.

Getting started with cost-benefit analysis

Cost-benefit analysis is not the boiled egg of impact measurement—Delia Smith would not include it in ‘How to measure impact, part 1’. It takes time and patience and a love of Excel spreadsheets—to continue the culinary analogy, it’s more like the soufflé. And yet because it results in a financial value being ascribed to your impact, in the current funding climate many charities are understandably intrigued.  

We are excited that charities are thinking about their impact and think that cost-benefit analysis can be an excellent way to demonstrate what you achieve. However, you can rush into doing a cost-benefit analysis. In our experience, it is only compelling—for fundraising purposes or otherwise—if it is based on good outcomes data.

Here are our tips for what to do before you even crack open a spreadsheet:

1. Calculate your unit cost. Okay, so this may require a spreadsheet, but you will need a unit cost to do pretty much any type of a economic analysis, including cost-benefit analysis. A unit cost is the financial cost to the charity of working with one beneficiary. There is some guidance on calculating your unit cost in NPC’s Little blue book.
2. Clarify your theory of change. A theory of change diagram shows how your activities lead to the outcomes that meet your aims. It is a useful way to establish what to measure whatever measurement you decide to do. It is particularly useful if you then use the data you collect in an economic analysis because it gives you a framework to talk about which outcomes you are giving a financial value to and which you are not. Without a theory of change, it is tempting to latch onto those outcomes that you know deliver a financial punch rather those that are convincingly linked to your activities. It also should get you thinking about the ‘so what’ question—important for estimating the counterfactual of what would happen if you were not there.
3. Do a data audit. It is likely that you already collect some data. Do an audit of this data to see what could be used as evidence for your theory of change. For a cost-benefit analysis, you will need evidence of outcomes that is robust enough to be linked convincingly to financial proxies (more on this in a future blog). A data audit will allow you to assess the quality of your data and, if appropriate, start collecting outcomes data that can be fed into a cost-benefit analysis without too many leaps of faith.

As we said in our SROI position paper last year (NPC sees SROI as a type of cost-benefit analysis distinguished by its emphasis stakeholder perspectives), charities need better outcomes data to make the most of more complicated approaches like cost-benefit analysis. Regardless of whether you then decide to do a cost-benefit analysis, these three things—calculating your unit cost, clarifying your theory of change and doing are data audit—are good practice anyways for developing the outcomes data you need to demonstrate your impact.

Communicating research

Bright and early yesterday morning, NPC hosted the last in our series of breakfast seminars exploring issues raised by the government’s Big Society agenda. The event focused on how charities can communicate their impact and showcased the experiences of three charities. Emma Donne from Fairbridge, Hugh Rayment-Pickard from IntoUniversity and Sarah Hall from Safe Ground all gave thoughtful presentations about what they have learned about talking about the difference their charities make to people’s lives.

One of the most interesting threads of the discussion, and one that Sue—our head of communications—and I were chatting about after the seminar, was whether a tension exists when deciding how to express impact. Is it a case of the research team fighting for all the caveats of the study to included, and the communications team battling for a headline-grabbing statistic? Sometimes. But we concluded that actually it can be a useful tension, and different perspectives can definitely improve how research is communicated.

Yes, there is sometimes a bit of back-and-forth, but this can be constructive in working out what is really important. In designing the report for our well-being measure, for example, we had lots of discussions about how to present the results, and ended up with something that we hope is easy to understand without sacrificing the detail of the analysis and statistical best practice.

At the seminar, the panellists were asked how, practically, they ensured that research was communicated well—a clear, persuasive message, but one that accurately conveys the substance of the research. Emma said shes explains to the communications team what the numbers actually mean, and which ones are the most impressive in context, rather than simply giving them a list of statistics. Sarah mentioned that she writes a crib sheet for other member of staff, so that key findings from evaluations are known and communicated consistently. Hugh said that ‘it’s important to keep it personal. Putting case studies and numbers together can be incredibly powerful.’

At NPC, we try to involve the communications team in the research process, speaking to each other early on to ensure we’re thinking about key messages right from the beginning. We hope this means that our research is communicated clearly, although perhaps I should ask Sue to read over this blog…

Candid communication

Last Thursday I attended a breakfast seminar hosted by Fairbridge, a national charity working with disadvantaged young people aged 13 to 25. The seminar was to discuss ‘Back from the Brink’, a report pulling together findings from three independent evaluations of Fairbridge over a ten year period. The research even involved a regression analysis.

Wow. At NPC, this is exactly the kind of report we like to see—not for the regression analysis per se (although the measurement team does love a bit of number crunching) but for the thoughtful use of data that such a report implies. Because as well as using these findings to demonstrate its impact, Fairbridge is using them to inform its service delivery.

What we particularly loved about the report was that it was honest—not a glossy attempt to prove everything is fine and dandy, but a look, as chief executive Andrew Purvis remarked, at the ‘underbelly of Fairbridge’. There’s obviously lots of good stuff in there too, but Fairbridge is candid that some teams have much higher success rates than others. The charity has now identified the factors critical to success, and is seeking to share best practice across its teams.

As NPC’s forthcoming paper on impact reporting will show, Fairbridge is an exception to the rule here. Charities are not yet routinely communicating their outcomes or impact in their annual reports, annual reviews, impact reports or websites, tending to talk instead about outputs and internally-focused objectives. We need more charities like Fairbridge communicating honestly about what they achieve.

Measure once, cut twice

Earlier in the week I was explaining to the newest member of the measurement team that my first project at NPC, unlike most of what I’ve done since then, recommended that charities actually collect less data.

Well, kind of. Our 2008 Turning the Tables research called for more to be done to reduce the unnecessary monitoring and reporting burden on charities. This could be accomplished, as we suggested then, by collecting less data. Or it could be accomplished by doing more with the data that is collected.

For example, a charity may collect outcomes data for its funder, but the funder is only really interested in the submission of data, not the outcomes achieved. (What a shame! The suspicion is that this monitoring report is soon relegated to one of the funder’s many filing cabinets.) The charity gets a tick next to its name because a target is met, yes, but what does it actually learn about its impact?

Not much. Yet the charity could perhaps slice and dice this very same data in different ways, giving it a new understanding of its impact. For example, as the data is outcomes-focused, the charity could use it to answer questions such as—for whom does its services work best? And if the data is quantitative, it could even consider an economic analysis. This would allow the charity to demonstrate that its services are generating valuable outcomes.

Measure once, cut twice. Not quite the carpenter’s adage, but one of the ideas to come out of our Turning the Tables research and a good one for a charity trying to do more with less.

Taking the long view

Last week I attended a conference on longitudinal research, focused on the factors across the life course that contribute to healthy ageing. What struck me was how potentially useful these studies could be for charities to understand the impact of their work, particularly in the long term.

Usually, untangling cause and effect in social research is a tricky business. But just pick up a packet of cigarettes*, and you will find one of several unambiguous health warnings, such as:

Smoking when pregnant harms your baby.

This is not just a statement designed to shock; it is something on which we have some pretty incontrovertible evidence: children born to mothers who smoke heavily during pregnancy have poorer health and reduced height and weight, as well as worse English and maths scores at 16 years old.

We know this—particularly the long-term consequences of maternal smoking during pregnancy—with such certainty thanks to 17,000 people born in Britain during one week in 1958. These people, who turn 52 this year, are part of the National Child Development Study (NCDS), which aims to track a cohort of babies ‘from the cradle to the grave’.

Longitudinal studies such as the NCDS are gold mines of evidence because the same individuals are observed over time, which allows for some indication of cause and effect. For this reason, they often influence social policy. For example, the UK government’s emphasis during the recent recession on getting unemployed young people into jobs is in response to findings from the 1970 British Cohort Study (BCS70), showing the long-term negative consequences of being out of education, employment or training as a young person.

Incidentally, we also used these findings in our report on the issue last year to demonstrate the value of charities such as Fairbridge intervening to help young people develop the skills necessary for work.

But how useful would longitudinal studies be to charities? NPC’s Measurement team are itching to do some analysis of longitudinal data, so if you have an interesting research question (check out this website if you want to find out what sorts of topics are covered by the longitudinal studies), get in touch.

*Well, last year: from October 2009 cigarette packets must contain picture warnings.

Getting back on track

Youth unemployment has hit a record high. The latest government figures, released yesterday, show that 952,000 young people aged 16–24 are currently out of work—the highest number since records began in 1992.

This number is also dangerously close to the one million mark, which the government is understandably keen to avoid. Memories from the 1980s still linger, when youth unemployment actually did hit one million, and continued to rise for several years after the recession ended. A deep and long-lasting ‘wage scar’ afflicted many of those who were unemployed for a year or more, affecting their job prospects for decades.

To prevent history repeating itself, the government has announced a scheme giving young people the guarantee of a job, training or work experience. It hopes this will lead to a fall in youth unemployment by the second half of next year. This would be an achievement, but it should not be forgotten that some young people will not bounce back or into jobs as easily as others.

Young people with serious obstacles to unemployment—drug abuse, caring responsibilities, low self-esteem—may need additional help. Our recent research looking at the issue of young people not in education, employment and training found that charities are often best-placed to do so. They employ a variety of approaches to help young people others whom others have struggled to reach, from social and emotional support to children in primary schools to intensive one-to-one support to young people already unemployed. By recognising that behind the headline statistic lie a multitude of individual experiences, charities have an important role to play in reducing the number of unemployed young people.

Getting back on track

Today we launch our report Getting back on track, which looks at what the charitable sector is doing to tackle the problem of young people not in education or work. With news only today that the number of long-term jobless young people has hit a 15-year high, this is an extremely topical issue, and one that we think charities are often best-placed to address.

We hope you find the report interesting, and would love your feedback.

Charities helping young people get back on track

Young people today are having a tough time of it. The phrase ‘lost generation’—first used in the 1980s and then again in the early 1990s—has hit the headlines once more, as youth unemployment screeches upwards. The unemployment rate for 16–24 year olds in the UK is currently 17.5%, almost 5 percentage points higher than the same quarter in 2008.

The transition from education to the workplace is arguably tougher than it’s ever been. Young people fresh out of education are struggling to find jobs. Even those clutching degrees from good universities are joining the dole queue. But as the economy recovers, these young people should be able to take advantage of new job opportunities. But what about those young people who have the most serious obstacles to employment? Those who are battling drug abuse, those who are caring for a terminally ill parent, those who are struggling with extremely low self-esteem—in short, those with experiences that we wouldn’t wish on anyone.

The charity Fairbridge works with young people in this group. Those like Emma, age 22 and unemployed since leaving school age 16. She came to Fairbridge a year ago, suffering from depression and unsure of where she would be living one month to the next. Through the charity, she has received one-to-one personal support and done all sorts of different courses. She is now planning on going to college next year.

Next week NPC will publish its report Getting back on track, which looks at what the charitable sector is doing to tackle the problem of young people not in education or work. It highlights charities such as Fairbridge, which are helping individuals such as Emma. We find that charities are well-placed to help those whom others struggle to reach. Resources permitting, charities will prevent these young people from becoming ‘lost’.

Cutting red tape for charities

Yesterday the National Audit Office (NAO), the UK government body that accounts for how public money is used, published some welcome guidance on ‘intelligent’ monitoring. This new guidance follows directly from NPC’s Turning the Tables research recommending more could be done to reduce the unnecessary monitoring and reporting burden on charities.

We’re pleased to see that the guidance incorporates many of the points that we made in our report, which found the average burden for UK public funding agreements is 9%—much higher than for that for other types of funding. Of course, the government is spending public money and so greater accountability is to be expected, but we concluded that very little is known about whether the information produced for monitoring and reporting purposes is useful, to either funders or charities.

Time spent producing information that is not useful, or time wasted producing information in an onerous way, could be better spent delivering services. The guidance takes this as its starting point, essentially defining intelligent monitoring as that which strikes the right balance. Critically, it is one of the first examples of guidance that we’ve seen that actually explains what is meant by ‘proportionate’ and we hope should be of practical help to government funders in implementing the ‘principles of proportionate monitoring and reporting’ that the Office of the Third Sector (OTS), a UK government department, issued yesterday.

The guidance, along with the principles, is one of the steps the UK government is taking to cut red tape for charities. They are a step in the right direction, although implementing them will take a lot of work. Our research shows it is near impossible to unilaterally change monitoring and reporting requirements. It is therefore up to charities as well as funders to work hard to implement them to ensure monitoring and reporting are useful to everyone.

Can you compare a charity in Italy with one in the UK?

Today we launch a piece of research, comparing charities providing out of school hours activities for children in Italy, with similar organisations in the UK. The report, Everyday cares, was commissioned by an Italian foundation, to help it understand the success of the Italian charities it funds.

Comparisons across countries may seem tricky, but they can be done. Using an adapted version of NPC’s charity analysis framework, we found that charities in both countries recognise improvements in well-being as one of the main differences they can make to the lives of the children, although interestingly those in Italy focus on better relationships and those in the UK emphasise improved self-esteem.

Of course context must be taken into account—Italian charities wanting children to have better relationships is perhaps a reflection of the traditional emphasis in Italy on the importance of the family, for example.

But cross-country comparisons can help charities to:

• Identify their strengths and weaknesses; we found charities in Italy were better at involving the family but UK charities were better at promoting independence.
• Question why they are doing things in a particular way. An Italian charity may reconsider its strict attendance requirement after finding out that charities in the UK think making attendance voluntary encourages children to value activities more, for example.
• Learn from the results of others. We found that charities in both countries struggle to demonstrate what difference they make, particularly to the well-being of children. Finding out about what charities in other countries are doing to measure results can be useful; the Italian charities were particularly interested to learn about our well-being questionnaire.