NEETs charities must work together to prove their worth

Abi Levitt is Marketing and Communications Director for national employment charity Tomorrow’s People. Founded in 1984, the charity has a long track record of success in helping people out of long-term unemployment, homelessness and welfare dependency, and into sustainable jobs. Tomorrow’s People has helped over 440,000 long-term unemployed people on their journey back to work. The charity is one of six featured in NPC’s latest shared measurement pilot, looking at developing a shared outcomes framework for the NEETs sector.

For the last 30 years Tomorrow’s People has been supporting people from disadvantaged communities to get and keep a job, but we have a special concern for young people on the margins who have failed to make a successful transition from school. Since 2004, we’ve focused our support for young people on those 16-24 year olds who are not in employment, education, or training and who other agencies have found hard to reach.

Until last year, our youth programmes were funded only by private donations and those funders have always expected to see tangible results and clear impact measurement to justify their investment. However, as anyone working with disadvantaged young people will know, it is very difficult to understand and evaluate the progress that participants make on a range of hard and soft levels and to keep track of them once they ‘graduate’, so that impact and sustainability can be measured.

It is critical for all organisations working with NEETs to be clear about what success is.

But it is crucial that we have a credible story to tell investors about impact as it is a key indicator of the success of our approach, and the ability to record hard as well as soft data is as important for programme development as it for investment analysis. As a result, we make a great deal of effort to engage with participants for a year after a programme finishes, both to offer support and to measure outcomes.

It is critical for all organisations working with NEETs to be clear about what success is and to articulate it in a way that has currency for investors so that progress can be evaluated fairly and value proven. Unfortunately, the wide range of third sector organisations currently working with young people are doing so without common standards of measurement and evaluation, which make it very difficult for potential investors to judge where they are best to focus their resources.

That’s why Tomorrow’s People was delighted to be invited to join the NPC pilot and to take part, with other like-minded organisations, in a forum to look at the practical steps we can take to build a common standard of assessment. However, there is still much to do, both to convince the sector that it is possible to build a common language across a range of activities and specialisms, and then to convince commissioners and funders that what is being measured is important and of value.

Charities working with young people can and should be collaborating and having an open discussion on how to create universal standards and measure the soft outcomes – like confidence, self-esteem, motivation and happiness – that are often considered immeasurable, but which we all know are vital for development and progress.

Read more about the shared measurement pilot in which Tomorrow’s People took part in Impact measurement in the NEETs sector, published today and available to download for free from NPC’s website.

Supporting the academic spring

In NPC’s monitoring and evaluation team we spend the bulk of our time advising charities on how they can better monitor and evaluate the services they provide. Invariably our advice includes them getting familiar with relevant academic research. All too often we find that charities are not aware of relevant research. The risk is they can waste goodwill, time, effort, and money, by running developing services that may not be so effective as alternatives.

But our advice can be very difficult to follow not least because evidence generated by academics can be inconsistent and inconclusive. And this assumes that the charity can actually make sense of the research. Most academics are primarily thinking of their peers as the audience when writing up research, not the general public. I am accustomed to ploughing through academic articles and books to find the relevant information, but it is not easy.

Precious information that could help charities improve their services lies behind a very high and very thick wall.

But the main problem starts before you have an article in front of you. The challenge is getting hold of what is relevant in the first place. We are currently involved in a project where our academic partner (the Institute for Criminal Policy Research, based at Birkbeck, University of London) are having to review hundreds, if not thousands, of summaries of articles to identify the relatively few that are relevant to the project. This is time consuming and hard enough for them, and they are experts. A charity without that expertise and without access to the publishers’ databases that contain those articles faces an enormous struggle to get hold of, let alone use, the relevant research. So precious information that could help them improve their services and the lives of those they serve lies behind a very high and very thick wall.

So I was delighted to read that the government has drafted Jimmy Wales, the founder of Wikipedia, to help make all taxpayer-funded academic research in Britain available online to anyone who wants to read or use it – another step in the push for open data that’s been dubbed the ‘academic spring‘. There are already various initiatives that help make academic research more readily available, such as the Social Care Institute of Excellence’s website, the Campbell and Cochrane collaborations for the criminal justice and healthcare sectors, and the library run by National Children’s Bureau. The British Library has also been involved in trying to make research more accessible to charities. But more support is needed.

In a couple of weeks I will be speaking at two conferences (one in Manchester and one in London) on measuring and evaluating outcomes in the charity sector. I will cover what has happened in the past, where the sector is now, and where the sector is headed. Much improved access to evidence is part of that future, and we will work to help charities get there sooner rather than later. I do not (yet?) have a particular recommendation or proposal for charities. But in the meantime I encourage you to add your support to the Academic Spring however you can.

The ideology of impact

Against a dark background of austerity, double-dip recession and record levels of inequality, it’s no surprise the spotlight shines on the numbers—GDP growth, output, unemployment, credit ratings—the list goes on. And in the world of charities, it seems that the subject of impact falls into the same category—data, numbers and ratios.

We often debate what impact means—a discussion that’s usually about definitions and technical distinctions. But recently as I was talking about impact with a group of trusts and foundations, the question of what impact means ideologically was raised.

A concern was raised that the increasing focus on charities measuring their impact was motivated by a desire to marketise the charity sector. The more we focus on impact, the more charities compete with each other in a marketplace driven by contracts to deliver services. If we wait long enough, the big charities won’t look any different from Serco, the argument goes.

Understanding impact isn’t about making charities jump through hoops.

But I don’t think that’s what impact is about. A focus on planning, managing, and measuring impact means a focus on understanding the lives of the people that charities aim to help, and the changes they bring about in those people’s lives. By understanding this, they can work to make the biggest positive change possible. Impact measurement is a tool for social justice, not marketisation.

Take the charity The London Pathway, for example, which works within hospitals to integrate and improve healthcare for homeless people. It grew from a recognition that healthcare outcomes for homeless people were much poorer than for an average person—they are admitted to hospital four times as often, they stay in hospital three times as long—their overall health is much worse. By focusing on the needs of the people they aim to help, and understanding their life situation, The London Pathway has been able to massively improve these outcomes. And the reason it understands these needs, and can make a big positive change, is because it monitors the impact of it’s work. That’s social justice—better outcomes for people suffering multiple disadvantage.

The thing is that The London Pathway, and other charities that deliver outstanding impact for the people they help, not only improve lives but also save society a pile of cash. In the words of Harvard Business School Professor Michael Porter, it ‘is an excellent example of how a value-based approach can better serve patients, reduce inequalities, and deliver better outcomes that matter to patients per pound spent’.

At NPC we’re passionate in our belief that by focusing on impact—the real change they create in people’s lives and society—charities can achieve better outcomes for those who are worse off. And this will reduce inequalities and improve social justice.

Of course there’s a danger that impact measurement is used as a technocratic mechanism to bludgeon charities, favour those with the resource to build fancy systems and research methodologies, and marketise the sector. But if we really get to grips with impact, embedding it in charities’ practice and funders’ decision-making, what we can do is something much more positive, much more powerful: we can help charities to back up their passion and conviction with a compelling case that should help them get the resources their work deserves.

I hope we remember that understanding impact isn’t about making charities jump through hoops. It’s about putting the real changes to people’s lives at the heart of what charities do.

Tris will chair two conferences in London and Manchester later this month on how charities can embed an impact approach in their work, which will focus on the practical, with a wealth of guidance, experience, support and tips on measuring and managing impact.

Should we do a SROI?

It’s a question that the measurement and evaluation team here at NPC gets asked a lot. Like any impact measurement, Social Return on Investment (SROI) is an investment in itself—it takes time and a mix of skills to do one properly. In the current funding environment, many charities want to know whether that investment will pay off.

Perhaps unfairly, SROI is often regarded as an approach to impact measurement that is much more about proving value to funders than improving value for beneficiaries. Of course, this may have something to do with the headline grabber that sums up a SROI—usually expressed as ‘For every pound spent, charity X creates Y pounds of social value’.

These ratios are a neat summary but having a number to stick in your marketing materials should not be the only benefit from doing a SROI. In principle, SROI is all about stakeholder perspective and value. You involve your stakeholders in most stages of the SROI process, from deciding what to measure to deciding how to value what you measure.

So why not check with your stakeholders about whether you should do one?

• Check with your organisation whether it has the data and skills to do a SROI, or at least the capacity to develop these necessary ingredients. Thinking about this at the beginning will make it much lower cost for your organisation.
• Check with your funders about what they think about assigning financial values to outcomes. This should help inform whether it worth going that final, technical, and sometimes controversial step.
• And last but not least, check with your beneficiaries about whether SROI would be interesting and relevant to them. At the end of the day the results should reflect the change that they experience as a result of what you do.

If you are still not sure, then it may be worth taking a look at just the first few stages of SROI. Not actually unique to SROI, these stages can help you to develop a logic model for your intervention based on stakeholder perspectives. (If you want to go a step further, then you may want to read about theory of change.) Regardless of whether you then go the whole hog, they should be valuable in understanding what difference you make.

Small is beautiful

Knowing where to start with impact reporting can be a challenge, but for small charities with limited money, time and staff, it can be even more daunting. Good impact reporting is within reach whatever your size—but smaller charities need to keep their reporting in proportion and think carefully about the most important things to measure.

This morning, NPC and CFG hosted a seminar for small charities, Keeping it in proportion, focusing on effective impact reporting for charities whose resources—money, time and people—may be limited. Cath Lee, Chief Executive of the Small Charities Coalition, chaired the event, and speakers included Dr. Hugh Rayment Pickard from IntoUniversity, Meredith Niles from Impetus Trust, and Mark Gilham, chief executive of Harrow Mind. Several themes emerged from the day which are worth bearing in mind when thinking about impact reporting:

• Know what you need to measure: All our speakers agreed on the need to be clear about what it is you want to measure before you start, so you don’t waste valuable time and resources. Working through a theory of change is an excellent place to begin. A theory of change is a tool which works backwards from your charity’s big, ambitious goal, through the outcomes you need to achieve this, and the activities which lead to these outcomes. It helps you take a step back and check you are doing the right thing. It also helps you identify clear points where you can measure your outcomes to show whether you are making the change you want.

• Start with two or three key things: Don’t feel like you have to jump straight in and monitor everything you do. If your resources are limited, start by measuring the things you think are the most important elements in your theory of change. Once you’ve mapped a theory of change, it is usually possible to identify which activities and outcomes are absolutely key to achieving your aim, so start by concentrating on these.

• Use the tools and data available to you: Surveys you can administer yourself, testimonials from beneficiaries, or published data already available about the people you help are a good place to start, and tend to be more accessible to small charities. First Story, for example, carries out before and after surveys on students and teachers at participating schools, looks at existing data on a school’s exam results, and collects testimonials from students, which combine to give a really thorough and engaging picture of the difference their work makes. This morning’s speakers also highlighted the value of using existing research to think about your long-term goals, and provide evidence for why you think your approach will work.

• Know why you’re doing it: There are many reasons to start impact reporting, from engaging donors and funders, to inspiring staff, to boosting your credibility, and ensuring people focus on the right things. But ultimately, as Dr. Rayment Pickard summed up at this morning’s session, the most important reason to measure your impact is for yourselves: ‘you’ve dedicated your life to this, you want to make sure it works.’

You can read a selection of tweets from this morning’s event below, and if you are inspired to get going with your own impact reporting, NPC’s new report, Theory of change: the beginning of making a difference is a good place to start!

[View the story "Keeping it in proportion: Impact measurement for small charities" on Storify]

Answering the ‘so what’ question

Paul Farmer is chief executive of Mind, the mental health charity. He has worked in mental health for the past 20 years, joining Mind in 2006 following posts at mental health charity Rethink and the Samaritans. Here he writes about Mind’s work with NPC to develop a theory of change.

Mind is one of Britain’s best known charities. We want to see everyone experiencing a mental health problem get the support they need and the respect they deserve. We provide help and support to nearly 300,000 people every year. We are a high profile campaigning organisation giving people with mental health problems a voice. And we’re working in partnership with Rethink Mental Illness on Time to Change, the biggest campaign to tackle mental health stigma and discrimination this country has ever seen.

So what?

That was the question senior managers and trustees within Mind wanted to answer. We might distribute millions of pieces of information each year, but does it make a difference? Does our campaigning work really change lives? Does our local work help people achieve their potential? Like many charities, we believe the answer to these questions to be yes, but do we really know?

As senior managers we know that much of our work is valued, but was there too much emphasis on output rather than outcomes? For trustees, as long term custodians of the charity, the issue was whether it’s possible to find a way to evaluate progress towards our vision over a longer period of time

So we turned to New Philanthropy Capital to help us devise and develop a new evaluation framework. We asked them to work with us to create an evaluation framework to answer the “so what” question. This has coincided with developing a new strategy for the next four years.

It’s been an interesting journey. We started by creating our own Theory of Change—how we believe our work achieves real change for people with mental health problems. We used this framework to create a strategy that is focused on the outcomes that our stakeholders told us were most important to them.

We worked with a small number of departments to build the skills required and an evaluation toolkit. At a governance level, we’ve kept our trustees fully involved. Locally, we’ve run a pilot with five local Minds.

Eighteen months later, we’re here. We have a new strategy with a comprehensive evaluation system in place. We will use that information in our annual reporting so that beneficiaries, donors and other stakeholders can see the impact we’re making. We have the best possible chance of being able to answer the “so what” question.

This week, NPC will publish a guide to theory of change for charities, which will be available to download free from our website.

We need your views!

Impact measurement has come a long way in the last ten years. It’s no longer just the niche interest of organisations like NPC; it’s now a key part of many charities’ everyday operations. Today, impact measurement is the focus of major conferences, popular tools and guides, and even entire organisations like the Social Impact Analysts Association. Some charities even have members of staff dedicated to monitoring impact.

But although we hear a lot of anecdotal evidence to support the view that charities are serious about measuring the difference their work makes, we don’t have the data to back this up. We don’t know how many charities in theUKare actually measuring their impact, what they are measuring, and how they are doing it.

This is why we’re very excited to be working with NCVO, ACEVO, CES and TSRC to carry out a major research project to dig deeper into questions around how charities are measuring the difference they make. Over the next few months, we will be working with Ipsos Mori to conduct a representative survey spanning the wholeUK charity sector—looking at charities across England, Scotland, Wales and Northern Ireland, and building on research carried out in 2007 and 2008 by CES. We want to build up a detailed picture of what charities think about impact measurement, how they approach it, and what kind of results they get.

We want to know if charities are collecting output data or outcomes; how sophisticated their level of evidence is; whether they are using quantitative research methods, randomised control trials, matched samples or academic literature. But we also want to find out what charities think are the major barriers to measuring impact, so that the sector can start to think about how these can be overcome.

Ultimately, we hope that the information we gather from this survey will help charities to measure better. By finding out what is holding charities back from measuring their impact, the organisations involved in the project can act to help overcome these stumbling blocks. For example, one common complaint we hear is that different funders demand different sets of information on impact, or information presented in a different way—something which might ultimately be changed with work in different subsectors on standardised reporting.

We’ll be publishing our findings in a free report later this year, and we also hope to make the anonymised data from the survey available to those who want to do further analysis. So keep an eye out for an email about the survey from Ipsos Mori, and a follow up phone call to hear your views and experiences.

Your responses will help us create a true picture of the impact measurement landscape in theUK. We hope to use this to help set the agenda on impact measurement over the next few years, and help make it easier for charities to prove the difference they make.

We’re grateful to our funders at the Big Lottery Fund, the City Bridge Trust ,The Northern Rock Foundation and the Paul Hamlyn Foundation for making this project possible.

Five reasons to get involved with impact reporting

The demands for charities and social enterprises to provide more information about their work are increasing from all angles. Funders, supporters, beneficiaries, policy-makers, even the media are keen to know about the difference charities make to the people they help and the causes they champion.

But communicating impact through reporting can be a challenge: even when we are sure what we are doing is achieving something, proving it can be difficult.

With money tight and resources stretched, why should charities care about impact reporting? Why is it worth the investment and what’s in it for them?

Here are five reasons why we think impact reporting is important.

It helps engage donors and funders and makes them passionate about your work

Engaging supporters involves telling them a compelling story about what you do. Talking about the impact you make is an essential part of this: in order to win the trust of your stakeholders, you need to tell the real story of what you do and how you do it. We hope that the impact reporting principles that NPC has developed in collaboration with a host of other sector bodies can help you develop this narrative into something funders really care about.

It diverts your donors from focusing on admin costs

Funders and donors can only make funding decisions with the information you give them. At the moment, much of that information is financial based on the accounts charities must produce every year. This is contributing to the myth that you can assess a charity’s effectiveness based on overheads, fundraising expenditure and other admin costs. Impact reporting provides supporters with the right information they need to make an informed decision.

It boosts your accountability and credibility

Charities have a privileged position in society: they are some of the most trusted organisations in the UK and enjoy tax breaks and other advantages in recognition of the vital work that they do and the people they help. Impact reporting is a way of being accountable to your supporters, staff, beneficiaries and the general public and acknowledging the special status charities have.

Impact reporting can also be important for maintaining this credibility—particularly when a charity hits the headlines. The US charity, Invisible Children, came under fire for its Kony 2012 campaign partly because it was perceived to have poor accountability (to which the charity has responded).

It motivates and inspires staff

Impact reporting is often thought of as part of a charity’s external communications. But it can have powerful internal applications as well—not least motivating and inspiring your staff. Impact reporting can help track a charity’s progress towards achieving their vision and it enables staff to see how their work is contributing to the over all goals of the organisation.

It’s probably easier than you think to get started

You don’t need fancy measurement systems to report on your impact (although they can help). Much more important is having a clear narrative explaining what you achieve and how for the people you’re trying to help—we’ve suggested five questions you can use to think through how to write an impact report. Charities we’ve worked with have found they’ve made a lot progress very quickly and they actually had more of the information they needed than they realised.

Last night saw the launch of ‘Principles into practice: How charities and social enterprises communicate impact’ from NPC, CDG and ACEVO. Find out more about the launch here, and read the report here.

A revolution in measurement? Using the internet to make high quality evaluation accessible

Over the last three years NPC has been busy working on an idea that we hope will change the way charities measure and understand their impact. Broadly speaking, it’s all about using the internet to put high-quality evaluation within reach for a great many organisations.

The idea is NPC’s Well-being Measure, an online tool that helps organisations working with 11 to 16 year olds get a handle on ‘soft outcomes’, such as self-esteem, emotional well-being and relationships. Based around a set of carefully-designed and researched questions, it’s a way of quantifying outcomes robustly and reliably. Each subscriber to the Well-being Measure gets their own account where they can create, store and manage surveys, and see their results.

The advantage of using online technology is that you can overcome some of the barriers to better evaluation – cutting down the cost of administration, reducing room for error in data capture, automating analysis, and providing all this for a relatively cheap price.

It has allowed us to take a sophisticated method and package it up in an easy-to-use, convenient format. For a modest cost it does what you would otherwise have to pay a great deal for if you commissioned academics or consultants. And unlike other tools, it is designed specifically with charities in mind.

Of course it does not do everything. It can measure how you are doing, assess how well-being has changed as a consequence of a new intervention, and let you compare your well-being outcomes with other similar organisations. But it cannot on its own tell you everything about causation, attribution or additionality, nor can it tell you what to do with the information that you find. There are in any case many, many more issues around evaluation – developing ‘theories of change’, measuring other impacts and communicating results. For all these issues you will still need to employ the other skills NPC (and others) can offer you. But it is a big step forward.

So where are we? The Well-being Measure has been up and running for a while now and we’ve received some great feedback. To hear from three of our customers, you can watch a short video here.

We are immensely excited by the potential of the Well-being Measure and the possibility of applying the concept to charities working with other groups, such as older people or the early years. We also want to apply what we have learned to our other work, for example in helping to inform the partnerships we have created through the Inspiring impact programme. Overall, it’s something that we see as an important development in NPC’s work so we promise to keep you up-to-date with progress.

To read more check out our site www.well-beingmeasure.com, where you can also sign up for updates or visit our blog.

We would like to thank Public Zone for its work in the development of the Well-being Measure. We are also grateful to the funders that have supported it: The Paul Hamlyn Foundation, The Esmée Fairbairn Foundation and The Private Equity Foundation.

We regret that NPC’s Well-being Measure is currently not available to organisations outside the UK.

Picking out the signal from the noise

Evaluating social programmes can resemble a dark art: the terms and methods can be obscure; evaluators don’t like making definitive claims; evaluations often appear to support contradictory conclusions; and the strongest recommendations are usually the need for more research! All this can give the impression that researchers are engaged in a mysterious set of self-serving practices that throw up more questions than answers and hence make the case for more evaluations. Which means evaluation reports are all too often read once then put on the shelf to collect dust.

The tragedy is that a well-conducted evaluation can add a lot to knowledge and understanding of what works and what does not. But it is often difficult to tease this out. The Ministry of Justice’s (MoJ) evaluation of  the HM Prison Service’s Enhanced Thinking Skills (ETS) programme provides a case in point.

Based on promising research from other countries, ETS was a programme of exercises, assignments, role playing and discussions developed by the Prison Service to help change how offenders think about their criminal behaviour. It was accredited for use in custody in 1996.

The MoJ conducted evaluations of the programme in 2002, two in 2003 (study one, study two), 2006, 2009, and 2010. Most of these evaluations used a robust “matched pair” approach that compared the reconviction rates of programme participants against similar offenders who did not participate in the programme. The 2009 study used a randomised control trial—the “gold standard” of evaluation.

On first glance the results seem inconclusive:

• Consistent with international research, the first evaluation found that two years later male offenders who participated in the programme had lower reconviction rates than those who did not.
• Two evaluations (one published in 2003 and the other in 2006) found no difference in the two year reoffending rates of  people who participated in the programme compared to those who did not.
• The 2009 evaluation found that adult male offenders who completed the programme became less impulsive, and hence less likely to reoffend, than those who did not complete the programme.
• The 2010 study found a reduction in one year reconviction rates, as did one of the 2003 studies but only when programme dropouts were excluded.

Based on this summary, these results do not seem very helpful is answering the main question of whether ETS works. But there are answers in the details. For example:

• The 2010 study should count more than the others because it was the most robust of the matched pair evaluations as it controlled for factors that may influence reoffending that were missed out in the previous studies.
• The 2010 study confirmed that proper targetting makes a difference. The programme was found to be more effective for offenders identified as most suitable for the programme than for participants deemed less suitable.
• The 2003 study found that offenders who dropped out of the programme had higher reconviction rates than people who did not participate in the programme at all. It appears that for some offenders (those least suited to ETS) the programme may have increased their risk of reoffending, while decreasing the risk for others, at least in the short-term.
• The 2006 study focused on female offenders, most likely a partial explanation for some of the different results because originally ETS had been developed predominantly by men for men.

Good evaluations provide important and useful findings even if they do not provide unambiguous answers to the central question, does the programme or service work? There is rarely a simple answer to that because different things work for different people. This may be frustrating but should not be surprising.

Few charities can match the evaluation and research resources of the MoJ, but all charities can collect some data to help identify what works, when, and for whom. As the ETS case study shows, this cannot be done in a one-shot deal. Evaluation is not so much a dark art but a process of building your knowledge base brick by brick. The payoff is continuous improvement and greater impact. The alternative is not being sure whether you make much difference or, potentially, cause harm.

Impact measurement: A how-to guide

We are all natural-born researchers. All you need is a sense of curiosity and a dogged nature to continue to ask questions, test new approaches and meticulously measure any change that occurs.

Most of us have this curious and inquisitive nature. It is only natural to want to know the drivers behind, say, changes in human behaviour, or the difference a particular service or intervention has made in a person’s life. This is why I find it hard to believe that impact measurement can sometimes be met with confusion–or even trepidation. Perhaps it is because impact measurement is still shrouded in an air of technical difficulty. Even the term itself can be prone to differences in interpretation. It shouldn’t be like this.

When we use the term impact measurement, all we mean is the tracking (or monitoring, or evaluation or measurement, whatever word you prefer) of the difference your particular charity, organisation, intervention or approach is making. And it is not difficult. Start with your own natural curiosity to think about the types of questions you want to answer. Deciding on the right questions can be the most difficult part of the process. If you aren’t measuring the right things, the results you get are meaningless. This is where theories of change can come in useful, to help you map a logical, causal chain of events that need to happen for you to achieve your final goal or mission.

Once you’ve decided on the right things to measure (or outcomes as we call them at this stage), you need to decide what tools you are going to use to start tracking progress. These tools can be on a sliding scale from the more robust to less robust. On the more robust side, and more straightforward to measure, are hard outcomes: definite, tangible changes that have occurred. For example, gaining a qualification, completing a course or moving into employment. But many charities work with people to improve the softer skills they need for these hard outcomes to happen. These softer skills can be around improving confidence, changing attitudes, fostering a sense of purpose and ownership, improving communication skills or improving someone’s social networks, amongst others.

These soft outcomes can be trickier to measure robustly. Charities often rely on case studies and interviews to show how these softer changes have occurred. The issue with this type of evidence is that although it is rich in detail and incredibly useful (and necessary) to communicate individual stories, they are too prone to bias (because they rely on one person’s own subjective feedback) and there is a risk that they are not representative of a charity’s entire client base.

This is where standardised questionnaires and scales come in useful. Psychologists and sociologists who study these softer skills and emotions have developed (over decades of research) robust scales to measure exactly the types of soft skills mentioned above. If the right scale is chosen and is given before and after an intervention, they can provide extremely robust evidence that change has occurred. What’s more, dozens of these scales are freely available for use online.

Charities needn’t measure the world but starting with a wisely chosen outcome and scale can make a world of a difference.

Get started!

• CES’s guide to theory of change
• NPC’s training on theory of change
• NPC’s well-being measure (a validated questionnaire for children) free taster workshops.
• Forum for Youth Investment’s guide to measuring soft skills in youth programmes
• A list of the most common psychological scales and questionnaires
• Tools for out-of-school time programmes
• Research and evaluation centre for outdoor education programmes

If you’re still confused, NPC’s measurement team can always help!

Blue Monday doesn’t add up

Pundits are arguing today over which day is Blue Monday. Is today the most depressing day of the year or is it in fact next Monday?

The dubious formula behind the day looks like this:

[W + (D-d) x Tq]
______________
 M x Na

where W = weather, d = Debt, T = time since Christmas, Q = time since failing our new year resolutions, M = low motivational levels and Na the feeling of a need to take action.

I agree with Dr Dean Burnett, who poured scorn on the idea in The Guardian today. Burnett criticises the pseudoscience of Blue Monday and argues that the concept is ‘disrespectful to those who suffer from genuine depression… who often face an uphill struggle being taken seriously, especially as “depression” is such a general term.’

Of course it is ridiculous to suggest that a predetermined set of external factors could trigger depression in an entire population one Monday. However, the idea of upacking a wide-ranging concept like “well-being” (or the lack of it) into its component parts seems to me to be a sensible one.

This is something NPC has been pursuing through the development of its Well-being Measure. An online survey tool designed for 11- to 16-year olds, the Well-being Measure enables charities and schools to ask the young people they work with how they feel about their lives. The survey explores eight different aspects of well-being:

1. self-esteem
2. emotional well-being
3. resilience
4. satisfaction with friends
5. satisfaction with family
6. satisfaction with community
7. satisfaction with school
8. life satisfaction

The Well-being Measure is one of the tools and approaches that will be explored in an upcoming half day event that NPC and Third Sector are putting on at the end of the month.

I doubt that speakers will be sharing any  neat formulae at Measuring soft outcomes: How to demonstrate improved services and prove value to funders. But I’m sure there will be fascinating insights and practical tips about how different charities have approached the challenge of understanding and demonstrating the soft outcomes of their services.

• To find out more and sign up click here (you’ll need to be quick as there aren’t many places left).

There should be no I in Impact

Impact without an ‘I’ would, granted, be a lot harder to pronounce. ‘Mpact’ sounds like a late nineties R&B group, not the difference the social sector makes to society. However, more consensus and collaboration would make it much easier to assess impact. That is the fundamental message of the Inspiring impact report, launched today by NPC and Views, and with the endorsement of almost two dozen organisations.

Pretty much everyone agrees that we need to get better at measuring the social sector’s impact. And yet, very few of us seem to agree on how exactly the sector should measure. At the moment, very few charities or social enterprises measure their impact in the same way; even those that work in the same field (eg, mental health) or intervention (eg, counselling). With each organisation developing their own approach, a lot end up reinventing the wheel rather than building on existing approaches. You could forgive them for feeling confused, given they have to pick from thousands of different methods. All this makes assessing your impact feel like a pretty hard slog for charities and social enterprises. And with so many different ways of assessing impact, the data is rarely comparable, making it near-impossible for funders to target resources on the most effective interventions. This means data is not used nearly as often as it should be…. making all this data-collecting feel like a bit of a waste of time. Given the significant costs and (seemingly) unclear benefits, chief execs of charities and social enterprises would be forgiven for asking ”why bother?”

The question is: what’s going wrong? The answer is: not enough collaboration. Too much ‘I’ and not enough ‘we.’ And the fault lies not mainly with frontline organisations but with the organisations that are supposed to support them to assess their impact; including NPC. These organisations-consultants, think tanks, academics-have spent the last decade working largely in isolation to help the social sector measure its impact. The result is complexity and confusion for the organisations we’re supposed to help.

That’s why 30 leaders in the field of social impact measurement, including NPC, got together on 23rd September. We agreed that there are currently too many different ways of measuring, analysing and reporting impact data, and this is leading to duplication and wasted effort. We also agreed it is still too hard and costly for organisations to measure their impact to a high standard. We agreed that we need to work together more if we’re going to overcome these issues. During discussions we came up with five long-term goals:

• Embedding a focus on impact into the leadership and culture of thousands of charities and social enterprises.
• Adopting shared methods across certain fields (such as mental health) and key interventions (such as counselling).
• Making it easier and cheaper for charities to measure their impact by providing appropriate, affordable, and accessible data, tools and systems.
• Funders, commissioners and investors supporting a focus on impact and incentivising impact-based approaches in organisations they fund.
• Creating an effective network of support, linked to shared measurement approaches and following best practice, so that most organisations know of and use support where needed.

If we achieve these goals, it will be easier to do impact measurement to a high standard, and the resulting data will be more useful. In short, impact measurement would be ‘working’ a lot better than it does today.

We’re not going to achieve these goals without a big collective effort. That’s why twelve organisations have agreed to put in some hard graft through the ‘Inspiring Impact Group.’  At NPC, we hope that this group will show just what collaboration can achieve for the sector, and its beneficiaries.

In the impact measurement field, we’ve spent too much time working in isolation. Let’s make sure the next ten years is all about collaboration. By working together ’we’ can achieve a lot more than ‘I’.

Social Impact Analysts Association (SIAA) launches today

Back in May 2009, NPC and Bertelsmann Stiftung broke new ground when they brought together over 200 charity professionals to talk about social impact analysis. Over two thirds of those who came to the Valuing impact conference agreed that an organisation dedicated to helping professional non-profit analysts would be useful. Conversations were continued following the event, and the seeds were sown for a new organisation.

Today, two-and-a-half years on, and after much planning and discussion about what such an organisation would look like, the Social Impact Analysts Association (SIAA) launches with a sold-out ‘unconference’ in central London. The launch will bring together over 100 attendees from non-profits, funders and grant-makers, government, academia and the media.

If you aren’t joining us today, you can follow events as they unfold via live coverage on the SIAA blog or via Twitter with the hashtag #SIAALaunch. After the launch, videos, photos and other content will be available to download from our website.

The morning of the launch will focus on the current state of play for social impact analysis, with speakers including Sir Stuart Etherington, chief executive of NCVO, Michael Green, co-author of Philanthrocapitalism, Caroline Hartnell, editor of Alliance Magazine, Darin McKeever, deputy director, charitable sector at the Bill & Melinda Gates Foundation, and Rob Owen, chief executive of St. Giles Trust. The afternoon will centre on discussing specific challenges practitioners of impact analysis face, and what SIAA’s role will be in helping its members to overcome these issues in the future. The day will conclude with the opportunity to sign up to some discrete tasks with which we can move forward, followed by some well-deserved drinks.

The aim of the conference is to engage with broader questions about what we mean by social impact, who is involved and why it matters, while also pinning down tangible and practical next steps to help move forward the practice of social impact analysis. Our full programme is available here.

SIAA’s launch is part of an ongoing conversation with social impact analysts about how we can best help them. Between September and end of November 2011, SIAA has recruited over 70 founding members from ten different countries, who we have extensively consulted with. SIAA’s launch, with participants coming from twelve different countries, will bring even more people into this conversation. We hope you are as excited as we are to be a part of it.

Sarah Duncan is an intern for the Social Impact Analysts Association (SIAA). To find out more about SIAA, and how to join, visit their website.

Has this impact thing gone too far?

As a Director at NPC you would expect me to be a paid-up member of the impact brigade and you’d be right. That isn’t a paramilitary organisation, although I suspect some in the sector may disagree, but rather a cohort of charities, funders and individuals who recognise the importance of thinking about and measuring the impact of charitable activity.

Impact measurement is now part of the mainstream, but there remain a substantial number of sceptics. Those who doubt the wisdom or the possibility of measuring impact in a meaningful way. I was reminded of how widespread this scepticism still is at a recent conference, where a panel member accused NPC of ‘bludgeoning’ the sector. Strong words and I’ve heard stronger still, but sitting behind them is a sensible question, has this whole impact thing gone too far?

My perspective is heavily influenced by my experience as trustee of a grant making foundation. As a funder we care a great deal about how we can have the maximum impact with our resources, which sometimes seem meagre given the scale of the problems we aim to address. We also care, therefore, about the impact that the charities we fund have, and crucially about the extent to which they have really thought it through. Those that have stand out very clearly, there aren’t that many.  And no grantee has ever sought a discussion with us about how their grant should be monitored, based on their thinking about impact. We would be delighted to have that conversation. Our standard monitoring approach is not always a perfect fit, and we would be open to grantees taking the initiative, indeed we would take it as a very positive sign.

In my view this provides a clear answer to our question of whether the whole impact thing has gone too far. Personally I believe we have a moral responsibility to think about the impact of our activities, but whatever your opinion about that there is also a clear self-interest argument. Those organisations that are willing to challenge themselves about their impact can alter the balance of power with funders in their favour. At the moment that remains the exception not the rule. Far from having gone too far, this impact thing hasn’t gone far enough.