In 2003 Dr. Allen Roses, a senior executive of GlaxoSmithKline, admitted at a scientific meeting inLondonthat most patients do not benefit from taking the company’s drugs. He noted that “most drugs work in 30 to 50 per cent of people.” This was an open secret to those in the pharmaceutical industry but surprising to many outside. Was this admission a gaffe, refreshing sincerity, or clever marketing of “pharmacogenomics” – the use of human genetics to develop more personalised, and hence more effective, drugs? More relevant to the charitable sector, how many chief executives of charities would dare to admit that their services help less than half of their users? And what would their funders say if they did? Would they applaud the charity for being effective, or stop funding because of ineffectiveness?
The comparison between the pharmaceutical sector and human service charities puts the challenges facing charities of measuring effectiveness into perspective. The Association of the British Pharmaceutical Industry estimates that almost £5 billion was spent on research and development in 2007. Medical drugs are only allowed on the market after being subjected to extensive testing using the gold standard of evaluation, randomised control trials. By contrast, no “proof” of effectiveness is required when setting up a charitable programme and allocating funds to measure effectiveness is often an after-thought if it happens at all. And typically we are far from having sufficient data to distinguish what works for different individual characteristics.
So at the minute we do not know what is a reasonable level of effectiveness for charities. The experience of the pharmaceutical industry suggests that a claim by any charity that all or possibly even most of its users are helped in the long-term by its services is implausible. At NPC we very much welcome the growing interest in measuring and demonstrating effectiveness. We will know when we have succeeded when chief executives of charities have the confidence to make comments like those of Dr. Roses—with robust evidence to back them up—without fear of a backlash.
Pingback: work time life space » Better than placebo?
Excellent post. Wonder if you could also include side-effects in the debate. What levels of unintended adverse reactions provoked by charitable interventions would it be reasonable to tolerate? And how mature would we all have to be to start discussing that openly and fully?
I really welcome the idea that the sector could become sufficiently confident to engage in this debate. I hope it’s not too defeatist to say I’m not holding my breath….
The closer we can come to measuring cost-per-outcome – requiring defined and measurable outcomes – instead of just cost-per-output, the easier it will be to overcome the percent effective hurdle.
It’s striking though that this pharma guy (a)knows what his hit rate is, and (b)will get more money into his enterprise if he increases it. Charities don’t have the latter, and therefore don’t normally know the former, for which they can’t really be blamed.
Caroline: Thanks for the comment. This is a critical issue. If funders (trusts and commissioners) don’t seek evidence of impact in making decisions of what to grant, are charities obliged to track their impact? I think there is in that there is a general obligation to have integrity. That is, if the organisation’s mission is to do X, then it should track (to the best of its abilities and recognising all the challenges around generating evidence) the extent it actually achieves X, regardless of what funders think. One can choose not to track impact, of course, and possibly still thrive, but at the cost of being criticised, fairly, I think.
David: broadly I agree, though ‘obligation’ is a pretty wobbly concept. Incentives are much clearer and frankly, as a charity CEO, I’m amazed at how little incentive there is to either create much impact or to measure it.
What you’re describing is a ‘God’s eye view’: “people should track their effectiveness for purposes of their own learning irrespective of the interest of any external party in that”. But people respond to incentives… which, honestly, are to be good at raising funds rather than to be good at creating impact.
Caroline: As a consultant / researcher I have the luxury of making such pronouncements rather than having to play the cheerleading / fundraising role of the chief exec! I guess what we, NPC, want to do is help the sector get to the point so that the three key incentives to measuring impacts support each other: 1) its the right thing to do, 2) to help raise funds, and 3) to find out whether one is having a positive impact.