UK Aid, accountability and optimal logo placement

DFID has just produced a new version of its UK Aid logo. While there is general grumbling about the jingoistic addition of the Union Jack and its similarity to the USAID logo – the current iteration is not vastly different than the original – introduced three years ago (one of the first things I blogged about) by the previous government.

These sort of emblems have always made me uneasy. When I worked as a civil servant in Malawi, my printer was branded with a “from the American people” sticker (as was my USB stick). The presence of the sticker made me feel like I should be worshipping some unseen god who delivered me office supplies which only ran on 120 volts.

Douglas Alexander, DFID’s last minister under the Labour government, once said¬†that he wished every DFID-funded classroom would have some notice telling children and their parents that the UK was responsible, and that this would help accountability. The rest of us¬†ridiculed¬†that idea, dismissing it as a Trojan horse for self-promotion.

However, perhaps Mr. Alexander was correct in his assumption that¬†emblazoning¬†everything with “UK Aid” could – in theory – increase accountability. If DFID funded something which utterly failed, then it would be incredibly obvious to everyone around. Just one photo of a derelict Union Jack-stamped school would make for pretty poor press. This might create incentives to make aid more effective.

Yet, if the folks at DFID realize this and are rational – instead of trying to be more effective, it’s much easier just to be more careful with sticker placement. Put stickers on high-profile, “successful” ventures (think bags of food rather than say, good governance) and avoid putting stickers on anything that looks like it might fail. So DFID won’t need to be more effective, just more discerning with their stickers.

If you’re not with us, you’re against us

"Only a Sachs deals in absolutes"

This post could also be titled “Taking credit, part¬†deux.”¬†Writing in the Guardian, Jeffrey Sachs considers the impressive reduction in child mortality rates across sub-Saharan Africa.

The critics of foreign aid are wrong. A growing flood of data shows that death rates in many poor countries are falling sharply, and that aid-supported programmes for healthcare delivery have played a key role. Aid works; it saves lives.

For the rest of us who are still burdened with the ability to question,¬†this narrative seems a little too convenient. While the last decade was characterized by a massive increase in health aid to African countries, many of these countries also experienced significant economic growth and improvements in governance and safety. As Charles Kenny pointed out in his book many of these gains in survival may be technological, a result of to interventions which were made readily available. Of course, some of this was due to aid – but the resulting relationship is much more complex than “aid goes up, infant mortality goes down.”

All of this is not to suggest that health aid did not play a role – it almost certainly did – but waving one’s hand and giving all the credit to aid is a dangerous simplification. It also ignores a significant amount of heterogeneity – some countries did better than others, so we really need to start asking ourselves “why?” before we start patting ourselves on the back.

Yet, it isn’t the simplistic narrative that bothers me, it is what comes after: a declaration that aid skeptics are not only completely wrong, but that they could be responsible for the death of children:

Unfortunately, at every step during the past decade ‚Äď and still today ‚Äď a chorus of aid sceptics has argued against the needed help. They have repeatedly claimed that aid does not work; that the funds will simply be wasted; that anti-malaria bed nets cannot be given to the poor, since the poor won’t use them; that the poor will not take anti-Aids medicines properly; and so on and so forth. Their attacks have been relentless (I’ve faced my share).

The opponents of aid are not merely wrong. Their vocal antagonism still threatens the funding that is needed to get the job done, to cut child and maternal deaths by enough to meet the MDGs by 2015 in the poorest countries, and to continue after that to ensure that all people everywhere finally have access to basic health services.

Emphasis is mine. While Sachs is probably referring to pundits on the other extreme of the distribution, his rhetoric¬†leaves no room for shades of grey; writing what I just wrote doesn’t make me a cautious optimist, it makes me an aid¬†sceptic.

Then he tries to quietly paint aid¬†sceptics¬†as responsible for the deaths of children. Astonishingly, if you read the sentence in bold carefully sentence carefully, it’s clear that Sachs is putting much more weight on reductions in child and maternal death before¬†2015 than after. Does Mr. Sachs not care about children of the future? That interpretation might seem a bit unfair to you. What a shame.

I’ll take an evaluation please, but hold the scientists

You need an econometrician dear, not a doctor

Following the massive kerfuffle over the Lancet article on the child mortality impact of the Millennium Village Project, both the authors of the paper and the journal itself have finally responded.

The first response, by Paul Pronyk of the Earth Institute, is reassuringly humble: the authors accept all of the mistakes highlighted by Gabriel Demombynes and Espen Prydz, and even claim that subsequent results will be analysed in a more transparent manner:

The project will invite an independent panel of experts, including critics of the project, to participate in scrutinising the vital events and survey data and in assessing their validity.

The second response, by the editors of the Lancet, is more defensive, arguing that even after failing to show that the fall in infant mortality in Millennium Villages was due to the the MVP intervention, the study still had merit – pointing to several other results which were not the focus of the study (and in two instances, were not significantly different than the `control ‘ villages). I was perturbed by the final statement, which suggests that more independent oversight by medical science professionals is the solution to our concerns:

To ensure that all future data from the project are fully and fairly evaluated, Prof Jeffrey Sachs, the Principal Investigator of the Millennium Villages project, is establishing new internal and external oversight procedures, including the creation of an International Scientific Expert Advisory Group, chaired by Prof Robert Black, Chairman of the Department of International Health, Johns Hopkins Bloomberg School of Public Health, which will report to the Principal Investigator and also communicate its findings to The Lancet. The goal is to provide a further independent means of verifying the quality of the project’s design and analysis. It is important that this work, which is of considerable significance for understanding how countries scale up multiple complex interventions across sectors, receives proper scientific evaluation before, during, and after publication.

Emphasis is mine. This suggestion for a solution is missing the point: the problem with the evaluation of the MVP isn’t that it needs more scientists (narrowly defined as researchers from the health community) paying attention. The problem with the evaluation of the MVP is that it has too many scientists paying attention. Let me be clear: while researchers from these fields are amazing at what they do well (especially randomized controlled trials) – they are not as adept at the careful statistical analysis needed for non-random, complex impact interventions. This is why – sadly all too frequently – incredible journals like The Lancet publish research which would be laughed out of a graduate-level applied economics seminar.

Now, to be fair, economists and other social scientists probably do enough injustice to the health literature to give your average¬†epidemiologist¬†an¬†aneurysm, but there’s a difference between wallowing in within-discipline ignorance (economists or health researchers choosing not to know any better) and knowing better and choosing the path of least resistance. If one wanted to be overly cynical, the precise reason why the MVP is publishing in top medical journals has less to do with seeking the most appropriate audience for assessing impact and more to do with choosing a less critical one.

If they want to convince the world that the Millennium Villages are a big deal, they need to at least bring in some social scientists with the statistical know-how to properly evaluate the evidence. Let’s hope that Dr. Pronyk’s independent panel of experts will have an econometrician or two, rather than just relying solely on those who have solid record of publishing in The Lancet.

Sachs the rainmaker

"But kemosabe, this would not stand up to a diff-in-diff"

Many of you will already be familiar with the ongoing debate over the efficacy and evaluation of the Millennium Village Project, the brainchild of the Earth Institute’s Jeffrey Sachs. Due primarily to the work of Michael Clemens at the CGD and Gabriel Demombynes at the World Bank, the MVP’s claims of development impact have finally faced substantial scrutiny, although frequently the debate has felt more like a war of attrition than productive discourse.

Enter the Lancet, a reputable medical journal which has a worrying tendency to publish really¬†disreputable¬†social science research, which just published a study by Sachs et al. showing that, over three years, child mortality (under the age of five) has fallen by roughly 25% across nine¬†Millennium¬†Villages. When compared with `control’ villages (which were chosen later and differ from the MVs in many, substantial ways), the drop was even larger – close to 31%.

Suddenly the bells starting ringing: after all the doubt, the MVP is hailed as being successful in reducing child mortality, with the editor-in-chief of the Lancet rallying behind the paper and the Guardian reporting the results with an astonishing lack of scrutiny. Only in the twitterverse/blogosphere has the response been largely negative (Lee Crawfurd disassembles the results of the Lancet article here).

However undeserved, this might have been a good opportunity for the the Earth Institute to¬†bask in its momentary glory. Yet, the results might have already been undermined by awful timing: the Lancet study arrived just days after another¬†study by the World Bank’s¬†Gabriel Demombynes and Karina Trommlerov√° showing absolutely massive decreases in child mortality across most of sub-Saharan Africa in the past few years.

To understand why this is a problem for the Lancet study, consider the table below, which I’ve assembled from results from that study and some figures from the World Bank one (admittedly swiped from Michael Clemens’s¬†post on it).

From the WB study I’ve taken the same nine countries used in the Lancet article, listed their declines in mortality and (assuming a linear trend) calculated the average decline in under-5 mortality per year. One caveat: the years considered in the World Bank study do not¬†necessarily¬†coincide¬†with the timing of the Millennium Villages in their respective countries, so we may be comparing trends from different periods. Even so – these figures still provide a rough idea of the relative magnitude of the mortality decline.

Per-country figures are not available in the Sachs et al. study (which is it a bit worrying in itself), so I can only compare the average declines in these countries to the average decline in all Millennium Villages. What do the results suggest? While child mortality dropped by 24.6 (less children dying per thousand births) over a 3 year period, average declines for all countries in the study are broadly similar: 22.5.

The first and most important thing to take from these results is that the Millennium Villages aren’t vastly outperforming aggregate gains in the same countries. This makes it very difficult for the MVP to claim it is making an impact – it’s a bit like claiming credit for rain in Oxford, when it has been raining all over the UK.

The second thing worth noting: if you look at the above table, taken from the Lancet study, you’ll see that under-five mortality is actually increasing¬†in the control villages. This strongly suggests that control villages are quite different from the rest of the country at large. The Earth Institute has argued that Millennium Villages (and their control counterparts) were selected because they were different – but even if these odd trends in the control villages don’t disqualify them as a¬†counterfactual¬†(which I still think they do), the differences seen here certainly prevent the MVP from having any sort of claims of external validity.

The argument that the Millennium Villages aren’t outperforming the rest of their host countries is not new: Clemens and¬†Demombynes made it over a year ago, when they found that many other claims of `impact’ by the MVP were reflected in national statistics. ¬†Let’s hope the hype from the this study is similarly deflated.

The temptation of the empirical knockout punch

Admit it, you love watching popular development preconceptions being destroyed by cold, hard empirical reality just as much as I do. Despite the slightly queasy feeling I got knowing that Nicholas Negroponte was still out there wasting people’s time and money, these feelings were recently swept away by the satisfaction of knowing that the One-Laptop-Per-Child program was, for the umpteenth¬†time, proven to be ineffective¬†by a rigorous RCT.

These knockouts are especially welcome when a program’s hype far outstretches its evidence base. Such was the case with the Global Alliance for Clean Cookstoves once Hilary Clinton endorsed it, much to the ire of the developmentistas who pointed out that there was nothing new or particularly encouraging about the use of cleaner stoves. This didn’t stop Madeleine Bunting and Julia Roberts (yes, Julia Roberts) from claiming clean cookstoves would work wonders and save millions of lives.

Finally, some more rigorous evidence arrived this month, with the knockout delivered by a group of MIT researchers Рincluding the prolific Esther Duflo Рwho released a new study basically showing cookstoves had little long term impact. Charles Kenny, who resists the temptation to declare a K.O, offers a good summary of the results:

So the results of the MIT study will come as a disappointment to the clean cookstove movement: 2,600 households in India were sold simple improved cookstoves at a highly subsidized price ‚Äďthey cost $12.50 to put in but families paid just 75 cents.¬† Yet after three years, hardly any of the stoves were being used, and most had fallen into disrepair.¬† The stoves ended up no more efficient than traditional models ‚Äďthey burned as much wood‚Äď and levels of indoor air pollution were not improved.

Disheartening results, to be sure.¬† But they shouldn‚Äôt come as a surprise.¬† There are piles of previous evaluations of cookstove programs that may have been less rigorous but still pointed in the same direction.¬† In fact, seventeen years ago, researchers at the Indian Institute of Technology¬†published¬†a review article noting that ‚Äúin spite of quite ambitious programmes‚ÄĚ in support of renewable energy technologies for cooking, they had ‚Äúnot met the expectations of the planners and implementing organisations.‚Ä̬† Amongst the reasons that improved cookstoves in particular were proving a disappointment, the researchers pointed to findings which suggested the stoves did not in fact save fuel, and they were hard to use and maintain (sound familiar?).

So this is an open and shut case, right? Well, not quite. The MIT paper, the Washington Post article which covered it and Kenny all seemed to have missed something: a different RCT on improved cooking stoves which was released just last month. That paper, by Gunther  Bensch and Jörg Peters, studies the impact of a randomised lottery of stoves in rural Senegal. The results suggest that, a year later, households receiving an improved cooking stove used less wood, spent less time cooking meals, reported better indoor air quality and (for women, who presumably did all the cooking) were significantly less likely to have respiratory disease symptoms, eye problems. Nearly all recipients of a stove used it at least seven times a week, in sharp contrast to the lack of use seen in the MIT paper.

Make no mistake: Duflo, Hanna and Greenstone’s study has many advantages over the Bensch/Peters paper. The India paper benefits from a much larger sample size, repeated follow-ups and much more sophisticated measurement techniques. Yet the Senegal paper is still worthwhile because it is – well – written about Senegal and not India. It is perfectly possible for an intervention to fail in one setting but work in another. The J-Pal study strongly suggests that we need to visit treated households more than a year later, as it is possible that the families in the Senegal sample might still stop using the stoves in the future. However, the timing of the latter study provides an excellent opportunity: the intervention was carried out in November, 2009, so if a follow-up survey was conducted this November at the three year mark, we’d be able to identify a long run impact which could either reinforce or undermine the MIT researchers’ result.

Sadly, I doubt anyone will take advantage of this opportunity. The incentives for replication in academia are still incredibly weak, and compelling studies which knocks down popular ideas can be just as persistent as those with novel, positive result. Even if Bensch and Peters return in a year with compelling evidence that cookstoves do have long term impacts in Senegal, it won’t have the quite same impact that the Duflo paper did.¬†We should be a bit more cautious about embracing papers which¬†confirm our priors¬†– a knockout is sometimes just too good to be true.

Invisible Children, Michael Jackson edition

 

Remember that for every successful viral video, there are millions of ones which somehow fail to capture catch on. This dance-laden attempt by the Invisible Children is one of them. Make sure you at least watch until the 2 minute mark:

http://www.youtube.com/watch?v=QWACLKaOC08&amp

I think I need to lie down – but before I do, I felt I should inflict this on the rest of you.

Hat tip to Boing Boing.

Bono meet Russell

Perhaps a little too soon, Baobob/The Economist asserts that the Live Aid mentality has finally died:

¬†But those days of poverty porn at rock¬†concerts (slo-mo famine on giant screens to accompany the music) have¬†also drawn to a close. The thinking about poverty reduction in Africa is¬†less weepy, with greater emphasis on transparency and technology.¬†Innovative new players come from unexpected places, like¬†BRAC, a Bangladeshi organisation. Win or lose, Mr Sach’s bid for the¬†World Bank marks the end of the Live Aid era.

As we’re all aware, the weepiness has returned in full force in the form of Kony 2012.

I don’t have much new to add on this. However, I’m astonished by how strong and well-covered the push-back has been, with most major news outlets taking the time to describe the problems with Invisible Children’s approach.

It might even be possible that the net impact of this whole thing is positive – while we’re always going to be wary about the unchecked desire to do good, the push back might have been enough to inform at least some of the previously-ignorant. We might reach a moment where we all are secretly happy this thing happened, even though we’ll still need to condemn it in order to keep the ignorance in check.

We’re all the 1%, but we’re no Scott Bakula

Did anyone ever notice that Sam Beckett avoiding leaping into African people?

Writing in Foreign Policy, Charles Kenny points out that, when you consider the global income distribution, middle-class Americans are actually part of the global 1%.

So by global standards, America’s middle class is also really, really rich. To make it into the richest 1 percent globally, all you need is an income of around $34,000, according to World Bank economist¬†Branko Milanovic. The average family in the United States has more than three times the income of those living in poverty in America, and nearly 50 times that of the world’s poorest. Many of America’s 99 percenters, and the West’s, are really 1 percenters on a global level.

His point is sound – it’s one that has been made repeatedly by development gurus in response to the Occupy Wall Street protests. He goes on to make the case that we not only need more redistribution from the middle-class within countries, but across them as well.¬†I’m sympathetic to this argument – `local’ inequality surely pales in comparison to the injustice of global inequality, the latter being associated with millions of people still stuck in absolute poverty.

Yet it’s clear from his writing in general that, when considering public policy, Kenny gives the welfare of people within and outside of the US equal weight. This is typical of those involved in international development – we wouldn’t be in this field if we cared only for our own (although maybe that statement is a little self-serving).

These preferences are consistent with John Rawls’s veil of ignorance (discussed earlier here), the argument that policy should be set before we know what position in society we will assume, a bit like Scott Bakula/Sam Beckett in¬†Quantum Leap. For example, we might feel differently about US immigration policy if we thought there was a chance we might be born in Haiti. Kenny touches upon this briefly while discussing Herbert Simon:

Nor did the Western 99 percent “earn” most of their wealth, any more than the top 1 percent “earned” theirs. It’s the luck of where you’re born, according to the late Nobel Prize-winning economist Herbert Simon, who estimated that the benefits of living in a well-functioning economy probably account for 90 percent of individual income.

While I completely agree with this line of thinking – welfare weights shouldn’t be nation-specific, I think it’s a problematic way to convince others¬†to embrace policies which are already unpopular, like taxing the middle class or easing restrictions on¬†immigration.

Why? I think that most people just don’t feel the same way: ¬†average levels of altruism for foreigners are certain to be lower than for other citizens, so we should be wary of making arguments which are too dependent on non-discrimination. People still see citizenship as part of a social contract – we’re all in this boat together, even if we were randomly assigned to it. Those that ended up in leaky boats are not our immediate concern (again, not the way I feel).

There are no simple solutions to the challenge of getting people to broaden their concept of the `boat’ to include other nationals, although one could argue that the international Occupy protests, for all their faults, have actually helped in this regard.

The CSAE Blog

Now for a bit of shameless self-promotion (although this should be the first and last time I’ll bring this up) –¬†we’ve just started up a blog for the Centre for the Study of African Economies, the research centre where I’m based. New posts can be found on the main page here.

Expect updates discussing the centre’s work and also some material from the researchers themselves. The hope is to make it a little more accessible than your average institutional blog. For Twitter users, there’s also a CSAE twitter account now – enjoy!

The Worf effect and NGO rhetoric

Today is a good day to fight poverty!

It is a huge challenge for TV writers to convincingly put their characters in peril. On a show like Star Trek, every new alien threat has to be at least temporarily convincing, even though we all know the crew of the Enterprise will eventually prevail at the end of the day.

One way to convince the audience that a threat is plausible is to let the toughest character on the show be easily defeated by this new threat. If the biggest badass on the show can be conquered so quickly, we suddenly have good reason to believe that the new threat is very real and very credible.

As this is a particularly effective way of creating tension, writers abuse it all the time. However, over time this has the unfortunate side effect of undermining the tough character’s credibility, as audiences rationally update their beliefs about the character’s badassery. Eventually, this `tough’ character ends up appearing comically ineffectual, which itself cripple’s the writer’s attempt to convince the audience that anyone is under any real threat.

This is called the Worf effect, named after the Klingon security officer from Star Trek: The Next Generation, who is repeatedly and effortlessly knocked around by everyone else on the show, despite being renowned for his martial prowess.

Let’s think a little bit about the rhetoric over poverty, charity and aid. NGOs (and many donors) face a credibility problem – they need to convince us that they are effective,¬†bat’lethwielding badasses. Yet at the same time, they want to keep attention on the great challenge of global poverty. There are a number of ways to do this, but quite often, advocates depend on depressing statistics (number of children who die every second), poverty porn and grave warnings.

As one shots – these can be effective at convincing us that the fight against poverty is deadly serious, but over time, we begin to notice that the statistics are still depressing, despite the efforts of well-wishers. We begin to doubt the efficacy of institutions that constantly tell us that things are awful out there – if they were more capable, wouldn’t things be getting better more rapidly?

The answer? Focus on showing¬†your effectiveness, not just telling us about it. Another TV trope term is the informed ability – one we as the audience are supposed to take for granted, but that we never really see in action. We’re supposed to take the efficacy of all these organisations for granted when we donate. This is bound to undermine everything in the long run – Worf needs to win every now and then if we’re going to continue to take him seriously.