If you’re not with us, you’re against us

"Only a Sachs deals in absolutes"

This post could also be titled “Taking credit, part deux.” Writing in the Guardian, Jeffrey Sachs considers the impressive reduction in child mortality rates across sub-Saharan Africa.

The critics of foreign aid are wrong. A growing flood of data shows that death rates in many poor countries are falling sharply, and that aid-supported programmes for healthcare delivery have played a key role. Aid works; it saves lives.

For the rest of us who are still burdened with the ability to question, this narrative seems a little too convenient. While the last decade was characterized by a massive increase in health aid to African countries, many of these countries also experienced significant economic growth and improvements in governance and safety. As Charles Kenny pointed out in his book many of these gains in survival may be technological, a result of to interventions which were made readily available. Of course, some of this was due to aid – but the resulting relationship is much more complex than “aid goes up, infant mortality goes down.”

All of this is not to suggest that health aid did not play a role – it almost certainly did – but waving one’s hand and giving all the credit to aid is a dangerous simplification. It also ignores a significant amount of heterogeneity – some countries did better than others, so we really need to start asking ourselves “why?” before we start patting ourselves on the back.

Yet, it isn’t the simplistic narrative that bothers me, it is what comes after: a declaration that aid skeptics are not only completely wrong, but that they could be responsible for the death of children:

Unfortunately, at every step during the past decade – and still today – a chorus of aid sceptics has argued against the needed help. They have repeatedly claimed that aid does not work; that the funds will simply be wasted; that anti-malaria bed nets cannot be given to the poor, since the poor won’t use them; that the poor will not take anti-Aids medicines properly; and so on and so forth. Their attacks have been relentless (I’ve faced my share).

The opponents of aid are not merely wrong. Their vocal antagonism still threatens the funding that is needed to get the job done, to cut child and maternal deaths by enough to meet the MDGs by 2015 in the poorest countries, and to continue after that to ensure that all people everywhere finally have access to basic health services.

Emphasis is mine. While Sachs is probably referring to pundits on the other extreme of the distribution, his rhetoric leaves no room for shades of grey; writing what I just wrote doesn’t make me a cautious optimist, it makes me an aid sceptic.

Then he tries to quietly paint aid sceptics as responsible for the deaths of children. Astonishingly, if you read the sentence in bold carefully sentence carefully, it’s clear that Sachs is putting much more weight on reductions in child and maternal death before 2015 than after. Does Mr. Sachs not care about children of the future? That interpretation might seem a bit unfair to you. What a shame.

Taking credit

When you have a moment, I'd really like to see your source for that

While the Millennium Village Project’s shaky claim of reducing child mortality resulted in an impressive backlash, these sorts of assertions are not uncommon. Very frequently, donors, NGOs and philanthropists make unsubstantiated claims to impact which go unchallenged, either because they go unnoticed by those who know better or because we’re all just too busy to raise the alarm every time someone makes a bogus claim (or, perhaps, we’re being funded by said entity).

For example, take this tweet by Oxfam international:

That’s quite a claim. What does Oxfam have to back up this claim? The tweet links to an article in the Ghana Business News:

Speaking at a ceremony in Bolgatanga to introduce phase II of the project and to present the donation g, Mrs Rosemary Anderson Akolaa,, Health Advocacy Manager of Oxfam lauded the effort of the TBAs, the Community Health Committees and other stakeholders for their effort at bringing reducing mortality rate in the Region by seven per cent in 2010.

So, one of Oxfam’s managers in Ghana made the claim – what is it based on? To make this claim, Oxfam needs to:

  1. Describe the data it is using to estimate the 7% drop in maternal mortality.
  2. Convincingly show us that this drop is due to Oxfam’s (and partner’s) intervention. For example, did maternal mortality in the Upper East region fall faster than in other regions which did not receive the intervention?
As far as I can tell, Oxfam has done neither of these. Can we stop making claims we haven’t yet made an effort to back up?

The limits of handwaving and fancy graphs

Hans Rosling is quickly becoming famous for wowing audiences with fun, dynamic graphics of demographic changes. In the video below (hat tip to A View From The Cave), the audience applauds more watching total fertility decline in colour than it does when he tries to explain why it is happening.

I really like Rosling’s approach to making complicated data more interesting to the general public and his belief that secular trends can be incredibly revealing. I’m less happy when he draws broad conclusions from what really just looks like extended eyeballing of his charts over time. In the talk from the video, Rosling charts countries by their majority religion (as far as I can see, ignoring within-country variation) and shows that reduction in fertility rates is pretty uniform across countries. He concludes that what matters for fertility isn’t religion but instead:

  • Child mortality
  • Child labour
  • Women’s education and labour force participation
  • The acceptability of family planning

I don’t doubt that, if you had a regression with total fertility on the left side and you controlled for these four things, religion wouldn’t have as large an “impact” on total fertility. However, I think the most pertinent question is: how much does religion matter for these four things? Rosling’s conclusion is that we need to attach these four things directly – a standard public health approach – but if religion (or, perhaps, religiosity, something left out of Rosling’s analysis entirely) is a determinant of these things, the standard approach might be less successful than we would hope.

A Cost-Effective New Initiative That Puts Power in Poor People’s Hands

By Daniel Altman

One of the ideas making waves in global development has been that poor people know better than anyone else how to improve their standards of living.  Some experts have recommended replacing conditional cash transfer programs (rewarding poor people for behaviors such as vaccinating their children) with unconditional cash transfer programs (just giving money to the poor).  But to make real change happen, there is a more direct route.

The poor will only be able to raise their own standards of living if they have power.  Giving them a few dollars here and there may help them to invest in small businesses or improve their access to education, but they still won’t become powerful agents of change.  Perhaps not by coincidence, many poor people live in countries where power is centralized amongst a small elite.

Poor people will only be able to change their lives in a long-term, sustainable way if they have real power in their societies.  Clearly, no government or foreign donor is willing to give them enough money, without any conditions, for this to happen.  Yet there is another, comparatively inexpensive way to do it.

A new development initiative would just give guns to the poor.  As evidence from around the world has suggested, guns are a fast track to power.  Several developing countries have conducted randomized controlled trials of this concept by giving guns to poor villagers in a non-systematic way.  In almost every case, villagers with guns have been empowered to change their living standards much more than villagers without guns.

This kind of initiative has shown promise even in wealthy countries such as the United States.  Here, poor people with guns are disproportionately powerful in society, wielding influence at both the local and national levels.  Moreover, many gun owners have come to believe that their living standards are higher than they actually are, adding psychic benefits to the concrete benefits of gun ownership.

With funding for development programs becoming scarcer every day, the emphasis must be on cost-effectiveness and accountability.  Just giving guns to the poor scores highly on both of these criteria.  After all, what better way to guarantee accountability than at the end of the barrel of a gun?

Daniel Altman is founder and president of North Yard Economics, a non-profit consulting firm serving developing countries, and an adjunct associate professor of economics at the Stern School of Business.

I’ll take an evaluation please, but hold the scientists

You need an econometrician dear, not a doctor

Following the massive kerfuffle over the Lancet article on the child mortality impact of the Millennium Village Project, both the authors of the paper and the journal itself have finally responded.

The first response, by Paul Pronyk of the Earth Institute, is reassuringly humble: the authors accept all of the mistakes highlighted by Gabriel Demombynes and Espen Prydz, and even claim that subsequent results will be analysed in a more transparent manner:

The project will invite an independent panel of experts, including critics of the project, to participate in scrutinising the vital events and survey data and in assessing their validity.

The second response, by the editors of the Lancet, is more defensive, arguing that even after failing to show that the fall in infant mortality in Millennium Villages was due to the the MVP intervention, the study still had merit – pointing to several other results which were not the focus of the study (and in two instances, were not significantly different than the `control ‘ villages). I was perturbed by the final statement, which suggests that more independent oversight by medical science professionals is the solution to our concerns:

To ensure that all future data from the project are fully and fairly evaluated, Prof Jeffrey Sachs, the Principal Investigator of the Millennium Villages project, is establishing new internal and external oversight procedures, including the creation of an International Scientific Expert Advisory Group, chaired by Prof Robert Black, Chairman of the Department of International Health, Johns Hopkins Bloomberg School of Public Health, which will report to the Principal Investigator and also communicate its findings to The Lancet. The goal is to provide a further independent means of verifying the quality of the project’s design and analysis. It is important that this work, which is of considerable significance for understanding how countries scale up multiple complex interventions across sectors, receives proper scientific evaluation before, during, and after publication.

Emphasis is mine. This suggestion for a solution is missing the point: the problem with the evaluation of the MVP isn’t that it needs more scientists (narrowly defined as researchers from the health community) paying attention. The problem with the evaluation of the MVP is that it has too many scientists paying attention. Let me be clear: while researchers from these fields are amazing at what they do well (especially randomized controlled trials) – they are not as adept at the careful statistical analysis needed for non-random, complex impact interventions. This is why – sadly all too frequently – incredible journals like The Lancet publish research which would be laughed out of a graduate-level applied economics seminar.

Now, to be fair, economists and other social scientists probably do enough injustice to the health literature to give your average epidemiologist an aneurysm, but there’s a difference between wallowing in within-discipline ignorance (economists or health researchers choosing not to know any better) and knowing better and choosing the path of least resistance. If one wanted to be overly cynical, the precise reason why the MVP is publishing in top medical journals has less to do with seeking the most appropriate audience for assessing impact and more to do with choosing a less critical one.

If they want to convince the world that the Millennium Villages are a big deal, they need to at least bring in some social scientists with the statistical know-how to properly evaluate the evidence. Let’s hope that Dr. Pronyk’s independent panel of experts will have an econometrician or two, rather than just relying solely on those who have solid record of publishing in The Lancet.

How to avoid your social obligations: adopt a new religion

David Parkin’s (1972) study of the Giriama of Kenya has become a classic analysis of the balance between social relationships and material accumulation. Parkin argued that Giriama palm growers who wished to accumulate material wealth were faced with a challenging problem. To accumulate capital, palm growers had to distance themselves from community expectations that they would redistribute their wealth in the form of feasts involving large amounts of meat and palm wine. At the same time, access to land depended on social support. For palm growers to accumulate material wealth, they had to avoid redistributing their wealth  while maintaining the social ties necessary to ensure their access to land. In Parkin’s study, conversion to Islam enabled farmers to solve this problem. Islam prevented men from drinking palm wine and eating meat slaughtered by non-Muslims and allowed them to be more selective about their engagement in relations of reciprocity. Therefore religion provided a justification for refraining from expending one’s wealth on shared consumption without being exposed to accusations of selfishness.

That is from Daniel Mains’s Hope is Cut: Youth, Unemployment and the Future in Urban Ethiopia. Hat tip to my desk mate Stefano, who often makes me feel guilty about how little of the ethnographic literature I have read.

Perhaps you should be more cautious about always treated religion as exogenous in that regression you just ran.

 

Sachs the rainmaker

"But kemosabe, this would not stand up to a diff-in-diff"

Many of you will already be familiar with the ongoing debate over the efficacy and evaluation of the Millennium Village Project, the brainchild of the Earth Institute’s Jeffrey Sachs. Due primarily to the work of Michael Clemens at the CGD and Gabriel Demombynes at the World Bank, the MVP’s claims of development impact have finally faced substantial scrutiny, although frequently the debate has felt more like a war of attrition than productive discourse.

Enter the Lancet, a reputable medical journal which has a worrying tendency to publish really disreputable social science research, which just published a study by Sachs et al. showing that, over three years, child mortality (under the age of five) has fallen by roughly 25% across nine Millennium Villages. When compared with `control’ villages (which were chosen later and differ from the MVs in many, substantial ways), the drop was even larger – close to 31%.

Suddenly the bells starting ringing: after all the doubt, the MVP is hailed as being successful in reducing child mortality, with the editor-in-chief of the Lancet rallying behind the paper and the Guardian reporting the results with an astonishing lack of scrutiny. Only in the twitterverse/blogosphere has the response been largely negative (Lee Crawfurd disassembles the results of the Lancet article here).

However undeserved, this might have been a good opportunity for the the Earth Institute to bask in its momentary glory. Yet, the results might have already been undermined by awful timing: the Lancet study arrived just days after another study by the World Bank’s Gabriel Demombynes and Karina Trommlerová showing absolutely massive decreases in child mortality across most of sub-Saharan Africa in the past few years.

To understand why this is a problem for the Lancet study, consider the table below, which I’ve assembled from results from that study and some figures from the World Bank one (admittedly swiped from Michael Clemens’s post on it).

From the WB study I’ve taken the same nine countries used in the Lancet article, listed their declines in mortality and (assuming a linear trend) calculated the average decline in under-5 mortality per year. One caveat: the years considered in the World Bank study do not necessarily coincide with the timing of the Millennium Villages in their respective countries, so we may be comparing trends from different periods. Even so – these figures still provide a rough idea of the relative magnitude of the mortality decline.

Per-country figures are not available in the Sachs et al. study (which is it a bit worrying in itself), so I can only compare the average declines in these countries to the average decline in all Millennium Villages. What do the results suggest? While child mortality dropped by 24.6 (less children dying per thousand births) over a 3 year period, average declines for all countries in the study are broadly similar: 22.5.

The first and most important thing to take from these results is that the Millennium Villages aren’t vastly outperforming aggregate gains in the same countries. This makes it very difficult for the MVP to claim it is making an impact – it’s a bit like claiming credit for rain in Oxford, when it has been raining all over the UK.

The second thing worth noting: if you look at the above table, taken from the Lancet study, you’ll see that under-five mortality is actually increasing in the control villages. This strongly suggests that control villages are quite different from the rest of the country at large. The Earth Institute has argued that Millennium Villages (and their control counterparts) were selected because they were different – but even if these odd trends in the control villages don’t disqualify them as a counterfactual (which I still think they do), the differences seen here certainly prevent the MVP from having any sort of claims of external validity.

The argument that the Millennium Villages aren’t outperforming the rest of their host countries is not new: Clemens and Demombynes made it over a year ago, when they found that many other claims of `impact’ by the MVP were reflected in national statistics.  Let’s hope the hype from the this study is similarly deflated.

Innocent until proven likely

Some time ago I wrote about how randomness can obscure culpability. For example, let’s say you program a computer to roll a die for you, but – importantly- it keeps the result hidden.  You also program the computer to automatically donate $5 from your bank account to a charity if the resulting roll is a four or above.

At some point Toby Ord walks in the door and convinces you that giving is a moral duty, so you decide to lower the “charity-giving” threshold from a roll of four to a roll of three. The impact of Ord’s words is clear: your expected giving increases from $2.50 to $3.33, about 83 cents higher than before.

Let’s say you then tell the computer to make a roll, and you learn that you have donated to charity. Is Ord’s intervention responsible? We really can’t say – the `charity giving’ result was possible before Ord came in the room, we don’t observe the actual roll and we cannot see if the result would have cleared the pre-Ord threshold. While Ord’s impact is observable in a grander, statistical sense, individual results cannot be attributed to his intervention.

Suddenly, the world becomes a lot more uncertain. Take climate change, for instance. While there is growing evidence that global warming increases the probability of weather-related disasters, it is more difficult to tie individual disasters to global warming, as we cannot say for certain that a flood wouldn’t have happened anyway. This doesn’t stop us from embracing policy to reduce the probability that floods will happen in the future, but we do have to be careful about how we ascribe the blame for individual events.

Interestingly, according to an article in the Economist, death-row inmates in North Carolina are now allowed to apply statistical averages to specific cases:

Mr Robinson was the first person to have his death sentence vacated under North Carolina’s Racial Justice Act. Enacted in 2009, the law lets death-row inmates challenge their sentence (though not the underlying conviction) on grounds of racial bias. If a court finds that in the state, county, prosecutorial district or judicial division at the time of sentencing, death sentences were sought or imposed more frequently on members of one race than another, or were sought or imposed more frequently as punishment for killing members of one race, or if race was “a significant factor” in jury selection, the death sentence will be commuted to life without possibility of parole.

North Carolina’s law—unlike Kentucky’s, the only similar law in force—allows the use of statistical evidence to support an inmate’s claim, rather than requiring clear evidence of discriminatory intent.

This means that death row inmates don’t have to prove that racial bias made a difference in their case, they just have to prove that there is, in a statistical sense, racial bias. While I am happy that policies like this might be useful for pushing back against racism in court decisions in a grander sense (I’m also opposed to the death penalty), I’m uncomfortable with the assumption that a general result can be applied to a single observation. This is akin to saying Toby Ord is responsible for every single successful die roll, effectively giving him credit for $3.33 worth of charity, instead of the $0.83 difference he really made.

Extra points if you can rewrite this blog post using econometric equations.