I’ll take an evaluation please, but hold the scientists

You need an econometrician dear, not a doctor

Following the massive kerfuffle over the Lancet article on the child mortality impact of the Millennium Village Project, both the authors of the paper and the journal itself have finally responded.

The first response, by Paul Pronyk of the Earth Institute, is reassuringly humble: the authors accept all of the mistakes highlighted by Gabriel Demombynes and Espen Prydz, and even claim that subsequent results will be analysed in a more transparent manner:

The project will invite an independent panel of experts, including critics of the project, to participate in scrutinising the vital events and survey data and in assessing their validity.

The second response, by the editors of the Lancet, is more defensive, arguing that even after failing to show that the fall in infant mortality in Millennium Villages was due to the the MVP intervention, the study still had merit – pointing to several other results which were not the focus of the study (and in two instances, were not significantly different than the `control ‘ villages). I was perturbed by the final statement, which suggests that more independent oversight by medical science professionals is the solution to our concerns:

To ensure that all future data from the project are fully and fairly evaluated, Prof Jeffrey Sachs, the Principal Investigator of the Millennium Villages project, is establishing new internal and external oversight procedures, including the creation of an International Scientific Expert Advisory Group, chaired by Prof Robert Black, Chairman of the Department of International Health, Johns Hopkins Bloomberg School of Public Health, which will report to the Principal Investigator and also communicate its findings to The Lancet. The goal is to provide a further independent means of verifying the quality of the project’s design and analysis. It is important that this work, which is of considerable significance for understanding how countries scale up multiple complex interventions across sectors, receives proper scientific evaluation before, during, and after publication.

Emphasis is mine. This suggestion for a solution is missing the point: the problem with the evaluation of the MVP isn’t that it needs more scientists (narrowly defined as researchers from the health community) paying attention. The problem with the evaluation of the MVP is that it has too many scientists paying attention. Let me be clear: while researchers from these fields are amazing at what they do well (especially randomized controlled trials) – they are not as adept at the careful statistical analysis needed for non-random, complex impact interventions. This is why – sadly all too frequently – incredible journals like The Lancet publish research which would be laughed out of a graduate-level applied economics seminar.

Now, to be fair, economists and other social scientists probably do enough injustice to the health literature to give your average epidemiologist an aneurysm, but there’s a difference between wallowing in within-discipline ignorance (economists or health researchers choosing not to know any better) and knowing better and choosing the path of least resistance. If one wanted to be overly cynical, the precise reason why the MVP is publishing in top medical journals has less to do with seeking the most appropriate audience for assessing impact and more to do with choosing a less critical one.

If they want to convince the world that the Millennium Villages are a big deal, they need to at least bring in some social scientists with the statistical know-how to properly evaluate the evidence. Let’s hope that Dr. Pronyk’s independent panel of experts will have an econometrician or two, rather than just relying solely on those who have solid record of publishing in The Lancet.

Sachs the rainmaker

"But kemosabe, this would not stand up to a diff-in-diff"

Many of you will already be familiar with the ongoing debate over the efficacy and evaluation of the Millennium Village Project, the brainchild of the Earth Institute’s Jeffrey Sachs. Due primarily to the work of Michael Clemens at the CGD and Gabriel Demombynes at the World Bank, the MVP’s claims of development impact have finally faced substantial scrutiny, although frequently the debate has felt more like a war of attrition than productive discourse.

Enter the Lancet, a reputable medical journal which has a worrying tendency to publish really disreputable social science research, which just published a study by Sachs et al. showing that, over three years, child mortality (under the age of five) has fallen by roughly 25% across nine Millennium Villages. When compared with `control’ villages (which were chosen later and differ from the MVs in many, substantial ways), the drop was even larger – close to 31%.

Suddenly the bells starting ringing: after all the doubt, the MVP is hailed as being successful in reducing child mortality, with the editor-in-chief of the Lancet rallying behind the paper and the Guardian reporting the results with an astonishing lack of scrutiny. Only in the twitterverse/blogosphere has the response been largely negative (Lee Crawfurd disassembles the results of the Lancet article here).

However undeserved, this might have been a good opportunity for the the Earth Institute to bask in its momentary glory. Yet, the results might have already been undermined by awful timing: the Lancet study arrived just days after another study by the World Bank’s Gabriel Demombynes and Karina Trommlerová showing absolutely massive decreases in child mortality across most of sub-Saharan Africa in the past few years.

To understand why this is a problem for the Lancet study, consider the table below, which I’ve assembled from results from that study and some figures from the World Bank one (admittedly swiped from Michael Clemens’s post on it).

From the WB study I’ve taken the same nine countries used in the Lancet article, listed their declines in mortality and (assuming a linear trend) calculated the average decline in under-5 mortality per year. One caveat: the years considered in the World Bank study do not necessarily coincide with the timing of the Millennium Villages in their respective countries, so we may be comparing trends from different periods. Even so – these figures still provide a rough idea of the relative magnitude of the mortality decline.

Per-country figures are not available in the Sachs et al. study (which is it a bit worrying in itself), so I can only compare the average declines in these countries to the average decline in all Millennium Villages. What do the results suggest? While child mortality dropped by 24.6 (less children dying per thousand births) over a 3 year period, average declines for all countries in the study are broadly similar: 22.5.

The first and most important thing to take from these results is that the Millennium Villages aren’t vastly outperforming aggregate gains in the same countries. This makes it very difficult for the MVP to claim it is making an impact – it’s a bit like claiming credit for rain in Oxford, when it has been raining all over the UK.

The second thing worth noting: if you look at the above table, taken from the Lancet study, you’ll see that under-five mortality is actually increasing in the control villages. This strongly suggests that control villages are quite different from the rest of the country at large. The Earth Institute has argued that Millennium Villages (and their control counterparts) were selected because they were different – but even if these odd trends in the control villages don’t disqualify them as a counterfactual (which I still think they do), the differences seen here certainly prevent the MVP from having any sort of claims of external validity.

The argument that the Millennium Villages aren’t outperforming the rest of their host countries is not new: Clemens and Demombynes made it over a year ago, when they found that many other claims of `impact’ by the MVP were reflected in national statistics.  Let’s hope the hype from the this study is similarly deflated.

The headless heart

I just discovered that the Millennium Village Project has a blog, which (surprise!) is less concerned with a sincere discussion of development than with promoting the organisation’s beliefs and activities. One particular entry stood out that I found deeply disheartening:

Instead of fundraising, last year at Carleton University, Students To End Extreme Poverty campaigned to get a question to referendum whereby students would vote on whether or not they would pay an extra $6 per person annually in tuition fees to help finance a Millennium Village. We got 73% of the vote. The 20,000 students at Carleton now contribute over $120,000 annually to Millennium Promise.

Yes, that’s right, the students of Carleton University have locked themselves and their future peers into making yearly contributions to the Millennium Village project. And why is this?

We chose to support Millennium Promise because it offers the most effective approach for providing people with the tools they need to lift themselves out of extreme poverty. It is also the only approach that, if scaled, would see countless lives saved, an African Green Revolution and, amongst other things, every child in primary school.

All of this despite the lack of  evidence that the benefits Millennium Villages provide are:

  • Scalable
  • Sustainable
  • Persistent
  • Transferable – recall that Millennium villages were founded in places they were expected to make the most difference.

$120,000 a year could be used as to contribute to a proper  randomised or even pseudo-randomised study of the Millennium Villages!

The Millennium Villages are obviously welfare-enhancing; few people deny that (the sheer volume of money flowing into these villages virtually guarantees it). What we need to do is know more about whether these impacts would work as general development policy, but the MVP program has little incentive to ask the appropriate questions, because the appropriate questions don’t make for good sound bytes for the media or for potential donors!

Laura Freschi at Aid Watch has also posted today about the problems with discerning the MVP’s impact.

Update: Chris Blattman suggests that testing the impact of the Millenium Villages might be infeasible.

Poverty Safari

Keep a safe distance

Keep a safe distance

Through Aid Watch I stumbled upon this excellent article in the Huffington post by Senegalese businesswoman Magatte Wade. She tackles the implicit condescension in ventures like Jeffrey Sach’s Millennium Village project, slyly comparing it to “polite” racism she experienced in France. The main subject of her wrath is a cultural enrichment tour group organised by New Dawn Associates, a group of academics who take foreigners on guided tours around the Millennium Village. Wade fishes out some slightly perturbing recommendations made in the NDA brochure, including:

Please do not give anything to the villagers – no sweets, cookies, empty water bottles, pens or even money.

and

Please do not eat or drink in public. Many people in the Bugesera District are still suffering from malnutrition, and the public consumption of food or drinks is against the culture of the area.

Firstly, as Easterly points out: if this is one of the holy Millenium Villages, why are people still starving? Secondly, do these statements sound familiar? (Please do not feed the animals). The whole venture smacks deeply of a new, dasterdly form of poverty porn: the poverty safari! You too, from the safety of your 4×4, can get to experience the overwhelming poverty of the Rwandan people, only to escape back to your hotel in the evening.

Easterly is unsurprisingly outraged. This sounds like another case of good intentions gone awry. However, is this truly a case of a bunch of Western academics viewing Africans as cardboard cutouts? An actual visit to the NDA website reveals that most of the staff and the entire top management are actually African. Does this lend this venture any more cred? I really don’t know.

The Wade article from which all this sprung is quite a good read and can be found here.

UPDATE: Hmm, the NDA website seems to have inverted since I first looked at it. The top staff are now all white foreigners. The head is Dr. Michael Grosspietsch, who has responded to Bill Easterly here.