The temptation of the empirical knockout punch

Admit it, you love watching popular development preconceptions being destroyed by cold, hard empirical reality just as much as I do. Despite the slightly queasy feeling I got knowing that Nicholas Negroponte was still out there wasting people’s time and money, these feelings were recently swept away by the satisfaction of knowing that the One-Laptop-Per-Child program was, for the umpteenth time, proven to be ineffective by a rigorous RCT.

These knockouts are especially welcome when a program’s hype far outstretches its evidence base. Such was the case with the Global Alliance for Clean Cookstoves once Hilary Clinton endorsed it, much to the ire of the developmentistas who pointed out that there was nothing new or particularly encouraging about the use of cleaner stoves. This didn’t stop Madeleine Bunting and Julia Roberts (yes, Julia Roberts) from claiming clean cookstoves would work wonders and save millions of lives.

Finally, some more rigorous evidence arrived this month, with the knockout delivered by a group of MIT researchers – including the prolific Esther Duflo – who released a new study basically showing cookstoves had little long term impact. Charles Kenny, who resists the temptation to declare a K.O, offers a good summary of the results:

So the results of the MIT study will come as a disappointment to the clean cookstove movement: 2,600 households in India were sold simple improved cookstoves at a highly subsidized price –they cost $12.50 to put in but families paid just 75 cents.  Yet after three years, hardly any of the stoves were being used, and most had fallen into disrepair.  The stoves ended up no more efficient than traditional models –they burned as much wood– and levels of indoor air pollution were not improved.

Disheartening results, to be sure.  But they shouldn’t come as a surprise.  There are piles of previous evaluations of cookstove programs that may have been less rigorous but still pointed in the same direction.  In fact, seventeen years ago, researchers at the Indian Institute of Technology published a review article noting that “in spite of quite ambitious programmes” in support of renewable energy technologies for cooking, they had “not met the expectations of the planners and implementing organisations.”  Amongst the reasons that improved cookstoves in particular were proving a disappointment, the researchers pointed to findings which suggested the stoves did not in fact save fuel, and they were hard to use and maintain (sound familiar?).

So this is an open and shut case, right? Well, not quite. The MIT paper, the Washington Post article which covered it and Kenny all seemed to have missed something: a different RCT on improved cooking stoves which was released just last month. That paper, by Gunther  Bensch and Jörg Peters, studies the impact of a randomised lottery of stoves in rural Senegal. The results suggest that, a year later, households receiving an improved cooking stove used less wood, spent less time cooking meals, reported better indoor air quality and (for women, who presumably did all the cooking) were significantly less likely to have respiratory disease symptoms, eye problems. Nearly all recipients of a stove used it at least seven times a week, in sharp contrast to the lack of use seen in the MIT paper.

Make no mistake: Duflo, Hanna and Greenstone’s study has many advantages over the Bensch/Peters paper. The India paper benefits from a much larger sample size, repeated follow-ups and much more sophisticated measurement techniques. Yet the Senegal paper is still worthwhile because it is – well – written about Senegal and not India. It is perfectly possible for an intervention to fail in one setting but work in another. The J-Pal study strongly suggests that we need to visit treated households more than a year later, as it is possible that the families in the Senegal sample might still stop using the stoves in the future. However, the timing of the latter study provides an excellent opportunity: the intervention was carried out in November, 2009, so if a follow-up survey was conducted this November at the three year mark, we’d be able to identify a long run impact which could either reinforce or undermine the MIT researchers’ result.

Sadly, I doubt anyone will take advantage of this opportunity. The incentives for replication in academia are still incredibly weak, and compelling studies which knocks down popular ideas can be just as persistent as those with novel, positive result. Even if Bensch and Peters return in a year with compelling evidence that cookstoves do have long term impacts in Senegal, it won’t have the quite same impact that the Duflo paper did. We should be a bit more cautious about embracing papers which confirm our priors – a knockout is sometimes just too good to be true.