David Parkin’s (1972) study of the Giriama of Kenya has become a classic analysis of the balance between social relationships and material accumulation. Parkin argued that Giriama palm growers who wished to accumulate material wealth were faced with a challenging problem. To accumulate capital, palm growers had to distance themselves from community expectations that they would redistribute their wealth in the form of feasts involving large amounts of meat and palm wine. At the same time, access to land depended on social support. For palm growers to accumulate material wealth, they had to avoid redistributing their wealth  while maintaining the social ties necessary to ensure their access to land. In Parkin’s study, conversion to Islam enabled farmers to solve this problem. Islam prevented men from drinking palm wine and eating meat slaughtered by non-Muslims and allowed them to be more selective about their engagement in relations of reciprocity. Therefore religion provided a justification for refraining from expending one’s wealth on shared consumption without being exposed to accusations of selfishness.

That is from Daniel Mains’s Hope is Cut: Youth, Unemployment and the Future in Urban Ethiopia. Hat tip to my desk mate Stefano, who often makes me feel guilty about how little of the ethnographic literature I have read.

Perhaps you should be more cautious about always treated religion as exogenous in that regression you just ran.

Share

"But kemosabe, this would not stand up to a diff-in-diff"

Many of you will already be familiar with the ongoing debate over the efficacy and evaluation of the Millennium Village Project, the brainchild of the Earth Institute’s Jeffrey Sachs. Due primarily to the work of Michael Clemens at the CGD and Gabriel Demombynes at the World Bank, the MVP’s claims of development impact have finally faced substantial scrutiny, although frequently the debate has felt more like a war of attrition than productive discourse.

Enter the Lancet, a reputable medical journal which has a worrying tendency to publish really disreputable social science research, which just published a study by Sachs et al. showing that, over three years, child mortality (under the age of five) has fallen by roughly 25% across nine Millennium Villages. When compared with `control’ villages (which were chosen later and differ from the MVs in many, substantial ways), the drop was even larger – close to 31%.

Suddenly the bells starting ringing: after all the doubt, the MVP is hailed as being successful in reducing child mortality, with the editor-in-chief of the Lancet rallying behind the paper and the Guardian reporting the results with an astonishing lack of scrutiny. Only in the twitterverse/blogosphere has the response been largely negative (Lee Crawfurd disassembles the results of the Lancet article here).

However undeserved, this might have been a good opportunity for the the Earth Institute to bask in its momentary glory. Yet, the results might have already been undermined by awful timing: the Lancet study arrived just days after another study by the World Bank’s Gabriel Demombynes and Karina Trommlerová showing absolutely massive decreases in child mortality across most of sub-Saharan Africa in the past few years.

To understand why this is a problem for the Lancet study, consider the table below, which I’ve assembled from results from that study and some figures from the World Bank one (admittedly swiped from Michael Clemens’s post on it).

From the WB study I’ve taken the same nine countries used in the Lancet article, listed their declines in mortality and (assuming a linear trend) calculated the average decline in under-5 mortality per year. One caveat: the years considered in the World Bank study do not necessarily coincide with the timing of the Millennium Villages in their respective countries, so we may be comparing trends from different periods. Even so – these figures still provide a rough idea of the relative magnitude of the mortality decline.

Per-country figures are not available in the Sachs et al. study (which is it a bit worrying in itself), so I can only compare the average declines in these countries to the average decline in all Millennium Villages. What do the results suggest? While child mortality dropped by 24.6 (less children dying per thousand births) over a 3 year period, average declines for all countries in the study are broadly similar: 22.5.

The first and most important thing to take from these results is that the Millennium Villages aren’t vastly outperforming aggregate gains in the same countries. This makes it very difficult for the MVP to claim it is making an impact – it’s a bit like claiming credit for rain in Oxford, when it has been raining all over the UK.

The second thing worth noting: if you look at the above table, taken from the Lancet study, you’ll see that under-five mortality is actually increasing in the control villages. This strongly suggests that control villages are quite different from the rest of the country at large. The Earth Institute has argued that Millennium Villages (and their control counterparts) were selected because they were different – but even if these odd trends in the control villages don’t disqualify them as a counterfactual (which I still think they do), the differences seen here certainly prevent the MVP from having any sort of claims of external validity.

The argument that the Millennium Villages aren’t outperforming the rest of their host countries is not new: Clemens and Demombynes made it over a year ago, when they found that many other claims of `impact’ by the MVP were reflected in national statistics.  Let’s hope the hype from the this study is similarly deflated.

Share

At some point Toby Ord walks in the door and convinces you that giving is a moral duty, so you decide to lower the “charity-giving” threshold from a roll of four to a roll of three. The impact of Ord’s words is clear: your expected giving increases from $2.50 to $3.33, about 83 cents higher than before.

Let’s say you then tell the computer to make a roll, and you learn that you have donated to charity. Is Ord’s intervention responsible? We really can’t say – the `charity giving’ result was possible before Ord came in the room, we don’t observe the actual roll and we cannot see if the result would have cleared the pre-Ord threshold. While Ord’s impact is observable in a grander, statistical sense, individual results cannot be attributed to his intervention.

Suddenly, the world becomes a lot more uncertain. Take climate change, for instance. While there is growing evidence that global warming increases the probability of weather-related disasters, it is more difficult to tie individual disasters to global warming, as we cannot say for certain that a flood wouldn’t have happened anyway. This doesn’t stop us from embracing policy to reduce the probability that floods will happen in the future, but we do have to be careful about how we ascribe the blame for individual events.

Interestingly, according to an article in the Economist, death-row inmates in North Carolina are now allowed to apply statistical averages to specific cases:

Mr Robinson was the first person to have his death sentence vacated under North Carolina’s Racial Justice Act. Enacted in 2009, the law lets death-row inmates challenge their sentence (though not the underlying conviction) on grounds of racial bias. If a court finds that in the state, county, prosecutorial district or judicial division at the time of sentencing, death sentences were sought or imposed more frequently on members of one race than another, or were sought or imposed more frequently as punishment for killing members of one race, or if race was “a significant factor” in jury selection, the death sentence will be commuted to life without possibility of parole.

North Carolina’s law—unlike Kentucky’s, the only similar law in force—allows the use of statistical evidence to support an inmate’s claim, rather than requiring clear evidence of discriminatory intent.

This means that death row inmates don’t have to prove that racial bias made a difference in their case, they just have to prove that there is, in a statistical sense, racial bias. While I am happy that policies like this might be useful for pushing back against racism in court decisions in a grander sense (I’m also opposed to the death penalty), I’m uncomfortable with the assumption that a general result can be applied to a single observation. This is akin to saying Toby Ord is responsible for every single successful die roll, effectively giving him credit for $3.33 worth of charity, instead of the $0.83 difference he really made.

Extra points if you can rewrite this blog post using econometric equations.

Share

Please sir, may I have another dataset?

Over at Cheap Talk, Jeff Ely discusses the Reproducibility project, an attempt by a group of psychologists to replicate every study published in 2008 from three journals:

We should do this in economics.  But there is a less confrontational way to do it. Top departments in experimental economics attract PhD students who want hands on experience in the lab. These are departments like NYU and CalTech. They would benefit the profession, their students, and the reputation of their PhD programs, i.e. everybody concerned, if they were to add as a requirement that every student receiving a PhD must pick one recently published experimental article and attempt to replicate it.

I agree – and I don’t think this should be limited to lab experiments. Grad school is a great time to set norms for replication and to take advantage of cheap PhD labour. Before getting stuck into their own research, students could be required to replicate a study (preferably one that hasn’t been replicated before), with extra points the more citations that study has. Students should the be required present that replication attempt, with the results subsequently being published somewhere public.

This might put more pressure on researchers to be careful. There have already been a couple of high profile conflicts over replication problems  (see the Hoxby-Rothstein dust up and the Acemoglu, Johnson and Robinson cage match with David Albouy). The knowledge that a high-profile results is likely to attract the attention of a grad student with a prerequisite to conquer might make us all a bit more careful.

Such a system would also go a long way to knock down some well-accepted results: when I first started my PhD, the data I first started working on had been used to produce a fairly well known (within the sub-field) result. My supervisor suggested I try to replicate it, and I couldn’t (I’d tell you more, but I didn’t take it to the next level of asking the authors for help).

More more caveat: while replication of lab experiments and econometrics results would be relatively easy, field experiments are probably too expensive and time consuming to be included. All the more reason for us to start funding RCT replications independently.

Share

Hat tip to Xeni Jardin.

Share

Admit it, you love watching popular development preconceptions being destroyed by cold, hard empirical reality just as much as I do. Despite the slightly queasy feeling I got knowing that Nicholas Negroponte was still out there wasting people’s time and money, these feelings were recently swept away by the satisfaction of knowing that the One-Laptop-Per-Child program was, for the umpteenth time, proven to be ineffective by a rigorous RCT.

These knockouts are especially welcome when a program’s hype far outstretches its evidence base. Such was the case with the Global Alliance for Clean Cookstoves once Hilary Clinton endorsed it, much to the ire of the developmentistas who pointed out that there was nothing new or particularly encouraging about the use of cleaner stoves. This didn’t stop Madeleine Bunting and Julia Roberts (yes, Julia Roberts) from claiming clean cookstoves would work wonders and save millions of lives.

Finally, some more rigorous evidence arrived this month, with the knockout delivered by a group of MIT researchers – including the prolific Esther Duflo – who released a new study basically showing cookstoves had little long term impact. Charles Kenny, who resists the temptation to declare a K.O, offers a good summary of the results:

So the results of the MIT study will come as a disappointment to the clean cookstove movement: 2,600 households in India were sold simple improved cookstoves at a highly subsidized price –they cost $12.50 to put in but families paid just 75 cents.  Yet after three years, hardly any of the stoves were being used, and most had fallen into disrepair.  The stoves ended up no more efficient than traditional models –they burned as much wood– and levels of indoor air pollution were not improved.

Disheartening results, to be sure.  But they shouldn’t come as a surprise.  There are piles of previous evaluations of cookstove programs that may have been less rigorous but still pointed in the same direction.  In fact, seventeen years ago, researchers at the Indian Institute of Technology published a review article noting that “in spite of quite ambitious programmes” in support of renewable energy technologies for cooking, they had “not met the expectations of the planners and implementing organisations.”  Amongst the reasons that improved cookstoves in particular were proving a disappointment, the researchers pointed to findings which suggested the stoves did not in fact save fuel, and they were hard to use and maintain (sound familiar?).

So this is an open and shut case, right? Well, not quite. The MIT paper, the Washington Post article which covered it and Kenny all seemed to have missed something: a different RCT on improved cooking stoves which was released just last month. That paper, by Gunther  Bensch and Jörg Peters, studies the impact of a randomised lottery of stoves in rural Senegal. The results suggest that, a year later, households receiving an improved cooking stove used less wood, spent less time cooking meals, reported better indoor air quality and (for women, who presumably did all the cooking) were significantly less likely to have respiratory disease symptoms, eye problems. Nearly all recipients of a stove used it at least seven times a week, in sharp contrast to the lack of use seen in the MIT paper.

Make no mistake: Duflo, Hanna and Greenstone’s study has many advantages over the Bensch/Peters paper. The India paper benefits from a much larger sample size, repeated follow-ups and much more sophisticated measurement techniques. Yet the Senegal paper is still worthwhile because it is – well – written about Senegal and not India. It is perfectly possible for an intervention to fail in one setting but work in another. The J-Pal study strongly suggests that we need to visit treated households more than a year later, as it is possible that the families in the Senegal sample might still stop using the stoves in the future. However, the timing of the latter study provides an excellent opportunity: the intervention was carried out in November, 2009, so if a follow-up survey was conducted this November at the three year mark, we’d be able to identify a long run impact which could either reinforce or undermine the MIT researchers’ result.

Sadly, I doubt anyone will take advantage of this opportunity. The incentives for replication in academia are still incredibly weak, and compelling studies which knocks down popular ideas can be just as persistent as those with novel, positive result. Even if Bensch and Peters return in a year with compelling evidence that cookstoves do have long term impacts in Senegal, it won’t have the quite same impact that the Duflo paper did. We should be a bit more cautious about embracing papers which confirm our priors - a knockout is sometimes just too good to be true.

Share

Students who bring water into the examination hall may improve their grades, a study of 447 people found.

Controlling for ability from previous coursework results, researchers found those with water scored an average of 5% higher than those without.

The study, from the universities of East London and Westminster, also noted that older students were more likely to bring in water to exam halls

I don’t believe an RCT is needed to answer every question out there, but it is a little silly in instances like this where a simple intervention could test the same hypothesis: just hand out water bottles to a random group of students before an exam, and see who performs better.

Surely, even controlling for ability (lagged dependent variable, anyone?) students who choose to bring water into exams might be different in some unobservable way. Of course, this doesn’t stop the researchers from making policy recommendations.

Share

From the Guardian:

Sweden’s minister of culture has been accused of racism after cutting a cake depicting a naked black woman.

Lena Adelsohn Liljeroth was taking part in an event at the Moderna Museet in Stockholm, the capital’s museum of modern art and home to works by Picasso and Dalí. She was invited to cut the cake, an art installation meant to highlight the issue of female genital mutilation. She began, as instructed, by taking a chunk from the cake’s “clitoris”.

The artist, Makode Aj Linde, who created the installation for World Art Day on 15 April, took part in the cake-cutting, with his blackened face and head sticking up next to the cake’s stomach and arms. The cakes “insides” were a gruesome red. A video shows him screaming loudly every time a visitor hacks off another slice of the cake.

The Swedes seem to have mastered the art of combining blackface*, torso-shaped cake design and cringe-worthy cake-slicing into one, massive chocolaty faux pax. Bravo, Ms. Liljeroth, bravo.

Note: Despite being black himself, it appears that Makode Linde went for full blackface anyway. 

Update: nice analysis of the situation and resulting photo by Johan Palme over at Africa is a Country, who suggests that this whole thing might have been staged to snap the photo.

Share

If *only* we had included institutions in our prediction model

Over at the Why Nations Fail blog, Daron Acemoglu and James Robinson’s discuss a set of growth predictions made by Paul Rosenstein-Rodan, the father of  the Big Push model, illustrating just how wrong they were:

Acemoglu and Robinson argue that the these predictions were off primarily because the Big Push model ignored politics and institutions:

Of course, things didn’t quite work out that way. In fact, many of the economies about which Rosenstein-Rodan was bullish are not much richer today than they were in 1961. Liberia and Haiti’s economies contracted since then. Angola, Kenya, Nigeria and Uganda haven’t done so well either. We of course know that Afghanistan, India and Pakistan grew more slowly than South Korea, Taiwan, Thailand and Singapore. Argentina and Haiti were no match for Costa Rica, the Dominican Republic and Panama.

The main reason why Rosenstein-Rodan got it so wrong is because he completely ignored the role of institutions and politics.

It’s hard to disagree that Rosenstein-Rodan should have taken these into account – but are they the primary drivers? What about geography, natural resources, export commodity prices, health and the myriad other factors which might drive a country’s growth rate? Without a little more effort, the models lack of effectiveness doesn’t tell us anything about why it is ineffective. I understand that Acemoglu and Robinson consider institutions to be the chief determinant of everything since the beginning of time, but arguing that the Rosenstein-Rodan prediction is wrong because it ignored institutions is a little like arguing that a car missing all four wheels won’t drive because – damn it – it’s also missing four tires.

Slightly more disconcerting: A&J are only displaying a subset of predictions from Rodan’s original paper. Why? My guess is that eye-balling the full dataset doesn’t reveal as much. This calls our for a slightly more rigorous approach than pointing to a few bad predictions. Even better, does someone have the time to crunch the numbers and see if Rodan’s predictions are less useful than predictions being made today?

Share

Share