Replication rite of passage

Please sir, may I have another dataset?

Over at Cheap Talk, Jeff Ely discusses the Reproducibility project, an attempt by a group of psychologists to replicate every study published in 2008 from three journals:

We should do this in economics.  But there is a less confrontational way to do it. Top departments in experimental economics attract PhD students who want hands on experience in the lab. These are departments like NYU and CalTech. They would benefit the profession, their students, and the reputation of their PhD programs, i.e. everybody concerned, if they were to add as a requirement that every student receiving a PhD must pick one recently published experimental article and attempt to replicate it.

I agree – and I don’t think this should be limited to lab experiments. Grad school is a great time to set norms for replication and to take advantage of cheap PhD labour. Before getting stuck into their own research, students could be required to replicate a study (preferably one that hasn’t been replicated before), with extra points the more citations that study has. Students should the be required present that replication attempt, with the results subsequently being published somewhere public.

This might put more pressure on researchers to be careful. There have already been a couple of high profile conflicts over replication problems  (see the Hoxby-Rothstein dust up and the Acemoglu, Johnson and Robinson cage match with David Albouy). The knowledge that a high-profile results is likely to attract the attention of a grad student with a prerequisite to conquer might make us all a bit more careful.

Such a system would also go a long way to knock down some well-accepted results: when I first started my PhD, the data I first started working on had been used to produce a fairly well known (within the sub-field) result. My supervisor suggested I try to replicate it, and I couldn’t (I’d tell you more, but I didn’t take it to the next level of asking the authors for help).

More more caveat: while replication of lab experiments and econometrics results would be relatively easy, field experiments are probably too expensive and time consuming to be included. All the more reason for us to start funding RCT replications independently.



The temptation of the empirical knockout punch

Admit it, you love watching popular development preconceptions being destroyed by cold, hard empirical reality just as much as I do. Despite the slightly queasy feeling I got knowing that Nicholas Negroponte was still out there wasting people’s time and money, these feelings were recently swept away by the satisfaction of knowing that the One-Laptop-Per-Child program was, for the umpteenth time, proven to be ineffective by a rigorous RCT.

These knockouts are especially welcome when a program’s hype far outstretches its evidence base. Such was the case with the Global Alliance for Clean Cookstoves once Hilary Clinton endorsed it, much to the ire of the developmentistas who pointed out that there was nothing new or particularly encouraging about the use of cleaner stoves. This didn’t stop Madeleine Bunting and Julia Roberts (yes, Julia Roberts) from claiming clean cookstoves would work wonders and save millions of lives.

Finally, some more rigorous evidence arrived this month, with the knockout delivered by a group of MIT researchers – including the prolific Esther Duflo – who released a new study basically showing cookstoves had little long term impact. Charles Kenny, who resists the temptation to declare a K.O, offers a good summary of the results:

So the results of the MIT study will come as a disappointment to the clean cookstove movement: 2,600 households in India were sold simple improved cookstoves at a highly subsidized price –they cost $12.50 to put in but families paid just 75 cents.  Yet after three years, hardly any of the stoves were being used, and most had fallen into disrepair.  The stoves ended up no more efficient than traditional models –they burned as much wood– and levels of indoor air pollution were not improved.

Disheartening results, to be sure.  But they shouldn’t come as a surprise.  There are piles of previous evaluations of cookstove programs that may have been less rigorous but still pointed in the same direction.  In fact, seventeen years ago, researchers at the Indian Institute of Technology published a review article noting that “in spite of quite ambitious programmes” in support of renewable energy technologies for cooking, they had “not met the expectations of the planners and implementing organisations.”  Amongst the reasons that improved cookstoves in particular were proving a disappointment, the researchers pointed to findings which suggested the stoves did not in fact save fuel, and they were hard to use and maintain (sound familiar?).

So this is an open and shut case, right? Well, not quite. The MIT paper, the Washington Post article which covered it and Kenny all seemed to have missed something: a different RCT on improved cooking stoves which was released just last month. That paper, by Gunther  Bensch and Jörg Peters, studies the impact of a randomised lottery of stoves in rural Senegal. The results suggest that, a year later, households receiving an improved cooking stove used less wood, spent less time cooking meals, reported better indoor air quality and (for women, who presumably did all the cooking) were significantly less likely to have respiratory disease symptoms, eye problems. Nearly all recipients of a stove used it at least seven times a week, in sharp contrast to the lack of use seen in the MIT paper.

Make no mistake: Duflo, Hanna and Greenstone’s study has many advantages over the Bensch/Peters paper. The India paper benefits from a much larger sample size, repeated follow-ups and much more sophisticated measurement techniques. Yet the Senegal paper is still worthwhile because it is – well – written about Senegal and not India. It is perfectly possible for an intervention to fail in one setting but work in another. The J-Pal study strongly suggests that we need to visit treated households more than a year later, as it is possible that the families in the Senegal sample might still stop using the stoves in the future. However, the timing of the latter study provides an excellent opportunity: the intervention was carried out in November, 2009, so if a follow-up survey was conducted this November at the three year mark, we’d be able to identify a long run impact which could either reinforce or undermine the MIT researchers’ result.

Sadly, I doubt anyone will take advantage of this opportunity. The incentives for replication in academia are still incredibly weak, and compelling studies which knocks down popular ideas can be just as persistent as those with novel, positive result. Even if Bensch and Peters return in a year with compelling evidence that cookstoves do have long term impacts in Senegal, it won’t have the quite same impact that the Duflo paper did. We should be a bit more cautious about embracing papers which confirm our priors - a knockout is sometimes just too good to be true.

When an RCT would have been really handy

The BBC reports on a study by two psychologists, purporting that staying hydrated can improve grades:

Students who bring water into the examination hall may improve their grades, a study of 447 people found.

Controlling for ability from previous coursework results, researchers found those with water scored an average of 5% higher than those without.

The study, from the universities of East London and Westminster, also noted that older students were more likely to bring in water to exam halls

I don’t believe an RCT is needed to answer every question out there, but it is a little silly in instances like this where a simple intervention could test the same hypothesis: just hand out water bottles to a random group of students before an exam, and see who performs better.

Surely, even controlling for ability (lagged dependent variable, anyone?) students who choose to bring water into exams might be different in some unobservable way. Of course, this doesn’t stop the researchers from making policy recommendations.

Poverty porn, Scandinavian chocolate cake edition

From the Guardian:

Sweden’s minister of culture has been accused of racism after cutting a cake depicting a naked black woman.

Lena Adelsohn Liljeroth was taking part in an event at the Moderna Museet in Stockholm, the capital’s museum of modern art and home to works by Picasso and Dalí. She was invited to cut the cake, an art installation meant to highlight the issue of female genital mutilation. She began, as instructed, by taking a chunk from the cake’s “clitoris”.

The artist, Makode Aj Linde, who created the installation for World Art Day on 15 April, took part in the cake-cutting, with his blackened face and head sticking up next to the cake’s stomach and arms. The cakes “insides” were a gruesome red. A video shows him screaming loudly every time a visitor hacks off another slice of the cake.

The Swedes seem to have mastered the art of combining blackface*, torso-shaped cake design and cringe-worthy cake-slicing into one, massive chocolaty faux pax. Bravo, Ms. Liljeroth, bravo.

Note: Despite being black himself, it appears that Makode Linde went for full blackface anyway. 

Update: nice analysis of the situation and resulting photo by Johan Palme over at Africa is a Country, who suggests that this whole thing might have been staged to snap the photo.

Why predictions fail

If *only* we had included institutions in our prediction model

Over at the Why Nations Fail blog, Daron Acemoglu and James Robinson’s discuss a set of growth predictions made by Paul Rosenstein-Rodan, the father of  the Big Push model, illustrating just how wrong they were:

Acemoglu and Robinson argue that the these predictions were off primarily because the Big Push model ignored politics and institutions:

Of course, things didn’t quite work out that way. In fact, many of the economies about which Rosenstein-Rodan was bullish are not much richer today than they were in 1961. Liberia and Haiti’s economies contracted since then. Angola, Kenya, Nigeria and Uganda haven’t done so well either. We of course know that Afghanistan, India and Pakistan grew more slowly than South Korea, Taiwan, Thailand and Singapore. Argentina and Haiti were no match for Costa Rica, the Dominican Republic and Panama.

The main reason why Rosenstein-Rodan got it so wrong is because he completely ignored the role of institutions and politics.

It’s hard to disagree that Rosenstein-Rodan should have taken these into account – but are they the primary drivers? What about geography, natural resources, export commodity prices, health and the myriad other factors which might drive a country’s growth rate? Without a little more effort, the models lack of effectiveness doesn’t tell us anything about why it is ineffective. I understand that Acemoglu and Robinson consider institutions to be the chief determinant of everything since the beginning of time, but arguing that the Rosenstein-Rodan prediction is wrong because it ignored institutions is a little like arguing that a car missing all four wheels won’t drive because – damn it – it’s also missing four tires.

Slightly more disconcerting: A&J are only displaying a subset of predictions from Rodan’s original paper. Why? My guess is that eye-balling the full dataset doesn’t reveal as much. This calls our for a slightly more rigorous approach than pointing to a few bad predictions. Even better, does someone have the time to crunch the numbers and see if Rodan’s predictions are less useful than predictions being made today?

In which Malawi’s government decides to ignore the constitution

Update: despite earlier signs that the Government would try and block her constitutionally-mandated succession, Joyce Banda has been sworn in as the new president of Malawi, so crisis averted (well, for now). 

It has been over a day since Bingu wa Mutharika suffered a heart attack, and while the Government has still refused to release official confirmation that he has died, the public instead received a chilling indication that Mutharika’s regime will not go out without a fight.

As Kim Yi Dionne pointed out yesterday Malawi’s constitution clearly specifies that the Vice President is first in line for succession in the event that the president dies or is incapacitated. Malawi’s VP, Joyce Banda fell out of favor with the ruling party DPP some time ago, making the prospect of her assuming power unpalatable for Mutharika and the rest of the president’s Cabinet

For a long time the Government remained silent, both on the state of Bingu’s health (despite an alarming number of reports that he is dead, perhaps even before he made it to the hospital) and on the matter of his succession. This has made everyone uneasy – stalling on the Government’s part suggests that they are planning an strategy to block Banda’s succession. Perhaps to put a little bit of pressure on the Government to behave, both the British and American governments announced that they expected Banda to assume the office ASAP.

That silence was broken today when a subset of the cabinet, led by Information Minister and all-around-nefarious-person Patricia Kaliati, announced that Joyce Banda was ineligible due to her behavior while in office (video here – hat tip to Kim), including starting her own political party (which she did after she was thrown out by Mutharika). While they did not announce who the actual successor would be and would not reveal any information about Bingu wa Mutharika himself, this is a clear indication that the Government plans to circumvent the constitution.

It’s getting late in Denmark, so I’ll just offer a few thoughts before signing off:

Normally, I am wary of donors getting to involved in the decisions of recipient governments which are democratically elected, for fear that donor commitments will crowd out the natural accountability generated at the polls. This is quite different – we have a government which is very clearly going to choose to abandon its democratic principles and the international community might have the ability to make that decision too painful to bear.

Here are some options: donors could immediately and credibly rally behind Joyce Banda, not necessarily because she is the best candidate for president, but because Malawi’s constitution makes it clear that she is is the successor. Credible support can come in the form of donor dollars – no government which does not respect this succession should receive aid – the aim should be to make the decision very, very stark for the DPP. If this was combined with the threat of an odious debt sanction – where no new contracts signed by the Government will be enforced in courts, the squeeze might be even tighter.

What donors should not do is fall back on simple rhetoric about accountability. It has to be a clear choice – money flows if Banda succeeds. Money stops if she doesn’t. For it to be credible, donors need to be prepared to walk away – I fear that they won’t be.

On the death of presidents and ignoring the averages

"Perhaps this is an awkward time to talk about the line of succession"

While it is still unclear exactly what has happened, it is looking more and more likely that Malawi’s president, Bingu wa Mutharika, has either died or been severely incapacitated by heart failure.

As Kim Yi Dionne points out, the constitution clearly identifies the vice president , Joyce Banda, as successor. This is where things get a little awkward, as Kim points out:

Shortly after Mutharika and Banda won office (by large margins), President Mutharika had plans for his brother, Peter Mutharika to succeed him in office. Banda was marginalized and eventually expelled from the ruling party. She was also removed from ministerial posts and government attempted to have her removed from office (which is unconstitutional). Facing antagonism from the government, she later formed a new party, the People’s Party.

Whether or not the top brass at the the DPP (Bingu’s party) decide to swallow this bitter pill and embrace Banda remains to be seen.

An equally interesting question which has been bothering me all morning: why hadn’t anyone considered this in advance? Bingu wa Mutharika was a frail 78 year old whose increasingly autocratic rants were seen by many as a sign of impending dementia. The probability that he would die or become severely disabled during a five year term were significant enough to warrant some extra planning.

This isn’t an infrequent occurrence – looking back ten years reveals a startling number of African presidents falling off their perches: Zambia, Chad, Togo, Guinea, Guinea-Bissau, Gabon and Nigeria all had presidents die of poor health whilst in office, all from heart attacks, strokes or cancer. Most of these led to periods of political uncertainty and instability. Even the near-misses are problematic: Mwai Kibaki’s stroke in 2003 led to a substantial shift in both policy and personality – Michela Wrong suggested it drastically hampered the government’s ability to deal with corruption.

Unless the Government of Malawi soon reveals some cunning plan to prevent Joyce Banda from becoming president, it will be obvious that they never really considered this as a potential outcome. Was this sheer stupidity or hubris? Perhaps presidents can’t imagine their own deaths, so do little to prepare for it. There are a few shining outliers out there that might be biasing the results: both Teodoro Obiang in Equatorial Guinea and Mugabe in Zimbabwe have shown a remarkable ability to endure, despite cancer and general evilness.

Voters often associate old-age with wisdom and power, substantially more so in countries like Malawi. Yet these preferences constantly expose political systems to new risks. During the US elections in 2008, an actuarial firm rightly pointed out that John McCain only had a 75% chance of living through a full 8 year term. Despite these unnerving numbers (especially considering who would have been next in line), very little was made of McCain’s age-related morbidity or mortality.

As a 61-year-old-woman, Joyce Banda’s prospects of reaching the end of the current presidential term without succumbing to illness, senility or death are significantly higher than Bingu’s evidence successor, his 72-year-old brother Peter Mutharika. If the Government does find a way to thwart the constitution, and Bingu’s wishes are granted posthumously – likely guaranteeing his brother two terms in office – Malawi’s chances of suffering yet another presidential death rise substantially.