When the best of aid becomes the worst

By Daniel Altman

Yesterday I attended the “Best and Worst of Aid” conference at New York University, hosted by Bill Easterly and the Development Research Institute. This was an event that welcomed aid critics, but a couple of the speakers were openly pro-aid. Lant Pritchett of Harvard, for example, defended the past sixty years of aid as a necessary backdrop for some important, welfare-enhancing policy decisions in poor countries. And Isabel Guerrero, the World Bank’s vice president for South Asia, outlined some examples of official aid that, she said, had really worked.

Guerrero said she had spent twenty-seven years at the World Bank, and, on that basis, you might have expected her to cherry-pick some incredible successes. Her examples may indeed have been great achievements, but there was no way of knowing. When it came to evaluation, it was clear that the World Bank was still stuck in the past.

Guerrero’s first slide on the impact of successful aid programs focused on a rural roads project in Peru. Over a decade, she said, the project had improved access to transportation (you’d hope so), raised literacy and school attendance, and cut poverty by about nine percentage points. What she really meant, of course, is that those changes had occurred in the same region where the roads project was completed. There was no control or even a comparison to the other regions of Peru; causality was anyone’s guess.

Nor were all of her outcomes really outcomes. Who cares if access to transportation is enhanced, if no one feels like their well-being has improved? Only the poverty number really came close to an outcome, and it looked pretty flimsy: less than one percentage point per year for ten years versus an unknown but probably very high base, with no accounting for other influences. After the presentation, a Peruvian student came up to me and said that yes, other anti-poverty programs had been operating in the region at the same time as the roads project.

Now, I realize that it may be difficult to do a controlled study of a roads project that costs hundreds of millions of dollars, but there was no effort at anything but prima facie evaluation. The student said that she had been out to the villages, and that the roads really had helped people. Great, I said – so why didn’t anyone at the World Bank ask them how the project had changed their lives?

Surely, the student countered, the ballot box was a sufficient verdict. The ballot box! Even if all rural Peruvians voted, would they have cast their votes based solely on this one project, which wasn’t even funded by their own government? It was too easy just to ask them about the project directly, I said, not to do so. The cost of conducting a before-and-after survey would have been a tiny fraction of the World Bank’s budget for the project.

I was surprised and disappointed that neither the World Bank nor a Peruvian studying development would have deigned to survey the people affected by the roads project in order to gauge its true impact. This is another manifestation of a mindset we see again and again: if you want to help poor people, the last thing you should do is talk to them.

It’s also a sign that the World Bank still has little idea how to convincingly evaluate its own projects, especially in front of an audience of rigorous academics and experienced practitioners. Guerrero’s other cherry-picked example, a program to help women in India’s Andhra Pradesh state, came under heavy criticism from Pritchett later in the conference for having completely unverifiable results. With so much money at stake, surely we can do better.

4 thoughts on “When the best of aid becomes the worst

  1. didier

    March 8, 2010 at 3:50pm

    At what point is a road just road? It’s probably more important, in the case of such a project, to do a standard economic appraisal and environmental impact survey than an MIT PAL style controlled study. What would have been interesting is if the World Bank went back to see if their assumptions on the economic appraisal and EIS were on the mark or not. There is a time and place for everything and you don’t always need the same evaluative tool to determine if you did the right thing.

  2. Guest

    March 8, 2010 at 5:04pm

    I’m not calling for an RCT study in this case. But I am asking that something be done 1) to measure real impacts on well-being and not just intermediate factors, and 2) to try and parse out the effect of the road on those impacts from other contemporaneous programs. You could do a survey, regional comparisons, modeling of system dynamics… there are plenty of options to make your point besides an RCT, all of which would add significantly to looking at the same old trend lines. How can we justify _not_ doing even one of them?

  3. didier

    March 8, 2010 at 8:16pm

    I guess it depends. If the road was put in there to stimulate some economic growth inthe area, you should look at that. My bet is that those “contemporaneous programs” probably came in with the road and in that context you could look at their value added rather than just parsing them out; but in general you are right – it is worth looking at the impact of the project – just make sure you know what that means going in.

Comments are closed.