Yesterday I attended the “Best and Worst of Aid” conference at New York University, hosted by Bill Easterly and the Development Research Institute. This was an event that welcomed aid critics, but a couple of the speakers were openly pro-aid. Lant Pritchett of Harvard, for example, defended the past sixty years of aid as a necessary backdrop for some important, welfare-enhancing policy decisions in poor countries. And Isabel Guerrero, the World Bank’s vice president for South Asia, outlined some examples of official aid that, she said, had really worked.
Guerrero said she had spent twenty-seven years at the World Bank, and, on that basis, you might have expected her to cherry-pick some incredible successes. Her examples may indeed have been great achievements, but there was no way of knowing. When it came to evaluation, it was clear that the World Bank was still stuck in the past.
Guerrero’s first slide on the impact of successful aid programs focused on a rural roads project in Peru. Over a decade, she said, the project had improved access to transportation (you’d hope so), raised literacy and school attendance, and cut poverty by about nine percentage points. What she really meant, of course, is that those changes had occurred in the same region where the roads project was completed. There was no control or even a comparison to the other regions of Peru; causality was anyone’s guess.
Nor were all of her outcomes really outcomes. Who cares if access to transportation is enhanced, if no one feels like their well-being has improved? Only the poverty number really came close to an outcome, and it looked pretty flimsy: less than one percentage point per year for ten years versus an unknown but probably very high base, with no accounting for other influences. After the presentation, a Peruvian student came up to me and said that yes, other anti-poverty programs had been operating in the region at the same time as the roads project.
Now, I realize that it may be difficult to do a controlled study of a roads project that costs hundreds of millions of dollars, but there was no effort at anything but prima facie evaluation. The student said that she had been out to the villages, and that the roads really had helped people. Great, I said – so why didn’t anyone at the World Bank ask them how the project had changed their lives?
Surely, the student countered, the ballot box was a sufficient verdict. The ballot box! Even if all rural Peruvians voted, would they have cast their votes based solely on this one project, which wasn’t even funded by their own government? It was too easy just to ask them about the project directly, I said, not to do so. The cost of conducting a before-and-after survey would have been a tiny fraction of the World Bank’s budget for the project.
I was surprised and disappointed that neither the World Bank nor a Peruvian studying development would have deigned to survey the people affected by the roads project in order to gauge its true impact. This is another manifestation of a mindset we see again and again: if you want to help poor people, the last thing you should do is talk to them.
It’s also a sign that the World Bank still has little idea how to convincingly evaluate its own projects, especially in front of an audience of rigorous academics and experienced practitioners. Guerrero’s other cherry-picked example, a program to help women in India’s Andhra Pradesh state, came under heavy criticism from Pritchett later in the conference for having completely unverifiable results. With so much money at stake, surely we can do better.