Yesterday I attended the “Best and Worst of Aid” conference at New York University, hosted by Bill Easterly and the Development Research Institute. This was an event that welcomed aid critics, but a couple of the speakers were openly pro-aid. Lant Pritchett of Harvard, for example, defended the past sixty years of aid as a necessary backdrop for some important, welfare-enhancing policy decisions in poor countries. And Isabel Guerrero, the World Bank’s vice president for South Asia, outlined some examples of official aid that, she said, had really worked.

Guerrero said she had spent twenty-seven years at the World Bank, and, on that basis, you might have expected her to cherry-pick some incredible successes. Her examples may indeed have been great achievements, but there was no way of knowing. When it came to evaluation, it was clear that the World Bank was still stuck in the past.

Guerrero’s first slide on the impact of successful aid programs focused on a rural roads project in Peru. Over a decade, she said, the project had improved access to transportation (you’d hope so), raised literacy and school attendance, and cut poverty by about nine percentage points. What she really meant, of course, is that those changes had occurred in the same region where the roads project was completed. There was no control or even a comparison to the other regions of Peru; causality was anyone’s guess.

Nor were all of her outcomes really outcomes. Who cares if access to transportation is enhanced, if no one feels like their well-being has improved? Only the poverty number really came close to an outcome, and it looked pretty flimsy: less than one percentage point per year for ten years versus an unknown but probably very high base, with no accounting for other influences. After the presentation, a Peruvian student came up to me and said that yes, other anti-poverty programs had been operating in the region at the same time as the roads project.

Now, I realize that it may be difficult to do a controlled study of a roads project that costs hundreds of millions of dollars, but there was no effort at anything but prima facie evaluation. The student said that she had been out to the villages, and that the roads really had helped people. Great, I said – so why didn’t anyone at the World Bank ask them how the project had changed their lives?

Surely, the student countered, the ballot box was a sufficient verdict. The ballot box! Even if all rural Peruvians voted, would they have cast their votes based solely on this one project, which wasn’t even funded by their own government? It was too easy just to ask them about the project directly, I said, not to do so. The cost of conducting a before-and-after survey would have been a tiny fraction of the World Bank’s budget for the project.

I was surprised and disappointed that neither the World Bank nor a Peruvian studying development would have deigned to survey the people affected by the roads project in order to gauge its true impact. This is another manifestation of a mindset we see again and again: if you want to help poor people, the last thing you should do is talk to them.

It’s also a sign that the World Bank still has little idea how to convincingly evaluate its own projects, especially in front of an audience of rigorous academics and experienced practitioners. Guerrero’s other cherry-picked example, a program to help women in India’s Andhra Pradesh state, came under heavy criticism from Pritchett later in the conference for having completely unverifiable results. With so much money at stake, surely we can do better.

By Carmine Paolo de Salvo

In this guest post, I wish to focus on my direct experiences of a famous experiment in development, one which is meant to bring prosperity and progress to those that have never experienced it. I refer to the Grameen Bank in Bangladesh, and I am sure that the readers of this blog will know what the Grameen Bank is, so I don’t need to spend any time explaining it.

As with many people around the world, my interest in Grameen Bank was triggered by the award of the 2006 Nobel Peace Prize to it and to its founder, Dr. Muhammad Yunus. After reading “The Banker to the Poor” (Yunus’ bestselling autobiography) and finishing my studies, in September 2009 I went to the Grameen Bank headquarters in Dhaka, Bangladesh, to spend there a month as an intern.

What I want to do in this post is to compare briefly what I had read in Dr. Yunus’ book  and what I actually was able to see during my experience in Bangladesh. I honestly believe that in the book there is little that can be considered false, but, at the same time, the impression that Yunus gives of the Bank’s activities is, in some respects, misleading. My opinion, of course, is not based on any robust statistical evidence, but was rather built directly in the villages in which Grameen operates, among real people, notwithstanding the limiting but necessary presence of an interpreter (I’m not so fluent in Bangla, sorry!). I am not going to share my ideas on the classical critiques that are raised against microcredit (in particular the level of interest rates due to high management costs and the support it gives to a petty form of entrepreneurship, which many don’t consider helpful) because they are already well-known and it would take me too long to repeat them here. I will instead limit my comments to three points that caught my attention and on which I would be interested in hearing feedback from others with greater expertise in the field.

Specifically, my points are:

1. The absence of support to female entrepreneurship, something which is constantly underlined in the book and in everything I read about Grameen’s action as a central feature of its action. It Is completely true that women are the legal holders of the loans and that they physically receive the money, but it is also the case that, a few seconds after receiving it, they pass it on to the father or the husband of the household, who waits in the bank a few metres behind them (I personally saw this scene many times). These women don’t practice any economic activity (as frankly admitted by even some rural Grameen officers). They remain housewives as before, but now they also have to bear the burden of the loan that they have to pay back, whose use they have basically no control over. This does not seem to be a great improvement in their life, I would say. Of course, the fact that they meet with each other on a weekly basis is without any doubt a means for them to overcome their social segregation, but the deep revolution I had read about in the book and in other articles, in my opinion, does not exist.
2. Many of the women that receive the loans are illiterate and some of them don’t even have a clear idea of what the concept of a “year” means. Moreover, sometimes they don’t even know how much money they have actually borrowed. Therefore, I seriously wonder to what extent these people are exposed to the risk of exploitation by the officers that operate in those remote rural areas. Who can control them? Nobody, it appears, and to trick these people, given their conditions, is extremely easy. I can’t of course prove that these misbehaviors are a reality, but I think it is wise to take into account the possibility that they do exist. Of course, Grameen Bank as a whole would not be considered guilty for the misbehavior of any individual employee, but it is legitimate to think that, maybe, the enchanted world described in the literature might not match reality in at least some concrete situations.
3. The role of savings, which is ambiguous. Grameen Bank’s biggest pride is that its loans are disbursed without any request of formal collateral. Anybody that has access to its loans is, however, required to open a savings account. On questioning, some Grameen officers have admitted that in cases of insolvency, the bank can actually access these savings and repay the loan through them, even though this is the last resort. Despite the official declarations, if these savings don’t represent a form of collateral, they are very close.

Despite these observations, I would like to make clear that I consider, on the whole, that the Grameen Bank is a good thing, and I believe that its impact on the rural populations of Bangladesh, who previously did not have access to any form of credit, is beneficial. What does not convince me at all, though, is the description of Grameen’s activities as a social revolution, the beatification that has been afforded to Dr. Yunus, and the hagiographic tone which accompanies most discussions about Grameen, not least in the way Grameen itself talks about its activities.

This final image might summarize my thoughts: at the ground floor of the Grameen skyscraper in Dhaka there is an exhibition to celebrate the Nobel Prize. Visiting this exhibition, believe me, there is one thing that one can never be in doubt of: that the Nobel Prize was given because of their humility.

A similar version of this post will be published, in Italian, on the blog “iMille”, www.imille.org

The opinions I have expressed here are the result of many profitable discussions I had with other interns I met in Dhaka and in particular with my dearest friend, Cecilia Ragazzi, who shared this experience with me. It is fair to admit that my ideas were influenced by her fruitful contribution and I am glad to thank her publicly.

Carmine Paolo De Salvo is an economist working as an ODI Fellow at the Ministry of Finance and Economic Affairs in Zanzibar. Currently, he focuses on national planning and poverty reduction strategies.

As a current employee of the Malawian Budget Division, I read with interest Matt’s blog on technological innovation and I’m sure he’ll no doubt be saddened to hear that this computer system is no further ahead than when he left it a year ago.

However, it struck me that in the field of budgeting at least this trait of ‘leapfrogging’ is sadly by no means specific to technology – whenever reforms are undertaken, invariably ‘international best practise’ (normally from New Zealand or some thoroughly un-governable nation) is the recommended yard-stick for governments to aim at.

Another innovation in budgeting championed by the donor community in recent years was ‘Output Based Budgeting’. Every Kwatcha in the Malawian budget is allocated to specific activities with specific indicators and targets. These indicators include deliverables such as ‘number of meetings attended’ and ‘% of office supplies provided adequately’. Formulating a budget in such a way is a monumental effort that no donor’s government would ever dream of attempting.

Perhaps my favourite example of thoroughly unsuitable ‘best practise’ was a consultant who visited Malawi to make recommendations on how the budget should be classified. Fresh from a trip to Australia where he had been impressed by the ability of government to revise its forecasts when the price of diesel changed by a cent, he proposed that Malawi needed to further disaggregate the budgeting for fuel down into petrol, diesel, paraffin etc. What made this observation particularly startling was that it came:

1. During a 3 day black-out in the Ministry, with a generator unable to cover for the grossly over-stretched national grid.
2. In a building where there are no light switches – all of them are either ‘on’ or ‘off’

I could not help thinking that when it comes managing of government’s energy resources we had bigger problems on our hands than refining our diesel forecasts.

Initially, I was under the impression that this type of ‘best practise’ reform was based on a complete lack of understanding on the part of donors and consultants as to the realities of what is happening in government. While there is on occasion an element of truth in this, I fear the actual reason such interventions keep occurring is both more worrying and difficult to address.

‘Best practise’ reforms serve very well the individuals working for the three parties involved in governance reform: consultants, donors and government workers.

Working in a competitive market-place for services, consultants will seek competitive advantage through ‘innovation’. A ‘fully integrated financial management system’ is sold as the latest panacea for Malawi’s budget. It is hard to generate contracts for ‘helping to get the sums right’ or ‘basic accounting’ even if it is in fact what is required.

As for donors, innovations and reforms to budget processes such as ‘Output Based Budgeting’ are the type of concrete reforms that ambitious employees of  donors crave as they can point to government ‘making changes’ and ‘moving forward’. A budget that adds up does not sell as well with your seniors.

Finally, for government workers, the incentive mechanisms in place whereby training and workshops to discuss reform are particularly lucrative also leave government happy to give the impression of embracing the latest ‘best practise’ reform.

There are clearly no easy answers to breaking this cosy relationship, but I think two areas at least are worth considering further.

1. Making budget support conditional on this type of big-ticket reform can be highly counter-productive. While it may be harder to measure whether government is getting the ‘basics right,’ measuring something that can set government back is worse than no indicator at all.
2. An international effort to take a more mature and open approach aimed at tackling civil service remuneration and incentive packages may be a naive proposal, but one I think that needs to be perservered with.

Mark Miller is currently an ODI Fellow at the Malawian Ministry of Finance.