The fungible and the furious

New, unpleasant information requires careful analysis, not knee-jerk reactions

Last week the medical journal Lancet released an article suggesting that, on average, governments that receive more  health aid divert tend to shift domestic resources away from health.

The paper made some headlines and upset aid critics and much of the global health community. A part of this has to do with a misunderstanding of what the findings mean – a confusion which isn’t helped by  those that propagate incorrect and sensationalist interpretations of the study.

However, a lot of the anger over the results comes from those that do understand the implications of the study, but are angered by an apparent divergence in priorities between the global health community and recipient governments. Both Ranil and Owen Barder talk about this in more detail, although I’ll go through some similar arguments.

These are my scattered thoughts on the whole issue.

First, a quick primer on ‘fungibility’

True fungibility is the ability to take funds intended for one purpose and spend them on something else. If someone gave me £40 and told me to go spend it on food at Tescos, I could easily go spend it on dvds at HMV, because the funds are fungible.

The variation of fungibility that we’re concerned about, which I’ll call indirect fungibility, is the idea that funds earmarked for a specific purpose allows the recipient to divert funds elsewhere. For example, say I need to spend £40 in Tescos anyway to food myself in a week, and someone gives me a £40 gift certificate to shop there. I could spend £80 in tescos (my own money + the gift certificate) but I now prefer to my own £40 on other things. As the gift certificate has “displaced” my expenditure at Tescos, you can say that it the gift is indirectly fungible.

The paper isn’t making the case that health aid is being diverted or misused, but solely that it is correlated with shifts in domestic resources away from health, the same way I would shift my own expenses away from Tescos if you gave me a voucher.

Owen Barder discussed the nuances of the concept of fungibility and indirect fungibility more here.

The narrative that the econometrics can’t pick up

The results in the Lancet Article are reached through heavy-duty statistical (econometric) analysis. Studies like these offer us a window into the long-term processes behind resource allocation in developing countries. Econometric methods can reveal trends that are otherwise invisible to the casual observer, but they are not always useful in explaining the reason behind those trends.

The mistaken narrative that is being wrenched from the article is that donors are being mislead, tricked into letting governments spend their cash elsewhere. In reality, donors pay very close attention to national budgets while they deliberate with governments over future disbursements. Donors, on a yearly basis, will usually know in advance what the recipient’s budget will look like. I spent two years working in the Ministry of Finance’s budget department in Malawi – if there’s anything the donors paid attention to, it was the domestic health budget. Ranil explains is better than I:

…when donors earmark, they also don’t generally expect to peg the Governments’ domestic expenditures. When they do, they make this explicit. In Malawi, for example, under the conditions of the Health Sector Wide Approach, the Government were compelled to maintain real funding levels to the health sector, or donor funding would be reduced. This is easy enough to apply.

This means that visible indirect fungibility is either planned or, at a minimum, condoned by the donor community. Yet many are assuming something truly nefarious is going on. I wonder why? Maybe because the paper’s authors say things like this like this:

“We don’t know what countries are doing with their own money once the donor money comes in,” said Christopher Murray, director of the Institute for Health Metrics and Evaluation at the University of Washington and one of the paper’s authors. Murray said health aid saves millions of lives, but governments need to be more transparent about what they’re spending on.

Reality check: many governments are transparent about what they are spending on, but not every expenditure category has such a massive research and advocacy interest as global health. No one is running around collecting detailed time series data on government expenditures on science and technology expenditure. Just because it isn’t accessible on an internet database, doesn’t mean it isn’t sitting on shelves in the recipient country.

Furthermore, while the econometric results are, in a complicated sense, ‘averages,’ many of those vexed by the results have been assuming that this state of indirect-fungibility holds in very specific cases. Andrew Green at Change.org argues:

Limited budgets force governments to make choices. But in countries facing severe public health epidemics, funding health programming should be at the top of that list.

Yet because the study doesn’t look at indirect fungibility across subgroups of countries with different levels of disease burden, there’s no evidence to suggest that this is indeed a problem for countries with severe public health epidemics (except the broad conclusion that fungibility is worse in sub-Saharan Africa).

In fact, these concerns are brought up in one of the two criticisms of the study which were published alongside it in Lancet. Yet neither the media nor the blogging community has bothered to pick up on these criticisms, like one put forward in the second paper:

Although Lu and colleagues show that crowding out  is a real and widespread occurrence, it does not happen in all countries for which international aid for health increases. In countries in which it does take place, it seems to be the result of deliberate policy choices, which have to be reviewed on a case-by-case basis if alternative options are to be widely promoted.

The blogging community has skipped the most important step: critiquing the methods of the paper

Much of the blogging community seems keep on brandishing the results of statistical studies as truth, then arguing the consequences, rather than discussion whether or not the results might be accurate or not. Many of the development twitters (no names mentioned) will tweet “Study shows that X affects Y” without any sense of uncertainty. This is a shame, as it mirrors the way the media reports the results of statistical studies without much critical analysis.

UPDATE: David Roodman provides the much-needed double-fisted econometric analysis chokeslam here.

I have to admit that I’m not at my strongest when discussion complex dynamic panel data methods, but a couple of things jumped out after my first read through:

  • A huge proportion of the data in the study was imputed from missing data ( an astonishing 44% for low-income countries!). Imputation, the practices of replacing a missing data point with an “average” of nearby points (a bit more complicated than that) involves making very heavy assumptions about why data is missing. If data from countries is missing for non-random reasons (especially reasons that are connected to fungibility) the results could be heavily bias. Most modern imputation methods can attempt to correct for this, but are still limited in what they can accomplish.
  • The authors apply their econometric methods rather mechanistically, with no innate preferences on one approach over another, with large deviations in the results depending on these approaches (one method suggests that despite indirect fungibility, total health spending still goes up, where the other suggests that it might be going down!). These deviations suggest that there are biases here that aren’t very well understood. It doesn’t help that most of the methods used by the authors are used because they were “suggested,” rather than some prior belief about where bias might be coming from. Getting help and suggestions is a natural part of research, but you still have to understand why you follow those suggestions.

To be honest, standards for certain types of empirical research differ across fields of study. For all of our dismal failings, economists are intensely critical of each other’s statistical research. This makes us very hard for us to say anything definite about the world – which is why the debate over whether aid causes growth isn’t likely to be solved any time soon. Yet studies that would make a decent journal and raise a few eyebrows would still be taken with a large grain of salt (there are, of course, some exceptions).

Yet journals like the Lancet which, to be honest, don’t have the same grounding in non-experimental statistical methods are happy to publish interesting studies and let the people run wild with them. Please – when you read about empirical results, from any journal, don’t leap to conclusions. Read the papers – and don’t assume that results immediately represent “the true state of the world.”

Beware the dangers of tunnel-vision advocacy.

The negative reaction to the results of the article is coming mainly from the global health lobby. This is perfectly rational – those working in global health work, live, breath, and advocate it all day long. The news that a $1 in health aid may not results in an equivalent rise in health spending is understandably frustrating.

While health is an incredibly important part of aid, development and human welfare, it’s not the only component. Decisions about how much weight to put on different sectors is a difficult task – and may not be one that many recipient governments are particularly good at. However, I see absolutely no reason why global advocacy centred around different sectors could improve upon this – both Owen and I have written about this before – advocacy is too blunt an instrument for country-level planning.

To the health advocates: funding health is important in every country, but it’s not your call to make that it be the biggest spending priority. When I was in Malawi, the government decided that it would spend a lot of its domestic resources on an expensive agricultural subsidy programme. I didn’t agree with it – but it was their call. This limited their ability to put more money into the health sector. I don’t have the knowledge to say which of the two sectors needed the marginal dollar the most, but really, neither can you.

3 thoughts on “The fungible and the furious

  1. Ranil Dissanayake

    April 16, 2010 at 3:16pm

    Excellent title for an excellent piece.

    I particularly think your part on the statistical methods should be reproduced in every outlet that reported on this. Aid Thoughts is covered, lets hope the WaPo follows suit!

    Here’s another big missing part of the discussion, one I’ll blog about next week: audit.

    ALL OF THE ASSUMPTIONS ABOUT CORRUPTION ETC HAVE AN EASY METHOD OF VERIFICATION: CHECK THE AUDITS.

    You can’t worry about ‘Africa’ mis-spending money, because ‘Africa’ does not spend as one entity. You worry about a specific Government mis-spending and all Governments should be audited. If they aren’t there’s a simple solution: don’t give aid till they audit.

    I think it’s totally disingenuous of an anti-aid blog like Aid Watch saying that the burden of proof is on donors to prove that their funding doesn’t facilitate ‘bad’ spending.

    Not to put too fine a point on this, but this is pure balls. The burden of proof never shifts, regardless of aid etc: it’s always on the Government audit properly and transparently. Why? because the people who are affected most are the domestic players, and it affects both domestic and aid resources. Bad spending has no place in a developing country whether its facilitated by aid or not. The Government should always be auditing.

    Donors should never be auditing a sovereign state, because it’s not their money to audit. They can audit aid programmes, but should only be pressing for the state to get a third party to audit.

    And the big surprise? Every single OECD donor agency is already advocating for this.

    This whole issue is a shambles of misinformation, misinterpretation and a bunch of people without nearly enough experience of actual budget allocation and resource tracking in Africa pontificating.

  2. Ranil Dissanayake

    April 16, 2010 at 3:17pm

    btw, lest that be described as a rant, you should have seen it before I went through it with a ‘delete and moderate’ mindset.

  3. April Harding

    April 18, 2010 at 1:59pm

    Great post Matt.
    Ranil – if you come up with an app that will “delete and moderate” ranting blog posts or comments, please share it with me.

    I agree the discussions that have followed the publication of the Lu et al piece are disappointing.
    This morning I was reading a WSJ piece http://bit.ly/bWkSwf
    reviewing what is known about the causes of the massive, structural fiscal deficit in Greece. There is a very long list of documented problems: corruption/ fraud/ avoidance of tax collection (the problem is so severe that you are considered a dolt if you pay your taxes fully); significant corruption/ under the table payments to access healthcare; really large portions of public sector staff in all areas who don’t actually do anything at all ( a school was mentioned which has more teachers than staff); significant corruption is medical equipment procurement. The amount of corruption is so profound that if Greece improved even to the levels of Spain, their deficit would be under control (calculation from a Brookings paper by Dani Kaufmann).
    We know full well that all these problems are certainly as bad, and mostly likely far worse in Sub-saharan Africa where most health aid goes. If health aid is fungible anywhere in the ranges estimated by Lu et al, surely the takeaway message is that big health aid donors should be EXTREMELY concerned with the overall quality of government spending – since it is a direct determinant of the impact of their aid spending. No?
    And seriously, if every OECD donor agency really wanted these audits – they could easily make them happen. All they’d have to do is say, together, that no more aid will flow without the audits.

Comments are closed.