Angry post about empirical methods and philosophical plumbing

hendel_thewire_post

“We wish to announce we will no longer be reporting annual murder rates, because they represent a world view in which there is only one conceptualisation of “murder.” Murder is actually a fairly complicated, complex process, and to simply “count” these murders only stands to hide the philosophical basis for considering a crime a murder and ignores theories of change as to how or why murders happen. Anyway, the stats are all juked anyway.”

Perhaps this is nitpicking, but there was brief moment while reading Rosalind Eyben and Chris Roche’s rebuttal post on evidence in policymaking (part of a must-read exchange with Chris Whitty and Stefan Dercon), that nearly resulted in an early-morning brain aneurysm:

Let’s start by insisting that a criterion for rigorous research is that it should be explicit about its assumptions or world-view. We suggest that a weakness in many studies is that they usually focus solely on the methodological and procedural and render invisible their ‘philosophical plumbing’. The evidence-based approaches that Stefan and Chris advocate are imposing a certain view of the world, just as our approaches do. Their claims to the contrary foreclose any possible discussion about the different intellectual traditions in interpreting reality.  Theory invites argument and debate.

This argument is made time and time again with those who are both unfamiliar and intimidated by empirical methods. Let me be clear here: a comparison of means does very little to “impose a certain view of the world.” It is just a comparison of means. If I have run a randomised control trial on fertilizer use, I am answering the question “Did this treatment increase fertilizer on use, on average?” To argue that measurement has some sort of inherent, insidious philosophical underpinning is a dangerous and backward way to approach life. A breathalyser test uses various assumptions to measure a person’s blood alcohol level, but I can’t very well go about rejecting its validity because it doesn’t take into account the power relationship between the cop and the driver.

Can the use of rigorous empirical research be used to support theory or ideology? Of course. Are empirics often insufficient to answer really difficult questions. Of course. It is also the case that economists tend to think about problems a certain way, and this might not always be the way a problem needs to be thought about. Are sociological, anthropological and political methods often just as useful for providing evidence? Of course. Should these results often be considered carefully, keeping in mind the context and the various complexities and confounding factors? Of course. 

But measuring poverty, or infant mortality, while rife with methodological assumptions, does not rely on a certain view of the world, unless you classify “I believe some things should be measured” as a world view. So please, stop rejecting simple statistics as a “different intellectual tradition in interpreting reality” – it is really a very silly thing to say and diverts the argument from what really matters: what tools are best for promoting development, and how best can we implement these tools? Rigorous empirical methods are just another tool in the toolbox. Your view of the world will determine which of these tools you rely on the most.

I swear, I think this blog spends half its time trying to put the die-hard randomistas in their place and the other half trying to put the die-hard qualitatives in their place. I need to have a lie down.

Bad things and the GDP fallacy

There are many who are (quite rightly) worried about whether or not economic growth is sustainable. This has led to a number of attacks on Gross Domestic Project (GDP) as a measure of economic progress. One of the arguments that is repeatedly made is that things that we, as a society, would consider to be bad, lead to an increase in measured GDP. Consider this video of Oxfam’s Katherine Trebeck (video starts at 3:50 – watch it for a minute):

Trebeck, who is arguing that disasters, both big and small, add to GDP, is quoting economist Mark Anielski:

The ideal economic or GDP hero is a chain-smoking terminal cancer patient going through an expensive divorce whose car is totalled in a 20-car pileup, while munching on fast-take-out-food and chatting on a cell phone. All add to GDP growth. The GDP villain is non-smoking, eats home-cooked wholesome meals and cycles to work.

I would write a long post on why this is a fundamentally misguided reading of GDP, but Alex Tabarrock has already done it more succinctly than I ever could:

Imagine that you are your friends are going to see a jazz concert but on your way to the concert you have a little disaster, a fender bender. Instead of seeing the show, you and your friends have a miserable time waiting for the tow truck to come to have your car fixed. Spending on the tow truck and the auto repair counts as GDP but it does not add to GDP because it is counter-balanced by a decrease in spending on jazz, wine and cheesecake. Nothing Tyler says (see above) about gross substitutability changes this fact.

Consider a bigger disaster, the 9/11 attack. First, the point already mentioned, the resources used in the cleanup count as GDP but don’t add to GDP to the extent that they would have been employed on other projects. Now it is true that some of the workers could work overtime which they otherwise would not – this would tend to increase measured GDP more than real GDP since leisure is not measured in the national income and product accounts. Even this factor, however, must be balanced against the overwhelming fact that the destruction of the twin towers meant that tens of thousands of the most productive people in the United States were forced into unemployment or death. Since GDP can also be measured as the sum of wages, rents, interest etc. the immediate effect of all the unemployed and dead was to reduce GDP. Similarly, Hurricane Katrina has destroyed more jobs in New Orleans than it has added (and not all the added jobs represent real additions) hence the Hurricane reduced measured and real GDP.

There’s probably little else that people (including economists and including me) get wrong more than trying to figure out what gets counted as GDP. Tabarrock hasn’t completely diverted worries though – he’s assuming the balance sheets cancel out immediately, when borrowing and saving comes into play, there is a lot of spending which goes towards bad things which, on a year-to-year basis, will look like GDP growth, where the “displacement” effect he discusses is divided over subsequent years. Also, GDP won’t accurately take into account global externalities. Finally, GDP itself will still comprise a lot of bad things at the end of the day (if the car crash is perfectly offset by a drop in spending on jazz and wine, nothing has changed).

Still, the point is that the standard “bad things increase GDP” argument doesn’t always work, once we begin to think the problem through a little more carefully.

Update: a friend replies over Twitter with

@aidthoughts and if the vegan cyclist consumes less expensive leisure and needs less income therefore works less hard and is just as happy?”

I give up, Aid Thoughts will endorse the Happy Planet Index from now on.

Aid as policy

empire

“”The Empire is evil”? Don’t you realise what a silly statement that is? The Empire is actually a complex system of carefully targeted programmes and government departments. You should be asking specific questions like: “Is the Empire’s Death Star project effective at curbing population growth?”"

Lee Crawfurd over at The Roving Bandit recently wrote a compelling post about why the question “Does aid work?” is fundamentally flawed.

The question “does policy work” is jarring, because we immediately realise that it makes little sense. Governments have about 20-30 different Ministries, which immediately implies at least 20-30 different areas of policy. Does which one work? We have health and education policy, infrastructure policy (roads, water, energy), trade policy, monetary policy, public financial management, employment policy, disaster response, financial sector policy, climate and environment policy, to name just a few. It makes very little sense to ask if they all collectively “work” or are “effective”. Foreign aid is similar. Aid supports all of these different areas of policy. My colleagues and I at OPM work on aid-financed projects that support most of these different policy areas in different developing countries.

Lee, as he admits himself, is taking his cue from the combined work of Esther Duflo, Abhijeet Banerjee and Dean Karlan. Some policy questions are becoming more and more answerable: does X work in a given context is something that can be tested and applied. Lee is asserting that since aid is just used to fund policies, thus the question of whether aid works just boils down to the complex task of determining whether or not individual policies work.

Yet, when you examine this assertion a little further, it starts to fall apart. Let’s imagine we lived in a world where we, development economists from aid-giving countries, figured out all the good government policies. We knew exactly which actions developing country governments needed to take to save children’s lives, promote income and job creation, reduce hunger and conflict and so on. In the world which Lee has presented to us, policy works, because we knew what all the best policies were. Would aid work?

It certainly wouldn’t be guaranteed to. Aid isn’t just policy – it’s the transfer of financial resources and technical expertise from one country or entity to another. That transfer is inherently non-trivial: it can create huge differences in power, has the potential to distort the recipient’s decision-making, creates opportunities for rent-seeking and often is used for completely political purposes. We cannot, for instance, only judge a US-funded conditional cash-transfer programme in Afghanistan solely on its microeconomic impact – it has to be viewed within the context of the US’s ongoing military intervention in that country, and its likelihood of long-term success. The recent scrap between Rwanda and its donors over the security situation across the border in the DRC again shows that aid is vastly more complicated than simply choosing effective programmes.

Lee acknowledges some of these differences and potential problems, but then dismisses them as things which are hard to gather robust evidence on. This preference to stick to what we know is somewhat admirable and tempting, but ultimately dangerous: it is incredibly difficult to gather widespread, robust evidence on the effects of aid on the macroeconomy or on local political economy. This should make us more, rather than less, wary of possible deleterious effects.

I’m as equally horrified as Lee by the recent attention that right-wing attacks on aid have been getting in the UK, and I completely agree with him that aid cannot always be  judged as a whole. The question, “does aid work?” doesn’t get us very far in life, especially since we have little concept as to what metric we should be using. That said, we have to tread carefully around the argument that small, neat questions are sufficient for success. I agree that aid should be considered more carefully as a bundle of heterogeneous flows and relationships, but I also believe that “aid” is unique in several key ways, and it is only healthy if we continuously question whether the the things that make aid unique also undermine its effectiveness.

How policies end: not with a bang, but a whimper

Masimba Tafirenyika describes how dire the food security situation in Malawi has become:

Once again Malawi finds itself in a tight spot. A food crisis set off by erratic rains, rising food prices and economic hardships is slowly unfolding. For the first time in several years, the country’s ability to feed its citizens is at risk. Sadly and unexpectedly, Malawi has lost its hard-earned status as an agricultural success story — it used to produce enough maize for its people to eat and still provide a surplus to neighbours. Many are now wondering what went wrong and whether there could be lessons for other African countries.

More than 1.63 million people, or 11 per cent of the population, are facing severe food shortages, according to the World Food Programme, a UN relief agency. Malawi needed $30 million to the end of 2012 to cover the shortfall.

As Tafirenyika hints, this stands in stark contrast to reporting on Malawi over the past few years, where it was heralded as a shining example of how to tackle food security. Five years, ago Celia Dugger wrote in the NYT how the country’s president, Bingu wa Mutharika, despite the protests of many, “ignored the experts” and subsequently dealt with the country’s hunger problems by drastically scaling up its fertiliser subsidy programme. Malawi subsequently enjoyed a spate of bumper harvests and many were quick to tout the large-scale subsidisation as being both a success and worth of replication in other countries. Most notable was the support of Jeffrey Sachs, who’s incessant belief that the fertiliser subsidisation was a policy holy grail led him to write an oddly-appreciative obituary for Mutharika, who died at the end of a thuggish, repressive and disastrous second term in office.

Meanwhile, hunger returns to Malawi, but we have not yet established a convincing narrative. Many economists (including a few on this blog) have pointed out, time and time again, that the fertiliser subsidy programme was fraught with pitfalls, both political and practical. While the recent crisis is probably too complex to fully substantiate these concerns, now would be an appropriate time for the fertiliser advocates to turn their attention to the food situation in Malawi, and begin to ask why. Otherwise, we risk touting a policy that might actually have been a complete failure, or at the very least lacked the sort of robustness that anti-hunger policies desperately need.

Ask not what your country can do for you

bilbo

“Sorry Bilbo, I was going to take you on this *amazing adventure*, but then I checked your expenses from last year, and you seem to be spending your entire budget on food, not travelling.”

Economists can sometimes be a little sceptical of asking people what they want. If we’re trying to provide and finance a public good, for instance, we might be worried that beneficiaries will understate their value of that good to try and get away with paying less for it. Others – often those in the behavioural science camp – can be wary that people may not be reasonably informed as to what is good for them, or might let cognitive quirks and biases undermine their prioritisation.

Over at the African Can End Poverty blog, this scepticism seems to have been extended to Tanzanian businessmen, as Jacques Morisset argues that we should pay less attention to what local firms claim are policy priorities:

Allow me to illustrate. According to the entrepreneurs operating In Tanzania, electricity is their major constraint (85 per cent) followed by access to finance (52 per cent), taxes (37 per cent), and administrative red tape (25 per cent). Source: World Bank. Investment Climate Assessment, 2009. Surprisingly, labor and transports costs are only at the bottom of their concerns (less than 10 per cent). According to this ranking, the priority should be therefore given to reducing electricity costs, increasing access to finance and reducing taxation.

A closer look at the firms’ financial balance sheets provides a different picture. In reality, electricity counts for a marginal share of firms’ operating costs in Tanzania (see Figure). For example, it is equivalent to only 3 per cent for a standard firm operating in the apparel sector. In other words, a decline, say, of 50 per cent in electricity prices would only reduce its costs by 1.5 per cent – hardly a high number for such a big effort. By contrast, transport and labor costs are equivalent to 41 per cent and 38 per cent of its total operating costs. This means that reducing transport costs by only 4 per cent would achieve the same gains for the enterprise than cutting by half its energy costs.

I’m not entirely convinced by Morissets argument: he only presents data on the current breakdown of firm’s operating costs, but no evidence on how firm electricity usage might change if prices did come down. This is a little like arguing that that since poor, stunted children in a rural village only appear to consume maize, there’s little point in subsidising the cost of protein-rich foods.

Morisset admits that electricity access might be an issue, but then goes on to make his argumet based on the static view: that we should target inputs which are currently the most costly for Tanzanian firms. Perhaps this it the right course, but a difficult argument to make without more information on how firms change their behaviour when relative prices change.

Panel beating

panel

Spot the speaker from a developing country. Oh wait, you can’t.

 

In response to the under-representation of women on speaking panels in the tech industry, Rebecca Rosen suggested that speakers pledge to only participate when there will be at least one female speaker. This spurred a fair amount of self-reflection from the development crowd on Twitter, neatly summed up and subsequently critiqued by Duncan Green:

They may not be as extreme as geeky tech events, but lots of development gabfests do indeed feature men on the panel talking to women (and men) in the audience. That violates basic fairness, inhibits the profile and (possibly) career development of half of the potential talent pool, and is likely to distort the agenda and resulting discussion (less focus on care economy, women’s rights etc).

I won’t dwell on the mechanism design here – although whether or not pledges would work and what the resulting equilibrium would be is certainly worth further thought. However, I will point out a more glaring inequity in the typical composition of development panels. Let’s see if you can spot it: above is a photo of a recent IIES panel on using aid to reduce world poverty. Can you spot the missing party?  Perhaps our pledges need to begin with geography before gender.