Archive for category Economics

Random thoughts left lying around

There has been much talk of economists starting up a trial registry for randomised interventions, or at least promoting the use of pre-analysis plans. One of the chief reasons for doing this is to curb data mining – if researchers make it clear up front which hypotheses they plan to test, this will reduce the incentive to report new results, discovered only after the researchers have had time to dig around.

While I think trial registries are worth a try, I have already discussed my worries their effects on the quantity of viable research (even if quality increases). These concerns aside, my question here is: why are trial registries primarily associated with randomised trials? Shouldn’t we also be moving to an equilibrium where all empirical research begins with a published pre-analysis plan?

I suppose the main hurdle is honesty here – for any dataset which already exists, it’s easy for me to download it, mine the data, then base my pre-analysis plan on empirical results I already know to exist. Furthermore, for any given dataset, the number of potential  hypotheses (and thus the number of pre-analysis plans which can be written by different researchers) is very large. This suggests that there is something special about writing a pre-analysis plan before the data is even collected, rather than before someone opens up Stata.

Tags: ,

Ropey use of picture editor notwithstanding…

Here at Aid Thoughts, we tend to be on the cynical side of of the ‘blind optimism/dead-eyed cynicism’ divide that cleaves the development world (and there was a time when my professional responsibilities would not have prevented me from aiming a sharp, angry jab at certain famous development economists at this point).  One thing, though, that we have never disputed the value of, is having good, clever, questioning but ultimately hopeful people working day and night at unpicking all the difficult questions that we have to deal with on a daily basis.

So, it’s a pretty good thing that those four adjectives would be the first ones I choose when asked to describe Matt (well, okay, they’d come after ‘geek’ and ‘awesome’, but then they’re normally taken as read with him). So given that he’s got just 8 days before he submits his dphil and embarks on what will no doubt be a brilliant career, I’ve returned for one more Aid Thoughts post with a simple message for a change:

Matt, kick ass in your dphil and get a real job, you loafer. More people like you are needed.

Tags: , , ,

Get your assumptions out in the open

Oxfam has just released a new report calling for a moratorium on land deals in developing countries. Well worth a look – this should be subject to a robust debate. Lacking the time for more substantive analysis* (see end), let me turn to comparative advantage and look a little closer at one their killer facts:

Indeed Oxfam’s calculations suggest that the land acquired between 2000 and 2010 has the potential to feed a billion people, equivalent to the number of people who currently go to bed hungry each night.

Sounds like a lot of people. The defence of this calculation is tucked away in the footnotes:

The country and area of individual land deals for the purposes of agriculture, forestry, and livestock, covering a total of 40.3 million ha, were obtained from http://landportal.info/landmatrix/get-the-detail/database.csv (downloaded 25/07/12). The potential annual cereal production on acquired land was then calculated for each country by summing the product of the area of each deal and the average national cereal yield (data source: http://faostat3.fao.org (downloaded 25/07/12)). The food energy available from the potential cereal harvest on acquired land in each country was calculated by multiplying the potential production volume by the kcal available from one tonne of cereal in the given country (obtained by dividing the annual food energy supply by the annual food supply quantity, in both cases for cereals excluding beer (data source: Ibid.)). The number of people that could potentially be fed from acquired land in each country was then calculated by dividing the potential annual supply of food energy by the product of 365 days and 1,800 kcal (the FAO’s global minimum daily energy requirement per capita). National totals were then summed to arrive at a global total. On the assumption that the vast majority of the land acquired in the past ten years could be used to grow food, whether or not investors intend to use it in that way, and that the publicly available data is a reasonably representative sample of the total database, a conservative estimation was made that if about 40 million ha could feed about 240 million people, then 203 million ha is likely to have the potential to feed more than 1 billion people

So there’s an assumption here that the land being sold could, if left alone, supply food with the same yield as current production in the country it was purchased. I think this depends on assumptions over the availability of labour to farm this land if it were left free and the marginal product of labour (i.e. would we expect average yields to increase if people left some farms to take up the free land). Oxfam seems to be relying on a pretty optimistic opportunity cost to these land deals – that the counter-factual is a world in which this land sprouts average yields instantly and international food distribution gets the food to the right people who need it.

Of course Oxfam’s calculations may not be considering the counter-factual of no land deals – instead they might just be pointing out the value of the land, in terms of feeding potential, is quite high. This notion is reinforced by the fact that a large chunk of the land deals are for the purpose of growing food to feed people in other countries. A debate about the impacts of these land deals should also consider the impacts on the hungry in investing countries, as they are part of this one billion figure. Anyway, I’d like to hear your comments about the underlying assumptions.

Klaus Deininger, Rabah Arezki and Harris Selod have a recent paper showing that land deals more likely to happen in countries with poor land governance and tenure security. While the Oxfam report does point this out, the violation of the rights and lack of compensation paid to displaced landholders might be a more solid starting point than numbers generated primarily to alarm people. Then again, I suppose that’s the goal of these reports.

*Apologies for the lack of posting – I’m currently in the last couple of months before submitting my PhD, so expect I’ll be able to return full-force by the end of the year.

Randomised monkey trials

“What, you humans never tried calorie restriction?”

Following a series of animal studies showing benefits to health, cognition and longevity, the practice of calorie restriction has been getting a lot of attention over the past few years. While the mechanisms were never well-understood, limiting caloric intake by around 30% was seen as a shortcut to a longer lifespan. I’ve always been a bit skeptical and hesitant to embrace the calorie restriction camp, in part because a life in which my hummus intake falls by a third is not a life I would find worth living, but also because the most pertinant results have been based on a single randomised controlled trial of rhesus monkeys, conducted by the University of Wisconsin.

Those results were challenged recently by a similar trial conducted with rhesus monkeys by the National Institute for Aging, which found that treatment monkeys were healthier, but didn’t actually live any longer than those without the restricted diet.

Why am I writing about monkey trials on a development blog? The results of the two trials offer some important lessons for interpreting and relying on RCTs, which are quickly becoming the standard method of identifying development impacts.

The first thing to take away is that while RCTs can allow us to accurately identify treatment effects, we need to carefully consider what treatment we are measuring. Ideally, a control group should look identical to how the treatment group would have looked if they hadn’t been treated. In the University of Wisconsin study, while the treatment group was subject to calorie restriction, monkeys in the control group were allowed to eat as much as they wanted. While we’d really like to know the “impact of restriction above and beyond a normal diet,” the treatment effect measured in the Wisconsin study was something closer to “the impact of restriction above and beyond an all-you-can eat buffet at the Golden Corral.” It is hardly surprising then that the treatment group fared better and a good reason to be suspicious of the results. We cannot always be certain that a study has no effect on the control group – imagine a job-training programme which allows treated individuals to access jobs at the expense of untreated individuals – and so we should pay extra careful attention to what happens to controls groups, not just those who receive the treatment.

The second thing to note is that restricting your analysis to a particular subgroup of individuals or a limited set of outcomes can be tricky. The University of Wisconsin study limited its measure of mortality to “age-related deaths.” According to the New York Times, there was no difference in total mortality between the two groups, meaning that the reduction in age-related mortality (if it is to be believed) might have been offset by an increase in other types of mortality. Be wary of studies which subdivide outcomes like this without reporting aggregates, as rises in one indicator of success might easily be offset by another. In general, be skeptical when results don’t hold for the aggregate, but do for some magically-defined subgroup.

Finally, we need to keep going back to check up on our treatment and control groups for as long as possible. Both these monkey trials went on for over two decades, and while positive health results have been apparent for quite some time, it is only recently that the mortality rates have been high enough to detect (or reject) a difference. It’s possible that we’re measuring a lot of positive impacts today which aren’t going to amount to much in the long run.

Now, if you’ll excuse me, I’m going to break for elevenses.

Economics 101 and the wrath of Oxfam

“If only you knew the power of the dark side! In my day it was discussed in section 4.2 of the textbook.”

In the morning, before I get up and have breakfast or exercise, I like to stay in bed a little longer and read over the morning’s news and latest blog posts. This puts me at great risk, because it increases the chance that I’ll come across something that will really annoy me and thus spoil my pleasant morning. This is indeed a first world problem, but nevertheless one which has led to this blog post.

Over at From Poverty to Power, Duncan has linked to this blog post by Oxfamer Kate Raworth, who claims that economics textbooks lack the sophistication and tools necessary to deal with the issues that come about from human interaction with the environmental space. She asserts that this diagram represents the current way of economic thinking:

She then points out that this simplistic diagram (which I have yet to come across in a textbook – readers could you help me out? turns out it is from the Wikipedia page for circular flow of income, thanks @brettkeller) ignores three main things:

  1. Environmental degradation (use of natural resources, pollution, climate change).
  2. Unmeasured/non-monetary sectors of the economy (mainly work at home)
  3. Inequality

Let’s deal with her first point, shall we? Raworth claims:

First, the economy does not float freely against a white background. It is embedded within the planet’s environment, drawing on its natural resources and dumping pollutants back out into it. Mention that and an economist will say – ah yes, environmental externalities, we’ll come to those later. But calling nature’s resources ‘externalities’ and leaving them till later has led us to this crisis of climate change. How can it make sense to treat the fundamental resource on which all life depends as a factor external to the system?

First: “externality” doesn’t mean “external to the system,” it indicates a positive or negative impact which a given agent (person, firm, etc) doesn’t normally factor into their decision making. Economists label externalities as such because we think they are important, not because we are trying to shunt them under the rug.

Secondly, climate change hasn’t happened because people picked up Econ 101, didn’t see “the environment” mentioned as anything but an externality (or – admittedly – at all) and decided that it wasn’t important. Climate changed has happened because agents (citizens and firms and governments) don’t internalize the environmental impact of their actions. Not enough people think carefully about their energy usage. Not enough firms are given incentives to consider the environmental impact of their output. Economics has quite a lot to say about fixing externalities for a very, very long time (for example, the idea of Pigovian taxes has been around for more than 90 years), and to claim that climate chance is somehow the result of our way of thinking is a little silly.

Raworth also laments the lack of concern for the “unpaid care economy,” worrying that it will lead us to misunderstand the working lives of many of the world’s women. It is true that this is often left out of the discussion of the economy, but mainly because it’s pretty difficult to measure, not because no one cares about it. It’s also true that Econ 101 doesn’t have a lot to say about household chores or child care, but economics as a whole certainly does – Ms. Raworth would do well to have a look around the JEL classifications codes, especially in D and J. Her assault completely ignores decades of research on the household, the family and interactions between husbands and wives.

Similar things could be said about inequality – yes, Econ 101 is mum about it, often retreating to pareto efficiency without making any normative statements about initial allocations. In fact, much of Raworth’s objection seems to be that economics as a discipline isn’t normative enough, but I see this as a strength. It is up to us as a society, not economists, to determine how much we care about inequality. Yet, public policy requires a complex calculus of weighing different things that we care about; what economics and other social sciences try to do is make that calculus easier, by coming up with many different ways to measure income inequality and poverty, the impact of both on myriad indicators of well-being, including economic growth, health, (increasingly) the environment, and so on.

There is a missed opportunity here: I completely understand Raworth’s concerns and agree with her that we should be urgently focused on internalising the environment and inequality, and do more to measure what is currently unmeasured. Yet these concerns don’t validate her argument that economics is fundamentally lacking. Quite the contrary – we have been talking about externalities for a long time. We’ve been talking about inequality for a long time. We’ve been talking about the household economy for a long time. We have quite a large toolbox – some of the tools are better than others, and some of them are quite flawed, but perhaps a discussion as to which of them are more useful and which need refinement would be more fruitful than a casual dismissal of the way economics is taught.

Should more of these thing be making it into Econ 101? Some of them – like externalities – certainly do. It was one of the first things I learned in my undergrad economics courses. Then most of my lecturers -wrongly – dismissed externalities as something that would be dealt with by the Coase theorem. This is because many of them were libertarians who hated the idea of government intervention, not because economics told them to believe it to be true. Similarly, Ms. Raworth and I share very similar educational backgrounds: we both took the MSc in Economics for Development at Oxford and we both ended up as ODI Fellows in Africa. That our views diverge has less to do with the type of economics we were taught and more to do with who we are. This shouldn’t stop useful discourse – Ranil was taught quite a different style of economics during his masters, yet we managed to write a blog together.

I don’t believe I have the “wrong model of the world” stuck in the back of my head. Perhaps my tool set is weird, imperfect, and overly-mathematical, but that doesn’t stop me from caring about inequality and climate change.

Addition: this post could also be titled “In which Matt defends microeconomics, but has little energy for macro”

Tags: ,

If you’re not with us, you’re against us

"Only a Sachs deals in absolutes"

This post could also be titled “Taking credit, part deux.” Writing in the Guardian, Jeffrey Sachs considers the impressive reduction in child mortality rates across sub-Saharan Africa.

The critics of foreign aid are wrong. A growing flood of data shows that death rates in many poor countries are falling sharply, and that aid-supported programmes for healthcare delivery have played a key role. Aid works; it saves lives.

For the rest of us who are still burdened with the ability to question, this narrative seems a little too convenient. While the last decade was characterized by a massive increase in health aid to African countries, many of these countries also experienced significant economic growth and improvements in governance and safety. As Charles Kenny pointed out in his book many of these gains in survival may be technological, a result of to interventions which were made readily available. Of course, some of this was due to aid – but the resulting relationship is much more complex than “aid goes up, infant mortality goes down.”

All of this is not to suggest that health aid did not play a role – it almost certainly did – but waving one’s hand and giving all the credit to aid is a dangerous simplification. It also ignores a significant amount of heterogeneity – some countries did better than others, so we really need to start asking ourselves “why?” before we start patting ourselves on the back.

Yet, it isn’t the simplistic narrative that bothers me, it is what comes after: a declaration that aid skeptics are not only completely wrong, but that they could be responsible for the death of children:

Unfortunately, at every step during the past decade – and still today – a chorus of aid sceptics has argued against the needed help. They have repeatedly claimed that aid does not work; that the funds will simply be wasted; that anti-malaria bed nets cannot be given to the poor, since the poor won’t use them; that the poor will not take anti-Aids medicines properly; and so on and so forth. Their attacks have been relentless (I’ve faced my share).

The opponents of aid are not merely wrong. Their vocal antagonism still threatens the funding that is needed to get the job done, to cut child and maternal deaths by enough to meet the MDGs by 2015 in the poorest countries, and to continue after that to ensure that all people everywhere finally have access to basic health services.

Emphasis is mine. While Sachs is probably referring to pundits on the other extreme of the distribution, his rhetoric leaves no room for shades of grey; writing what I just wrote doesn’t make me a cautious optimist, it makes me an aid sceptic.

Then he tries to quietly paint aid sceptics as responsible for the deaths of children. Astonishingly, if you read the sentence in bold carefully sentence carefully, it’s clear that Sachs is putting much more weight on reductions in child and maternal death before 2015 than after. Does Mr. Sachs not care about children of the future? That interpretation might seem a bit unfair to you. What a shame.

Tags: , ,

Sachs the rainmaker

"But kemosabe, this would not stand up to a diff-in-diff"

Many of you will already be familiar with the ongoing debate over the efficacy and evaluation of the Millennium Village Project, the brainchild of the Earth Institute’s Jeffrey Sachs. Due primarily to the work of Michael Clemens at the CGD and Gabriel Demombynes at the World Bank, the MVP’s claims of development impact have finally faced substantial scrutiny, although frequently the debate has felt more like a war of attrition than productive discourse.

Enter the Lancet, a reputable medical journal which has a worrying tendency to publish really disreputable social science research, which just published a study by Sachs et al. showing that, over three years, child mortality (under the age of five) has fallen by roughly 25% across nine Millennium Villages. When compared with `control’ villages (which were chosen later and differ from the MVs in many, substantial ways), the drop was even larger – close to 31%.

Suddenly the bells starting ringing: after all the doubt, the MVP is hailed as being successful in reducing child mortality, with the editor-in-chief of the Lancet rallying behind the paper and the Guardian reporting the results with an astonishing lack of scrutiny. Only in the twitterverse/blogosphere has the response been largely negative (Lee Crawfurd disassembles the results of the Lancet article here).

However undeserved, this might have been a good opportunity for the the Earth Institute to bask in its momentary glory. Yet, the results might have already been undermined by awful timing: the Lancet study arrived just days after another study by the World Bank’s Gabriel Demombynes and Karina Trommlerová showing absolutely massive decreases in child mortality across most of sub-Saharan Africa in the past few years.

To understand why this is a problem for the Lancet study, consider the table below, which I’ve assembled from results from that study and some figures from the World Bank one (admittedly swiped from Michael Clemens’s post on it).

From the WB study I’ve taken the same nine countries used in the Lancet article, listed their declines in mortality and (assuming a linear trend) calculated the average decline in under-5 mortality per year. One caveat: the years considered in the World Bank study do not necessarily coincide with the timing of the Millennium Villages in their respective countries, so we may be comparing trends from different periods. Even so – these figures still provide a rough idea of the relative magnitude of the mortality decline.

Per-country figures are not available in the Sachs et al. study (which is it a bit worrying in itself), so I can only compare the average declines in these countries to the average decline in all Millennium Villages. What do the results suggest? While child mortality dropped by 24.6 (less children dying per thousand births) over a 3 year period, average declines for all countries in the study are broadly similar: 22.5.

The first and most important thing to take from these results is that the Millennium Villages aren’t vastly outperforming aggregate gains in the same countries. This makes it very difficult for the MVP to claim it is making an impact – it’s a bit like claiming credit for rain in Oxford, when it has been raining all over the UK.

The second thing worth noting: if you look at the above table, taken from the Lancet study, you’ll see that under-five mortality is actually increasing in the control villages. This strongly suggests that control villages are quite different from the rest of the country at large. The Earth Institute has argued that Millennium Villages (and their control counterparts) were selected because they were different – but even if these odd trends in the control villages don’t disqualify them as a counterfactual (which I still think they do), the differences seen here certainly prevent the MVP from having any sort of claims of external validity.

The argument that the Millennium Villages aren’t outperforming the rest of their host countries is not new: Clemens and Demombynes made it over a year ago, when they found that many other claims of `impact’ by the MVP were reflected in national statistics.  Let’s hope the hype from the this study is similarly deflated.

Tags: ,

The temptation of the empirical knockout punch

Admit it, you love watching popular development preconceptions being destroyed by cold, hard empirical reality just as much as I do. Despite the slightly queasy feeling I got knowing that Nicholas Negroponte was still out there wasting people’s time and money, these feelings were recently swept away by the satisfaction of knowing that the One-Laptop-Per-Child program was, for the umpteenth time, proven to be ineffective by a rigorous RCT.

These knockouts are especially welcome when a program’s hype far outstretches its evidence base. Such was the case with the Global Alliance for Clean Cookstoves once Hilary Clinton endorsed it, much to the ire of the developmentistas who pointed out that there was nothing new or particularly encouraging about the use of cleaner stoves. This didn’t stop Madeleine Bunting and Julia Roberts (yes, Julia Roberts) from claiming clean cookstoves would work wonders and save millions of lives.

Finally, some more rigorous evidence arrived this month, with the knockout delivered by a group of MIT researchers – including the prolific Esther Duflo – who released a new study basically showing cookstoves had little long term impact. Charles Kenny, who resists the temptation to declare a K.O, offers a good summary of the results:

So the results of the MIT study will come as a disappointment to the clean cookstove movement: 2,600 households in India were sold simple improved cookstoves at a highly subsidized price –they cost $12.50 to put in but families paid just 75 cents.  Yet after three years, hardly any of the stoves were being used, and most had fallen into disrepair.  The stoves ended up no more efficient than traditional models –they burned as much wood– and levels of indoor air pollution were not improved.

Disheartening results, to be sure.  But they shouldn’t come as a surprise.  There are piles of previous evaluations of cookstove programs that may have been less rigorous but still pointed in the same direction.  In fact, seventeen years ago, researchers at the Indian Institute of Technology published a review article noting that “in spite of quite ambitious programmes” in support of renewable energy technologies for cooking, they had “not met the expectations of the planners and implementing organisations.”  Amongst the reasons that improved cookstoves in particular were proving a disappointment, the researchers pointed to findings which suggested the stoves did not in fact save fuel, and they were hard to use and maintain (sound familiar?).

So this is an open and shut case, right? Well, not quite. The MIT paper, the Washington Post article which covered it and Kenny all seemed to have missed something: a different RCT on improved cooking stoves which was released just last month. That paper, by Gunther  Bensch and Jörg Peters, studies the impact of a randomised lottery of stoves in rural Senegal. The results suggest that, a year later, households receiving an improved cooking stove used less wood, spent less time cooking meals, reported better indoor air quality and (for women, who presumably did all the cooking) were significantly less likely to have respiratory disease symptoms, eye problems. Nearly all recipients of a stove used it at least seven times a week, in sharp contrast to the lack of use seen in the MIT paper.

Make no mistake: Duflo, Hanna and Greenstone’s study has many advantages over the Bensch/Peters paper. The India paper benefits from a much larger sample size, repeated follow-ups and much more sophisticated measurement techniques. Yet the Senegal paper is still worthwhile because it is – well – written about Senegal and not India. It is perfectly possible for an intervention to fail in one setting but work in another. The J-Pal study strongly suggests that we need to visit treated households more than a year later, as it is possible that the families in the Senegal sample might still stop using the stoves in the future. However, the timing of the latter study provides an excellent opportunity: the intervention was carried out in November, 2009, so if a follow-up survey was conducted this November at the three year mark, we’d be able to identify a long run impact which could either reinforce or undermine the MIT researchers’ result.

Sadly, I doubt anyone will take advantage of this opportunity. The incentives for replication in academia are still incredibly weak, and compelling studies which knocks down popular ideas can be just as persistent as those with novel, positive result. Even if Bensch and Peters return in a year with compelling evidence that cookstoves do have long term impacts in Senegal, it won’t have the quite same impact that the Duflo paper did. We should be a bit more cautious about embracing papers which confirm our priors - a knockout is sometimes just too good to be true.

Tags: ,

Why predictions fail

If *only* we had included institutions in our prediction model

Over at the Why Nations Fail blog, Daron Acemoglu and James Robinson’s discuss a set of growth predictions made by Paul Rosenstein-Rodan, the father of  the Big Push model, illustrating just how wrong they were:

Acemoglu and Robinson argue that the these predictions were off primarily because the Big Push model ignored politics and institutions:

Of course, things didn’t quite work out that way. In fact, many of the economies about which Rosenstein-Rodan was bullish are not much richer today than they were in 1961. Liberia and Haiti’s economies contracted since then. Angola, Kenya, Nigeria and Uganda haven’t done so well either. We of course know that Afghanistan, India and Pakistan grew more slowly than South Korea, Taiwan, Thailand and Singapore. Argentina and Haiti were no match for Costa Rica, the Dominican Republic and Panama.

The main reason why Rosenstein-Rodan got it so wrong is because he completely ignored the role of institutions and politics.

It’s hard to disagree that Rosenstein-Rodan should have taken these into account – but are they the primary drivers? What about geography, natural resources, export commodity prices, health and the myriad other factors which might drive a country’s growth rate? Without a little more effort, the models lack of effectiveness doesn’t tell us anything about why it is ineffective. I understand that Acemoglu and Robinson consider institutions to be the chief determinant of everything since the beginning of time, but arguing that the Rosenstein-Rodan prediction is wrong because it ignored institutions is a little like arguing that a car missing all four wheels won’t drive because – damn it – it’s also missing four tires.

Slightly more disconcerting: A&J are only displaying a subset of predictions from Rodan’s original paper. Why? My guess is that eye-balling the full dataset doesn’t reveal as much. This calls our for a slightly more rigorous approach than pointing to a few bad predictions. Even better, does someone have the time to crunch the numbers and see if Rodan’s predictions are less useful than predictions being made today?

Tags: ,

Kenya, exogenous governance shocks edition

From the BBC:

The Kenyan president said Tullow would drill more wells to establish the commercial viability of the oil.

“It is… the beginning of a long journey to make our country an oil producer, which typically takes in excess of three years. We shall be giving the nation more information as the oil exploration process continues,” he said

Cha-ching. Here we go again.

Tags: ,