To Find Certainty upon the Dreaming Air

David Roodman over at the CGD recently posted a great article about the new aid-growth paper published by WIDER, of which he remains a skeptic. Scroll down to the comments, and you’ll see Owen Barder making some very good points in response.

I also posted a comment on the piece. My concern was rather different and probably not best suited for that blog. Among various ramblings, I wrote:

At what point did finding universal answers become the only legitimate basis of economic study? Development has looked and proceeded differently in different places…

Statistics is just one type of evidence. Why have economists apparently forgotten this?

My plea was for an approach to development that took in far more specific case study analysis. By this I mean not the Duflo/Banerjee approach of randomized trials, but a holistic approach to development that focuses more on the actual historical process of development in specific places over real time than abstractions with pretense to universality based on cross section data or studies dealing with specific interventions.

The rationale behind this is simple to me: firstly, a disinterested analysis of what we know about successful development processes emphasises their diversity more than their similarities, though these exist and are important. There is little reason to assume that imagined future development processes will have more uniformity. Secondly, understanding of real development successes and real development failures (however they are defined) demonstrate that they are typically the result of a range of complex interacting factors. In most cases, causal mechanisms have shown inconstancy, with the same phenomenon having markedly different effects depending on context and time, even within the same country.

Of course, this kind of method is messy; it won’t ever give us unambiguous answers as to what works and to what extent. It requires that we formulate policy based on our understanding of the historical circumstances that led to the current state in a country, bearing in mind that the interplay between factors cannot always be modeled as there are concurrent effects on multiple levels (individual interactions, social interactions between groups and state-subject/citizen relations for example) and different effects in different regions or time periods. It will mean that we have to rely on what we know from other countries and other times, trying to tease out the central relevant lessons.

It’s an approach that is anathema to modern economics. The majority of our current work tends towards universality of analysis and conclusion. It seeks to pose theoretical relationships that hold under specific assumptions (which are often implicitly further assumed to hold everywhere if they hold anywhere) and then test them. If the tests work (and for the cynical, even if they don’t), they seek to tell us of a ‘robust’ concrete relationship: each unit of factor X contributes 0.1 units of GDP growth. Paradoxically, this often leads to just as much messiness and uncertainty as a historical analysis. The aid regressions Roodman looks at are a classic example: the exact same data produces opposite results depending on model specification. Neither result is unambiguous even on its own terms.

Economics was not always like this though. Early economists were multidisciplinary creatures by nature. They studied history, social relationships and the economy as interlinked phenomena, using a holistic method that took in historical evidence, theoretical abstractions from this evidence and some further statistical evidence to support their ideas. Statistics was necessarily a smaller part of their work, for their data and the sophistry of their statistical techniques was still quite basic. This period still produced what to my mind remain the two greatest works of economic thought: The Wealth of Nations and Capital.

Since this time, economics has undergone two major structural changes. One is undoubtedly positive: we have developed far better statistical modeling techniques and begun collecting more reliable (though often still flawed) data on economic phenomena. My problem is emphatically not with greater use of statistics.

The second phenomenon is where my concerns arise. Economics has increasingly adopted a theoretical approach called Methological Individualism (MI). MI is an approach that has been tried and discarded in most disciplines, one that builds models on the basis of ‘representative’ individuals and relations. For some reason, however, it has stuck in economics, retained some support in sociology and perhaps in political science. Unlike in sociology and political science, it is now the only acceptable approach for mainstream economics – to the extent that few economics students have even heard the term, for they have never questioned its status as the basis of economics.

This is so much the case that very few economics students actually read Adam Smith or Marx in their original writings; nor even Keynes or Schumpeter. They are not ‘rigorous’ (read: based on individualist mathematical models) enough to be studied except insofar as they have been adapted by modern economists. When studying economics myself I found to my horror that some of the self-professed ‘New Keynesians’ who were taking the same course as I had never actually read The General Theory, though they all claimed to know what he had argued (of course, I am not accusing real New Keynesians like Greg Mankiw of this).

In the period of MI’s dominance, economics has developed its two biggest contemporary flaws: universality and reductionism. They are closely related phenomena. Reductionism is the more direct effect of MI. Through MI, economists seek to model representative relationships; the problem with this is that models quickly become unmanageable when we try and capture the multiple levels on which a relationship works. Individuals react to specific phenomena on an individual level, as part of a group (or several groups) and as subject of a state which has coercive power over them. These reactions may pull in different directions. In some cases, individuals identify with multiple groups that contribute to contradictory responses to a single stimulus.

Of course, there is value to theoretically and analytically ‘cleaning up’ these relationships. It’s crucial, in fact, if we are to try and understand anything. All disciplines of study do this to different extents. The problem is that economics has gone to extremes through its modeling approach and assumes relationships work at their simplest level for most modeling. Even this can have significant benefits, provided we engage with other disciplines and learn from the complexity therein.

However, the insularity of economics as a profession has led to a cessation of such interaction. Instead we seek to apply the same reductionism to other disciplines. A great example is given in this Aid Watch post: It’s entitled History Matters, but the history it presents is remarkably reductionist, as several commenters have pointed out. It is not history, but ‘path dependency’ a different concept that is also based on analysis of past events. History is very rarely about linear relationships. Far more often it is about joint causality, confluence of circumstances, chance and unforeseen events.

Universality stems from reductionism. Because we try and strip away too much of the complexity from our world view, we are left with individual effects that are assumed to hold in all contexts. Econometrics is the statistical tool we use to look for these relationships, and it has many values, done well and with sensitive interpretation. Unfortunately, these two caveats often don’t hold, and the results of a regression are used to advocate for policy directly. Good econometrics can account for context, but sometimes this leads to very complex models and I would argue that our capacity for defining complex relationships is vastly overestimated by many econometricians. Furthermore, data for developing countries isn’t often strong enough to support these models.

Economics needs to restore balance to our consideration of development. Statistics, simplification and modeling can be valuable; but so too can be context, true historical analysis and complexity of causality. Matt is a far better econometrician than I, and we have often had this discussion in the past. I think both of our opinions have been revised on the back of these arguments. Our most recent such exchange was via e-mail. I can’t improve on his final analysis, so I’ll leave the last word to him:

Fruitful research has to incorporate both some rigorous statistics and case studies. I think the issue is that case studies alone have a tendency to conflate different effects. If we look at places where we think aid has a positive effect, we’re already selecting on just that: we’re likely to find positive answers when it conforms to our prior. Sometimes, but not always, you really do have to compare subjects using a large sample to tease out indirect effects. I’ve seen a world based on case-studies, and it’s an equally unsatisfactory world as a one based on pure econometrics.

(For the record, I’m probably more of an economist than a historian. I just find that writing with a more historical perspective adds greater value to development debates than being another economist among many.)

9 thoughts on “To Find Certainty upon the Dreaming Air

  1. Ash

    November 11, 2009 at 10:56am

    All this is very true, and perhaps the future of development economics lies more in a complexity theory direction than the rather pointless search for the perfect model that can explain everything, something like this interesting new blog (h/t Duncan Green):

    There is obviously a role for judgement in the balance between specious universality and useless specificity, and this is something that somehow most people seem to accept individually but that the academic establishment does not seem to produce.

    I think you got the point of the Aid Watch article wrong though, I think it was more to point out that overly simplistic modelling approaches can miss out important elements in a holistic understanding of institutions and cultures. This is an argument in favour of the more nuanced approach that you are advocating, I would say.

  2. Ranil Dissanayake

    November 11, 2009 at 11:21am

    Ash – I realise that this is the point of the Aid Watch post. My criticism is not of the intent ( nor of the blog: I like Easterly as a scholar, and think Aid Watch, both Freschi and Easterly, is a great blog).

    The point is that the representation of the research Laura Freschi put up (I can’t judge the original research as it’s not linked) is an example of how people using economistic techniques tend to reduce complexity to simple linear statements. If you read the comments, particularly the one from Chinaza (sp?) right below mine, you’ll see plenty of factors that have worked in tandem with early taxation structures to explain local differentiation in accountability and quality of Governance. Reducing the relationship to ‘early-stage taxation leads to better governance’ isn’t helpful, because the reality is that various other factors facilitated, worked with and worked against that relationship.

    Hans and Ash – thanks for the links. I read the Rodrik paper a while back, and I’ve just discovered the same blog from the same source. I don’t know enough about complexity theory to comment, but I’m eager to learn more.

    As for the micro/macro divide, the point I’m making is not about types of economics but approaches. Reductionism is a characteristic of a lot of work in both fields. This is not necessarily a bad thing. As the quote from Matt above implies, simplifying things through statistical analysis can help us understand ‘hidden’ or opaque relationships. We just need to supplement it with a more complex analysis.

  3. Matt

    November 11, 2009 at 11:41am

    I think if you look at the actual discussions going on in academia (at least in development econ), especially on the micro-side, you’ll find that many assumptions are anything but universalist. Most micro studies are similar to case-studies: context specific, and the authors will usually admit as much – that their study is just adding to a body of evidence.

    One has to find a balance in these things – the opposite of universalism is the insistence that nothing in one context can ever be applied in another context. The IMF used a very basic set of models on how the economy works (which, I must add, are also based on reductionist principles!) to reduce inflation in Malawi. There were of course some context-specific caveats. But if someone, for example. finds that a given micro relationship holds in Tanzania, Mozambique and Malawi. I wouldn’t condemn them for suggesting it might also hold in Zambia.

    The problem comes when overzealous people make universal claims based on shoddy cross-country data (i.e. only X matters for development). While some big names do this, most of us don’t.

    As far as reductionism goes – I think your assertion is mostly true for purely theoretical economics, but I think there’s nothing wrong with asserting that there are some basic, fundamental concepts that, more often than not, tend to hold in some fashion. Most of empirical economics is dedicated to finding out 1. Whether they hold, but more importantly 2. Why they do or don’t hold. I think most in academia find the second question much more interesting, and recognised that context matters. We have a multitude of models for fertility, but there’s not a single population economist out there who things that any given model is universal.

    I should note that, when Ragnarök comes, as continents crack under the weight of a great battle between Odin and Thor, Ranil and I will be there, still having this argument.

  4. Ranil Dissanayake

    November 11, 2009 at 12:03pm

    Matt – I love how your name links back to aid thoughts, on an aid thoughts blog!

    All of these points are good. You’re right about many micro studies (and you’re closer to the debates than I am); similarly that empirical economics looks a lot at testing theories and explaining their results. My point is that while this is a valuable approach, it should be supplemented far more with broader historical (and anthropological) evidence and analysis, focusing on holistic understanding more than specific relationships. I might be reading the wrong stuff (and freely admit to being more of a macroeconomist than a micro man), but I don’t see this happening much. This is of course partly the fault of historians etc. who have generally gotten thoroughly sick of modern economics and just retreated from the field of battle altogether.

    Micro studies analysing how specific things happen are great, and if they are doing it right, that’s fantastic. That needs to happen in macro economics as well, because economic transformation is a macro phenomenon, and we need to understand the roles of each micro change in that macro picture.

    finally, re the IMF. that was a good example of economics being the right thing at the right time. good call.

    (btw – did you see Chris Blattman’s blog today? ‘randomize Marx’ – I’m so intrigued I’m barely coherent).

  5. Laura F

    November 11, 2009 at 4:23pm

    Hi Ranil,

    Thanks for a very interesting post. Lots of food for thought here.

    Just wanted to mention one thing that is puzzling me: I *did* link to the original work by Daniel Berger in my post, and will link to it again here.

    I thought it was quite clear that Berger’s intention was not to generalize his findings to the whole of Nigeria, nor to make the overly reductive claim that tax structure is the only requirement for good government service delivery. If I inadvertently implied either of these things in my short summary of the research, this is my fault and not Daniel Berger’s.

    I think that the comments on the Aid Watch blog in response to this particular post show a couple things, among them that people are a little overly eager to generalize policy prescriptions (i.e. aha! If this paper shows a small effect of taxation on governance in a very specific area in Nigeria, we should redirect all development funds towards implementing poll taxes!) At the same time, while you may agree or disagree with the methods or conclusions, I don’t think there’s anything wrong with the idea that this narrowly focused paper (which in fairness does rely on both statistical analysis and the historical record) could provide one small but valuable slice of evidence towards one or two of Matt’s “basic, fundamental concepts that, more often than not, tend to hold in some fashion,” i.e. that direct taxation can help build bureaucratic capacity, or that institutional norms can persist over time in surprising ways.

    Also, in criticizing the paper for failing to consider historical or current differences among populations in the north vs the south of Nigeria today, other commenters seemed to really miss the point that the paper looked *only* at the populations just above and just below the temporary administrative boundary line.

  6. Ranil Dissanayake

    November 12, 2009 at 5:51am

    Hi Laura,

    Sorry about that – I completely missed the link, it’s very obvious now I look at the blog again. Many apologies. I’ll read the paper with interest (when my very slow internet finally loads it up), and my opinions may completely change.

    I have to say, I agree with both of the points you make in your comment. It is true that many people over-generalise their conclusions (and I’m sure this is true of some of my own pet issues); and yes, it is true that a focused paper looking at specific issues can provide useful new evidence.

    I also agree strongly with one of the final points you make – direct taxation can be very important. The second is also true: institutional norms can persist over time; but equally there are examples of how they have suddenly been destroyed (or as suddenly as something as intangible as a ‘norm’ can be destroyed); and other examples of how they evolve into completely new things.

    My argument is not that the basic points of the study are invalid or somehow unimportant, but that what might tell us most about them is looking at them in the nest of the other historical processes that were working with or against them over the time studied (again, I still haven’t loaded up the paper, so I apologise to Daniel if he’s done this); while it might be possible to isolate the effects of direct taxation, it might also not be particularly fruitful if we also find that these effects depend heavily or entirely on interceding factors.

    I used the example of that piece to demonstrate a wider point, which is that I think in general economists tend too much towards isolation of specific issues (and this tends to lead to or benefit from reductionism). Matt’s comments, and yours, present the counterpoint, that this is valuable in its own right. Perhaps your piece wasn’t the best one for me to cite in support of my point – but it was current, and I was sure most bloggers would have read it, because Aid Watch is always worth reading.

    It’s a question of balance. Discussions like this are a good way of getting the right one.

    thanks for the comment.

  7. Michael

    November 21, 2009 at 2:20am

    My plea was for an approach to colonialism that took in far more specific case study analysis. By this I mean not the Duflo/Banerjee approach of randomized trials, but a holistic approach to colonialism that focuses more on the actual historical process of colonialism in specific places over real time than abstractions with pretense to universality based on cross section data or studies dealing with specific interventions.
    The rationale behind this is simple to me: firstly, a disinterested analysis of what we know about successful colonialism processes emphasizes their diversity more than their similarities, though these exist and are important. There is little reason to assume that imagined future colonialism processes will have more uniformity. Secondly, understanding of real colonialism successes and real colonialism failures (however they are defined) demonstrate that they are typically the result of a range of complex interacting factors. In most cases, causal mechanisms have shown inconstancy, with the same phenomenon having markedly different effects depending on context and time, even within the same country.

  8. Ranil Dissanayake

    November 23, 2009 at 8:34am

    Funnily enough if you replace the word development / colonialism with pretty much any other adverb that passage will still make sense.

    any actual argument as to why development aid is colonialism? Do you consider the US to be a colony of Japan, given how much in the way of loans Japan provided to the US in the 80s and 90s?

    [comment edited to ameliorate the tone a bit!]

Comments are closed.