The external validity double standard

David McKenzie goes to town on those that complain about the lack of external validity in experimental methods. For one, the standard seems to be applied more often to research in developing countries:

So let’s look at the April 2011 AER. It contains among other papers (i) a lab experiment in which University of Bonn students were asked to count the number of zeros in tables that consisted of 150 randomly ordered zeros and ones; (ii) a paper on contracts as reference points using students of the University of Zurich or the Swiss Federal Institute of Technology Zurich; (iii) an eye-tracking experiment to see consumer choices done with 41 Caltech undergraduates; and (iv) a paper in which 62 students from an unnamed university were presented prospects for three sources of uncertainty with unknown probabilities; (v) a paper on backward induction among world class chess players.

And then, a swipe against those withing-development who argue that experimental methods aren’t externally valid:

Consider some of the most cited and well-known non-experimental empirical development research papers: Robert Townsend’s Econometrica paper on risk and insurance in India has over 1200 cites in Google Scholar, and is based on 120 households in 3 villages in rural India; Mark Rozenzweig and Oded Stark’s JPE paper on migration and marriage is based on the same Indian ICRISAT sample; Tim Conley and Chris Udry’s AER paper on technology adoption and pineapples is based on 180 households from 3 villages in southern Ghana; on a somewhat larger scale, Shankar Subramanian and Angus Deaton’s work on the demand for calories comes from 5630 households from one state in India in 1983.

From the perspective of a researcher (and one currently working on an experiment in a developing country), I completely agree with McKenzie here. Micro-empirical evidence is always useful, whether or not it is immediately generalizable or not – as long as we update our priors with care every time we read a new study.

From the perspective of a blogger who has taken swipes at the randomistas over external validity a few times, I think much of the push back on the external validity front has less to do with the research itself, and more with how the research is being trumpeted outside the academic sphere – there haven’t been any NYT articles about how eye-tracking experiments herald the end of poverty.

2 thoughts on “The external validity double standard

  1. Tim Ogden

    May 2, 2011 at 5:29pm

    With no snark intended, where are the links to those NYTs stories heralding the results of a micro-experiment as the end of poverty?

    I recognize, and appreciate, the need for caution in interpreting the results of experiments. I am personally prone to the intellectual sin of changing some of my priors too quickly. But in all of my experience with the randomistas, including David McKenzie, they have been the most cautious in limiting the interpretation of their results.

    In fact, I am befuddled by the dual critique that seems to be going on. On the one hand, there is the critique that the conclusion of every experiment is, “more experiments are necessary” and on the other hand that the studies lack external validity. Both cannot be truly valid in a meaningful way.

  2. Matt

    May 2, 2011 at 5:43pm

    Hi Tim,

    No snark taken – you’re absolutely right to call me out on it. It’s just hyperbole and snark on my part over what I feel is extremely optimistic coverage of the randomista approach by the press in general, which most social science research is treated a little more cautiously.

    And I agree that McKenzie is extremely cautious – in fact, most researchers are – I’ve said before that it isn’t the academics that typically go running with the results, it’s the journalists or the communications/PR people at the places the research is developed.

    And, for you last point… yes they can both be true in a meaningful way – not only do more experiments given us more context-sensitive information (we’re probably more comfortable making recommendations on policy in the places the experiments were performed) and also allow us to update our overall beliefs about our parameters of interest across contexts.

    I should point out again that I’m not saying there’s a problem with the research itself (I completely agree with McKenzie about the injustices within the journals).

Comments are closed.