On Hirschman, measurement and RCTs


Every intervention is unique, perhaps unintentionally so.

Recently, Dan Honig of Johns Hopkins forwarded Ranil and me some thoughts he had in reaction to an Albert Hirschman on development projects that he felt was pertinent to the discussion on the pros and cons of RCTs. What followed is a discussion (rant) between Dan, Ranil and me. I’ve edited out the e-maily bits for clarity:


Matt, just after I hit send on this I realized I should have included you on this – I generally think you’re right on RCTs and the stale-ness of the conversation (and discussed this with Ranil a few months back some 2 days after you had dinner with him, hence the cc to Ranil) but feel like I’ve never seen this Hirschmann frame and wondering if it struck you as interesting. And yes, basically I’m trying to catalyze you writing something cool on this so I can quote/reference it down the road

Reading Hirschmann’s Development Projects Observed for the first time, and as I read it he’s with [Lant Pritchett and Michael Woolcock] on RCTs and causal density in international development projects. The quote below is from page 186 of the 1967 edition; italics are his, brackets mine; just before this he suggests we may not be able to identify good indicators of effects ex-ante and thus presumably couldn’t be pre-specified in a trial, meaning presumably we would be ill served by an RCT on a particular intervention even if we ignored external validity concerns.

“The indirect effects [of development projects] are so varied as to escape detection by one or even several criteria uniformly applied to all projects. Upon inspection, each project turns out to represent a unique constellation of experiences and consequences, of direct and indirect effects.”



Hey Dan, that’s a really interesting quote by Hirschman. If my interpretation is correct, it seems to be more damning for empirical evaluation in general than for RCTs in particular.

I’m not sure how I feel about this. Even if you move away from a simple, reduced form causal framework, Hirschman’s critique seems like it would apply. Even if development is a messy, complex thing that can’t really be boiled down in an impact evaluation framework, we still rely on measurement when we talk about development, and any given set of measurements is going to leave out things which might matter which are unmeasured. We can point at improving test scores but leave out student stress, etc, and the set of things that we leave out that might be important will change depending on the context. I guess I see this as a problem of measurement rather than as a problem for RCTs.

I also wonder what this means for how an empirical researcher operates. Over the last few years, I have become incredibly suspicious of surprising, counter-intuitive results, where a researcher measures something outside of the standard set of outcomes and finds a result. In a world of multiple hypothesis tests, expanding the set of outcomes to include as much of Hirschman’s unique constellation as possible will open up the door to a lot of false positives which will end up getting written up and published.

So that was a rant. Um, what do you think Ranil?

Continue reading

Come work for me in London or Washington D.C.


Like development? Like statistics and coding? Come join me and Vij Ramachandran at CGD.

The Center for Global Development, an independent, non-partisan research organization in Washington, DC and London, UK seeks a Research Assistant (RA) to support the work of Matt Collin and Vijaya Ramachandran. The successful candidate will have experience with research in economics, public policy, political science or a related field and will be based either in London or in Washington DC.  Applicants must demonstrate strong quantitative, analytical and communication skills. The position is well-suited to those who are considering doctoral study in the future.



The Twelve Days of Christmas (Aid Edition)™

It’s an oldy but a goody, and the only thing I always  repost. Go on, sing along:

On the twelfth day of Christmas my donors gave to me

twelve delayed disbursements!

eleven sketchy studies

ten consultants calling

nine economists arguing

eight mission meetings

seven worthless workshops

six gender trainings

five RCTs!

four 4x4s

three acronyms

two empty schools

and a lecture on M&E!

Still on sale. And I promise this won’t be the only post I write for another six months.

Protected by randomness


From fusion.net:

The Random Darknet Shopper, an automated online shopping bot with a budget of $100 a week in Bitcoin, is programmed to do a very specific task: go to one particular marketplace on the Deep Web and make one random purchase a week with the provided allowance. The purchases have all been compiled for an art show in Zurich, Switzerland titled The Darknet: From Memes to Onionland, which runs through January 11.
The concept would be all gravy if not for one thing: the programmers came home one day to find a shipment of 10 ecstasy pills, followed by an apparently very legit falsified Hungarian passport– developments which have left some observers of the bot’s blog a little uneasy.

The title of the piece (Robots are starting to break the law and nobody knows what to do about it) elicits worries of AIs gone amok, but the basic conundrum of this piece and others about the Random Darknet Shopper is more complex: if I design an AI which takes a random, blind action in a space which is largely – but not uniformly – illicit, am I legally culpable?

Take this thought experiment: imagine going around your office with a ten dollar bill, offering to buy whatever your colleagues would be willing to sell to you at that price, but under the condition that you do not see the item until the transaction has taken place. If one of your colleagues slipped you some cocaine, who would be at fault? What if you chose to repeat the experiment in an area of town infamous for drug-deals, are you suddenly more culpable?

When I was young, I used to order what they called “Grab Bag” comic packs, where I would pay a set amount of money for an unknown, random assortment of comic books. If someone had slipped a pornographic comic into my grab bag, it’s hard to see how I would be at fault. But where I choose to make my blind transactions seems to augment how we perceive culpability.

Several years ago I wrote a piece about how randomness can complicate our standard notions of guilt. The intersection of randomness, culpability and the law sounds like an area that – if someone hasn’t written about it a lot already – is ripe for further work.

Branko Milanović and the Spam King of Development


A few weeks ago, I received a bizarre e-mail addressed indirectly to former World Bank economist and inequality/wealth guru Branko Milanovic. An excerpt:

Marry me, CUNY Prof. Branko Milanovic. Become part of our family of Expert Dreamers. Join the Serbia Strategic Team and help us design, create, imagine a Wonderful future for Serbia, for the World and for the Sollar System.

Yes I am positive. I want to become President of the Sollar System before 2050 with you in our team. Yes I am positive. I want to contribute my best efforts, to devote my lifetime to this wonderful challenge. And I want you, CUNY Prof. Branko Milanovic, I want you to come with me, to come with us.

Following this e-mail, a slew of my colleagues and I began to receive e-mails requesting that we receive chapters for a book known as “The Monfort Plan,” a sort of grand scheme to end global poverty by assembling a team of “Expert Dreamers” to essentially act as the world’s largest peer review body. A bit like The Avengers but without all the charisma and one-liners.

The source of these e-mails was a man named  Jaime Pozuelo-Monfort – or `JP Monfort’ as he likes to call him self. JP Monfort is a professional development spammer. As described on his bio for the Huffington Post (further proof that the only binding prerequisites for bloggers at HufPo are a hindbrain and access to a keyboard), Monfort has six or seven masters degrees in a variety of economics or finance-related subjects, although he apparently has yet to learn of the concept of diminishing returns.

Monfort is infamous for his frequent spamming of university departments and research institutes, either to recruit scholars to join his team of Expert Dreamers or to rope people into reviewing his inane book chapters. This incessant behaviour has actually resulted in not one, but four entire threads dedicated to him at econjobmarketrumors.com, a website which itself will likely confirm any suspicions you might have that economics comprises a disconcertingly-large number of self-obsessed sexist, racist, socially-inept douchebags. If you’re in the mood for a little more entertainment, JP Monfort has a slew of videos online in which he mumbles through various aspects of his master plan – awkward, amateurish productions set in bizarre locations such as the side of a road or what appears to be a swimming pool shower. He also has an online site dedicated to “The Clinton Letters”, chronicling correspondence between him and Bill Clinton, which appears to amount to standard brush-off replies from the latter’s office, including a thank you for the Monfort Plan t-shirt that was sent his way.

His book is currently available on Amazon, although plagued by one-star reviews from irate academics targeted by his e-mails. One reviewer notes:

 While I can’t comment on the content of the book, if it’s anything like the vague, grandiose drivel I receive weekly in my inbox, I doubt it has anything practical to say about modern economic, social, and political problems…

Indeed, from reading the book chapters he has sent my way, it is difficult to discern whether Monfort is deluded, a scam artist, or some sort of weird meta-troll. His prose is circular and vague, nonsensical but coherent enough to pass for an NGO report. Some sections are devoted to describing – essentially paraphrasing – the of work other mainstream development experts, including Milanovic himself, Daren Acemoglu and our own Nancy Birdsall, rather than contributing anything new to the development debate.  Chapter 29 of the seemingly-never-ending, rambling book describes the role of the “Chief Dreamer” (inevitably Monfort himself) who would be in charge with leading the way in the fight against world poverty:

“The Chief Dreamer must remain awake while others sleep, must work in the interest of the developing world and propose forward-looking ideas that are realistic so that the reader find sufficient matter to employ his or her speculations for the rest of her life. The Chief Dreamer must be a twenty-first century Jules Verne who conquers through persuasion and not imposition. The Chief Dreamer must combine the qualities of George Kennan and Jean Monnet and be determined to defend the priorities of the vulnerable. The Chief Dreamer must be multilingual to address a variety of audiences in different geographies and must be multidisciplinary to understand the complex roots that drive today’s increasing inequality gap and inequality.”

The only novelty in the book is born out data generated by the sheer number of people Monfort has managed to piss off. You see, so many people have requested to be removed from the ever-growing list of Expert Dreamert that Monfort dedicates an entire segment of his book to analysing the breakdown of which academic disciplines are most likely to click unsubscribe:

I soon realized academics were overwhelmingly requesting to be removed from The Decem List. The trend was so notorious that I decided to create this subsection to express a concern. There is a subset of academia who may be reluctant to embrace a new economic paradigm, as it relies on new ideas not supported by the orthodox theories that for decades have been feeding the intellect of university professors from the world’s best universities.


I’m curious as to what it is about Mr. Monfort’s incoherent ramblings that appeals more to anthropologists and agricultural engineers than to economists, psychologists and sociologists. Alas, it will remain one of life’s great mysteries.

Oh, by the way, Mr. Monfort could you please drop me from your mailing list?

Make love, not development goals

"Son, we need to talk about your development indicators."

“Son, we need to talk about your development indicators.”

I was pleasantly  surprised to see this post by Chloe Safier on Duncan Green’s excellent blog. From her final paragraph:

Ignoring that people have – and enjoy – sex diminishes the full reality of people’s experiences and relationships. If the development and donor communities, could shift their conversations around sexual and reproductive health and rights, empowerment, and gender to include the people’s whole sexual lives, we’d all be better off.

Safier is basically making the case that a focus on reproductive rights is not enough, that the development industry also needs to start discussing how to improve people’s sex lives – not so much in a broad sense (e.g. maximizing per capita orgasms) but more of a Sen-style capabilities approach to ensure everyone has the potential for a decent sex life.

I’m not sure I can fully embrace Safier’s recommendations – that the ever-growing list of NGO/donor priorities should include sexual enjoyment, but this is more due to a general unease about NGO mission creep than about a focus on sex.That said – I am astonished by infrequently developmentistas talk about how important sex is for human welfare. At risk of sounding ageist, I suspect this because the field is still dominated by the middle-aged and older crowds, cohorts who have a pretty solid history of looking down on talking openly about sex. I don’t know whether this is purely generational, or just a fact about getting older – I’ll report back in 10 or 20 years or so.

Economists, in turn, see almost incapable about talking sensibly about sex. Last year, at a seminar in Oxford – I witnessed a group of academic economists argue over why birth rates went up in regions of Spain after outbursts of Basque separatist violence. The consensus was that violence forces people indoors, which in turn lowers the opportunity cost to sex. That is, now we are unable to go to the park, we’re more likely to stay at home and, well, get it on. Upon reaching this conclusion, the room of mainly-male academics erupted in a bout of giggling.

This may well be true – but it’s emblematic of the way that we (economists) tend to bulldoze over sex with standard Econ 101 explanations. We can, in part, thank the late Gary Becker for taking all the fun out of sex.

Or the emotion – there’s a lot of reasons we jump into each other’s arms. To take the Spanish example: extreme stress tends to drive us to do things which mitigate stress, sex being at top of that particular list.

Time for a global sexual satisfaction index? Maybe not – but it’s still refreshing to see this being discussed openly.

Come work for me (in London)


CGD is hiring a research assistant to work full time here at in the London office with Vijaya Ramachandran and me. The work would primarily be on Illicit Financial Flows, but also to support other research which Vij and I work on, which include humanitarian assistance/firm growth in Africa as well as land tenure work.

You can read the full ad and apply here.

This is a great opportunity to get involved with a really important and exciting line of research. Let me also take this moment to say: CGD is a fantastic organization to work for. My 10 months in the London office have been some of my happiest to date – it is one of the rare places where I actually feel excited on a Sunday evening knowing that I’ll see my colleagues the next morning. Even though many of us all work on different things, we act like a big family, one which I would encourage anyone to join.

We wish to become your friends if we may

“Kikuchiyo, a fake birth certificate and stolen armour does not make you a samurai.”

I recently received an e-mail from the renowned “Journal of Economics World” whose tagline is “From knowledge to wisdom!”

Dear Matthew Collin ,

This is Economics World (ISSN 2328-7144), a professional journal published across the United States by David Publishing Company, New York, NY, USA.

We have learnt your paper “ Persistence in the effect of  birth order on child  development: evidence from the Philippines ” in the 2014 CSAE Conference on Economic Development in Africa, March 23-25, 2014, St Catherine’s College, University of Oxford, UK.

We are very interested in your research and also would like to publish your other unpublished papers in Economics World (ISSN 2328-7144). If you have the idea of making our journal a vehicle for your research interests, please feel free to send electronic version of your papers or books to us through email attachment in MS word format.

Currently, we are trying to invite some scholars who are willing to join our editorial board or be our reviewers. If you are interested in our journal, please send your CV to us. Hope to keep in touch by email and can publish some papers or books from you and your friends. As an American academic publishing group, we wish to become your friends if we may.

Economics World is, as best I can tell, a `paper mill’ – a place where the truly desperate send their work (paying roughly $60 a page!) to show that it can be published.

Not today EW, not today.