Help me test a (very silly) hypothesis by answering a few questions

boss

I’ve held a silly hypothesis in my head ever since I was a grad student, but never had the time/resources to test it. I just recently came across a publication which drastically reduced the costs to testing the idea. It will almost certainly result in a “jokey” paper, but a fun one nonetheless.

But I could use your help. I have constructed an online survey displaying photos of people, and I need respondents to tell me whether these people are smiling, frowning or have neutral expressions. There are over 170 questions, but they are randomized, so even if you only manage to answer a few (and then close the window), it still would help a lot!

I can’t tell you what the idea is just yet, because it might spoil how you answer the questions. More information to follow, once enough people have answered the survey.

Click through here to enter the survey.

On Hirschman, measurement and RCTs

11-baboon-in-pod

Every intervention is unique, perhaps unintentionally so.

Recently, Dan Honig of Johns Hopkins forwarded Ranil and me some thoughts he had in reaction to an Albert Hirschman on development projects that he felt was pertinent to the discussion on the pros and cons of RCTs. What followed is a discussion (rant) between Dan, Ranil and me. I’ve edited out the e-maily bits for clarity:

Dan:

Matt, just after I hit send on this I realized I should have included you on this – I generally think you’re right on RCTs and the stale-ness of the conversation (and discussed this with Ranil a few months back some 2 days after you had dinner with him, hence the cc to Ranil) but feel like I’ve never seen this Hirschmann frame and wondering if it struck you as interesting. And yes, basically I’m trying to catalyze you writing something cool on this so I can quote/reference it down the road

Reading Hirschmann’s Development Projects Observed for the first time, and as I read it he’s with [Lant Pritchett and Michael Woolcock] on RCTs and causal density in international development projects. The quote below is from page 186 of the 1967 edition; italics are his, brackets mine; just before this he suggests we may not be able to identify good indicators of effects ex-ante and thus presumably couldn’t be pre-specified in a trial, meaning presumably we would be ill served by an RCT on a particular intervention even if we ignored external validity concerns.

“The indirect effects [of development projects] are so varied as to escape detection by one or even several criteria uniformly applied to all projects. Upon inspection, each project turns out to represent a unique constellation of experiences and consequences, of direct and indirect effects.”

 

Matt:

Hey Dan, that’s a really interesting quote by Hirschman. If my interpretation is correct, it seems to be more damning for empirical evaluation in general than for RCTs in particular.

I’m not sure how I feel about this. Even if you move away from a simple, reduced form causal framework, Hirschman’s critique seems like it would apply. Even if development is a messy, complex thing that can’t really be boiled down in an impact evaluation framework, we still rely on measurement when we talk about development, and any given set of measurements is going to leave out things which might matter which are unmeasured. We can point at improving test scores but leave out student stress, etc, and the set of things that we leave out that might be important will change depending on the context. I guess I see this as a problem of measurement rather than as a problem for RCTs.

I also wonder what this means for how an empirical researcher operates. Over the last few years, I have become incredibly suspicious of surprising, counter-intuitive results, where a researcher measures something outside of the standard set of outcomes and finds a result. In a world of multiple hypothesis tests, expanding the set of outcomes to include as much of Hirschman’s unique constellation as possible will open up the door to a lot of false positives which will end up getting written up and published.

So that was a rant. Um, what do you think Ranil?

Continue reading

Come work for me in London or Washington D.C.

darth

Like development? Like statistics and coding? Come join me and Vij Ramachandran at CGD.

The Center for Global Development, an independent, non-partisan research organization in Washington, DC and London, UK seeks a Research Assistant (RA) to support the work of Matt Collin and Vijaya Ramachandran. The successful candidate will have experience with research in economics, public policy, political science or a related field and will be based either in London or in Washington DC.  Applicants must demonstrate strong quantitative, analytical and communication skills. The position is well-suited to those who are considering doctoral study in the future.

 

 

Universal Basic Income: The Next Generation

"Wait, you're saying that in the Federation you don't have to worry about money? You can just schlep around the galaxy seducing alien women?"

“Wait, you’re saying that in the Federation you don’t have to worry about money? You can just schlep around the galaxy seducing alien women?”

Over at Five Thirty Eight, there’s a nice piece by Daniel Flowers on the idea of a universal basic income (UBI). Proponents say it will allow people to choose their careers and live their lives without having to worry about ever being poor. A common criticism is that it will create a massive disincentive to work at all. Several experiments have already been run which have found small, but non-negligible effects on the willingness to work. It is one of the outcomes that a host of new experiments of giving people a long term guaranteed basic income will test.

I am a little worried that these new experiments won’t capture the long term, generational impact of a universal basic income. Let’s imagine I really wanted to be a filmmaker (*cough*), but decided to become an economist because filmmaking is more likely to leave me in poverty. If I’m half way through my career as an economist and I start receiving a basic income, it might be too late for me to really break into filmmaking. Even if I pull it off, adjustment costs will be high, and it’s likely I’d end up embarking on a career which would be less successful than if I had started at a much earlier age. It is these decisions that will largely be picked up when the targets of a UBI experiment are largely adult workers.

What is more interesting is the impact on the next generation. Let’s imagine the UBI is introduced and governments can credibility commit to providing it for one’s entire lifetime. Now, all those aspiring filmmakers can select into the job of their choice with lower adjustment costs and a higher likelihood of actually being accomplished at what they’d most like to do. Who knows if this would have a net positive or negative impact on the creation of value, but it certainly would lead to better sorting. But even the most ambitious UBI experiments which are being proposed are unlikely to pick up these effects. Instead what we need to do is find a group of high schoolers and offer some of them (randomly) a credible lifetime UBI, then sit back and see how it affects career decisions and labour market participation in the long run.

As a side note – how much does relative poverty in developing countries lead to sub-optimal career decisions?

The road out of hell

In the FT yesterday, Branko Milanovic suggested that we might be able to increase global migration by reducing the citizenship rights of migrants. This is not a new idea – Lant Pritchett brought it up ten years ago and it is widely practiced by many Gulf states.

Over at Crooked Timber, Chris Bertram made it clear he really doesn’t like this idea, comparing Milanovic’s suggestion to “recreating apartheid” (I suspect the word apartheid will eventually be subject to its own form of Godwin’s law):

Milanovic wants us explicitly to abandon the liberal and democratic principles of legitimacy that those who are subject to the laws of a society should (in time in the case of migrants) get to have the right to make those laws. In doing so, he goes far beyond similar proposals (for example from Martin Ruhs that have been explicitly temporary in nature and have largely focused on labour-market rights. Milanovic’s lack of commitment to the norms of liberal democracy also comes across in the fact that he holds up illegitimate and tyrannical states, such as the Gulf kleptocracies, as models for his proposed policy.

Part of what’s going on here is the economist’s perspective on policy, which just focuses on net improvements in well-being or utility, with income serving as a proxy, and which doesn’t, therefore, see human beings as possessed of basic rights which it is impermissible to violate. Rather, all and any rights can be sacrificed on the altar of income improvement, just in case someone is poor and desperate enough to make a deal (who are we, paternalistically, to stop them?). The road to hell is paved with Pareto improvements.

Let’s be absolutely clear: we are already in hell and we are trying to find a path out of it. International migration restrictions – as they stand – already enforce a global system of apartheid. Most of global inequality in income (and likely in health and happiness) is driven mainly by where  you are born. By preventing someone mired in poverty overseas from moving to a place where they can make a better life for themselves – even temporarily – we are implicitly denying that person the same rights that we enjoy every day (rights that most have us have inherited, not earned). These are also arguments that Pritchett made before.

Human beings have a proximity problem: inequalities in outcomes or rights which are proximate to us (on the right side of an arbitrary national boundary) are weighted much higher that massive, gaping inequalities which are harder to observe because the people bearing the brunt of that inequality happen to live overseas.

We would all agree that a migration system which allows for restricted freedom is a worse solution than a system which allows for the same amount of migration with no restrictions on freedom. But the latter system does not exist, nor has anyone managed to propagate a convincing way to get there. I don’t know if a Milanovic/Pritchett system would work, but I can think of two main reasons why we might not want to consider it:

  1. There is a lower cost path towards a system which does not limit freedom that we can implement sooner.
  2. Adopting a system based on limited citizenship now will somehow make it harder to move to a free system later on.

If Bertram really wants to make a convincing case against the Milanovics of the world, he needs to start by showing us a better road out of hell.

The limitations of the Absolute Palma Index, in two graphs

OVER_9000

Last year, the ODI’s Chris Hoy released a really useful and thoughtful paper pointing out that the basic maths of inequality are often not on the side of the poor. Even if economic growth is evenly spread, the absolute difference between the incomes of the poor and the richest must increase. That is, if you are 10 times as rich as I am and our incomes both grow by 10%, you’ll be taking home more money than I will at the end of the day. If we wanted to see a decrease in absolute differences of income around the world, it would require that the income of the poorest grow a great, great deal faster than that of the richest, something we are unlikely to see any time soon.

The unanswered question, and one that Hoy even posits himself end of the paper, is whether or not focusing on absolute differences in income makes more sense than doubling down on the relative differences in income that are captured by traditional inequality measures such as the Gini, Thiel or Palma indices. We know that income is correlated with lots of good outcomes for the beholder – better health, education, happiness and political power. However, if we are being truly honest with ourselves, we would have to admit that we don’t quite fully understand whether relationships are absolute or relative in nature (although we suspect both matter for happiness). Do the richest 1% of Americans have more political power in the US than the richest 1% of Nigerians have in Nigeria? These are the questions we must ask ourselves if we are to make a strong case for caring about absolute income differences.

In the meantime, I woke up this morning to find that Nick Galasso from Oxfam has made a pitch for using the “Absolute Palma Index” as the next big measure of inequality. The Absolute Palma is a variation of the Palma Index of inequality, which itself is the ratio of the share of income earned by the top 10% of the distribution and that of the bottom 40% of the distribution. The Absolute Palma, by contrast, is the absolute difference between the average income of the top 10% and the average income of the bottom 40%.

As the title suggests, I think there are limitations to the Absolute Palma Index, so consider the post a word of caution. I can think of one strong case against absolute measures: while they might be reasonable at describing immediate gains across a country’s income distribution after a year of growth, they aren’t very useful at describing differences between countries across the globe.

I happened to be playing around with data from Christoph Lakner and Branco Milanovic’s paper on the global income distribution, so I decided to see how the Absolute Palma Index varied across countries. Check out the graph below, which looks at how the Absolute Palma Index varies with mean income across countries. I’ve also highlighted countries which are either very unequal, very equal or somewhere in the middle as measured by the traditional Palma Index.

palma_avg

 

The first thing to note is that there is almost a one-to-one relationship between the log of GDP and the log of the absolute Palma. This is hardly surprising – take any income distribution and raise all incomes by a set percentage and by definition you will see an increase in the Absolute Palma. What this means is that on this index, poor countries do really, really well and rich countries do terribly. And that is most of the story. Log per capita income explains about 93% of the variance in the log of the Absolute Palma. The relative Palma explains most of the remaining unexplained variance, but on the whole has very, very little explanatory power.

The result is that we get some pretty counter-intuitive results. Even though Denmark, Sweden and Norway  are considered by pretty much every person I’ve ever ever spoken to be the most equal places on the planet, they come out as being more unequal than countries that are at the top of the relative Palma Rankings, places like South Africa, Honduras and Brazil.

Which of these countries would you rather be poor in? Presumably the one with the highest average income for the poorest 10%. If we graph the same relationship, instead using the average income of the bottom decile, we find the relationship is less strong, especially so for the poorest countries of the world. But if I had to choose whether I wanted to be born poor in a country with a high or low Absolute Palma index, sign me up for more inequality!

palma_b10

 

Now for the caveats: the data here is as good as 2008, so the basic cross-sectional relationship may have changed (although it hasn’t appeared to have done so ipapen the years leading up to 2008). There is also a difference between moving between countries of different average/median/poorest decile levels and observing individual countries as they grow richer or poorer. This means that there might be use in keeping track in how growth is `allocated’ across the income distribution, something which is already done (and was done carefully in Chris Hoy’s paper).

Absolute measures might tell us something interesting in the world, and I welcome more work on them. But there is a world of difference between adding a tool to the (now overflowing) box of inequality measures and pushing for headline measure that automatically penalizes rich, developed countries for being rich and developed. In addition, before we begin agonizing about absolute differences within countries, someone needs to make a pretty compelling case that they matter more than both absolute levels or relative differences, because these are things we already go through great pains to measure. If we are worried that the incomes of the poor aren’t growing fast enough, then why isn’t it enough to measure that?

Stata code and underlying data available here.

Update: good comments from Chris Hoy below.

The difficulty of getting good feedback

Most of us have very little clue if what we are doing makes any sense

In a piece for Project Syndicate released today, Ricardo Hausmann makes a grand case against evidence-based policies, specifically the rise of randomized controlled trials:

My main problem with RCTs is that they make us think about interventions, policies, and organizations in the wrong way. As opposed to the two or three designs that get tested slowly by RCTs (like putting tablets or flipcharts in schools), most social interventions have millions of design possibilities and outcomes depend on complex combinations between them. This leads to what the complexity scientist Stuart Kauffman calls a “rugged fitness landscape.”

After presenting a theoretical case of an RCT which tests for and fails to find an impact of tablets on learning in schools, he offers up an alternative approach, one that relies on rapid experimentation and adaptation:

Consider the following thought experiment: We include some mechanism in the tablet to inform the teacher in real time about how well his or her pupils are absorbing the material being taught. We free all teachers to experiment with different software, different strategies, and different ways of using the new tool. The rapid feedback loop will make teachers adjust their strategies to maximize performance.

Over time, we will observe some teachers who have stumbled onto highly effective strategies. We then share what they have done with other teachers.

Notice how radically different this method is. Instead of testing the validity of one design by having 150 out of 300 schools implement the identical program, this method is “crawling” the design space by having each teacher search for results. Instead of having a baseline survey and then a final survey, it is constantly providing feedback about performance. Instead of having an econometrician do the learning in a centralized manner and inform everybody about the results of the experiment, it is the teachers who are doing the learning in a decentralized manner and informing the center of what they found.

Hausmann makes a compelling argument here, but it all hinges on an exceptional premise: that teachers have access to a magical device that gives them *real time* feedback on student learning. Iteration and adaptation makes a lot of sense….. if you are in an environment where you can actually observe the immediate effects of your decisions and be sure that those decisions are having a causal impact.

But most of us are not in those environments. Many teachers might have an idea of how good their particular method is, but in absence of a technology which can provide them with high-quality real-time feedback, it would be very hard to be sure. Most of us are in an environment where we have little idea of what we are doing is effective at all. Even after 32 years of direct observation and some experimentation, I still can’t figure out if spicy food gives me indigestion.

Even when we can successfully parse the noise of life and match an action with a reaction, low-level experimentation still opens up the door to all sorts of internal biases. Human beings are fantastic at creating narratives (I feel good today, it must have been because of that thing I did yesterday) which would whither under larger-scale experimentation.

Of course there are clear examples of low level, rapid experimentation being successful when we have access to technologies that give us good, quick feedback. Bridge Academies, which is now one of the largest private school providers in the world, succeeded largely due to a very high degree of internal experimentation. But to accomplish this, Bridge had to have access to a wealth of real time data on student achievement and attendance as well as enough centralized control to be able to experiment across classrooms and schools.

But in reality these kinds of feedback technologies just don’t exist in many contexts, at least not yet. If I am working in a Ministry of Health in a developing country and I want to discern whether a given health intervention has had an impact, I won’t necessarily have access to real time data on hospital admissions. Instead, I would have to rely on costly household surveys which take time to collect. This slows down the process of iteration and adaptation to a point where a randomized controlled trial combined with some qualitative fieldwork actually looks pretty attractive.

RCTs are far from a perfect solution and Hausmann is correct to point out that they can be slow and blunt tools for figuring out exactly how an intervention should be implemented. But that is a reason to complement them with other methods – not to  chuck them out the door. If a teacher has come up with a new method of using a tablet through rapid experimentation and it is rolled out to the entire school, that method should be rigorously empirically tested. If an RCT of some new intervention finds no effect, we should turn to more rapid experimentation to find a better way.

We’ve been arguing about RCTs for years now – it is disheartening that this debate still feels very black and white.

The problem with nudges is that sometimes they don’t move things very much

Have you ever prescribed azithromycin when you didn't have to? Know what I mean?

Have you ever prescribed azithromycin when you didn’t have to? Know what I mean?

Over-prescribing of antibiotics is a problem because it speeds up the rate at which bacteria develop resistance. In a new study was published in the Lancet yesterday, researchers attempted to use a simple `nudge’ to get doctors in the UK to prescribe less often:

In this randomised, 2 × 2 factorial trial, publicly available databases were used to identify GP practices whose prescribing rate for antibiotics was in the top 20% for their National Health Service (NHS) Local Area Team. Eligible practices were randomly assigned (1:1) into two groups by computer-generated allocation sequence, stratified by NHS Local Area Team. Participants, but not investigators, were blinded to group assignment. On Sept 29, 2014, every GP in the feedback intervention group was sent a letter from England’s Chief Medical Officer and a leaflet on antibiotics for use with patients. The letter stated that the practice was prescribing antibiotics at a higher rate than 80% of practices in its NHS Local Area Team. GPs in the control group received no communication. The sample was re-randomised into two groups, and in December, 2014, GP practices were either sent patient-focused information that promoted reduced use of antibiotics or received no communication. The primary outcome measure was the rate of antibiotic items dispensed per 1000 weighted population, controlling for past prescribing. Analysis was by intention to treat.

This is a fairly standard behavioural intervention – use information (or, less graciously, spam) to nudge people into behaving in a more optimal way. The behavioural insights/economics crowd loves these interventions because they are cheap, so the cost-effectiveness hurdle is easy to overcome. However, that cheapness sometimes overshadows a bigger problem, that frequently these interventions just don’t have very large effects. Here are the results from the Lancet study:

Between Sept 8 and Sept 26, 2014, we recruited and assigned 1581 GP practices to feedback intervention (n=791) or control (n=790) groups. Letters were sent to 3227 GPs in the intervention group. Between October, 2014, and March, 2015, the rate of antibiotic items dispensed per 1000 population was 126·98 (95% CI 125·68–128·27) in the feedback intervention group and 131·25 (130·33–132·16) in the control group, a difference of 4·27 (3·3%; incidence rate ratio [IRR] 0·967 [95% CI 0·957–0·977]; p<0·0001), representing an estimated 73 406 fewer antibiotic items dispensed. In December, 2014, GP practices were re-assigned to patient-focused intervention (n=777) or control (n=804) groups. The patient-focused intervention did not significantly affect the primary outcome measure between December, 2014, and March, 2015 (antibiotic items dispensed per 1000 population: 135·00 [95% CI 133·77–136·22] in the patient-focused intervention group and 133·98 [133·06–134·90] in the control group; IRR for difference between groups 1·01, 95% CI 1·00–1·02; p=0·105).

Let’s focus on the intervention that worked: the peer information treatment. There was a clear decline in antibiotic use for the treatment group, and so the study focuses on the sheer number of prescriptions that were prevented (73,406). However, in terms of relative impact, the study barely changed behaviour. The treatment group’s prescription rate was a mere 3% lower than the control group’s rate. 

So if this is about finding cost effective ways to reduce prescribing, then I’m on board. But clearly these sort of nudges are not going to win the war on antibacterial resistance any time soon.

The ultimate Monopoly strategy and the British housing shortage

monopoly

Over at imgur, someone has discovered a Monopoly strategy that will not only let you win, but will also make the other players hate you:

A little-known rule of Monopoly is that the game has exactly 32 houses and 12 hotels. Once you run out of houses, no more can be purchased until they re-enter the supply by being sold or upgraded to hotels. If there are more players who want to build houses than there are houses available, they are auctioned off to the highest bidder, one at a time. The core of this strategy is to buy up as many houses as possible before anyone realizes what you’re doing, and DO NOT UPGRADE TO HOTELS to prevent people from improving their own properties.

Basically, you need to create several monopolies very quick and just focus on buying up all the available houses the game provides. Once there are no more houses, your friends can’t advance and so you’ll win, slowly and painfully.

Everyone suffers in this scenario, except for the lucky person who rushed their way onto the housing ladder. But note that the reason this strategy works and the first-mover will eat up all the resulting rent is because Monopoly has a purely arbitrary rule on the supply of housing. This is not a million miles away from the unnecessary planning restrictions which make it harder to build or upgrade in the UK and thus make land and housing far more expensive than they need to be.

Hate tip to kottke.org

So how do you feel about not winning the lottery?

"Here's to exogenous shocks to our neighbour's wealth"

“Here’s to exogenous shocks to our neighbour’s wealth”

Happy New Year. So I’ve been thinking a lot about the charity GiveDirectly recently. They were my charity of choice a year ago and I am planning to make another donation soon. For those of you who are not in the know, GiveDirectly makes unconditional cash transfers to poor people in Kenya and Uganda. For every dollar I donate, roughly 91 cents of that ends up with a household, which is then free to do whatever they want with it.

The other day GiveDirectly sent me an e-mail which linked to a series of interviews with residents of a single village that had been on the receiving end of these unconditional transfers. What is particularly astonishing is that the charity not only asked recipients how they were faring (pretty good, thank you very much), but roughly half of the interviews are with households which were not deemed eligible.

What I might have expected was a degree of unhappiness or animosity over not being selected to receive a $1000 USD transfer. GiveDirectly uses its own methods of determining whether or not a household is classified as “poor” (in the village in question it was households without a metal roof on their primarily residence). Even though (I presume) the charity goes through great pains to make the selection criteria transparent, to people on the ground the whole endeavour might seem a bit, well, random. A bit like a manna from heaven.

Recently, three academics who have previously studied GiveDirectly released a paper suggesting that these transfers do have some sort of negative spillovers on households that didn’t receive the transfer. Johannes Haushofer, Jeremy Shapiro and James Reisinger found that non-recipients in villages which received GiveDirectly transfers reported substantially lower levels of life satisfaction. So if this negative spillover, which I will go ahead and call the Haushofer Effect (there – I just branded it – coming to a book store near you), really exists, then I would expect a substantial amount of lamentation in the GiveDirectly interviews of non-recipients.

To the contrary, most non-recipients said that, overall, they were happy that their village had received the transfers. I found this hard to believe, but after going through 50 interviews of non-recipients, most replied positively to the question “Are you, overall, happy that GiveDirectly came to your village?,” a handful replied neutrally, and only one was vocally unhappy about it. There was another question aimed more at the negative effects of not being selected, and even then only about 25% responded with identifiably-negative comments.

So what is going on here? Why is the Haushofer Effect not appearing in these qualitative interviews? As much as I would like to believe that people do feel happy about seeing their neighbours get a shitload of money, I think I am more likely to believe one of the following:

(1) People don’t want to appear selfish, especially in front of a charity which might might might might give them a ton of cash some day. One respondent actually spelled it out: “”I am happy with your coming with the hope that one day I will also benefit.”

(2) The more complicated answer is that there is something about the conditionality of the question that changes its meaning. These families might be honestly happy about the fact that their neighbours (who are poorer) got transfers. But all the negative externalities associated with that (envy, local prices, etc) still make them unhappy in aggregate. A great example of this appeared in a recent episode of This American Life, where Neil Drummond tried to reconcile the fact that he really was happy his old friend Ta-Nehisi Coates had found fame and fortune with the reality that their friendship was slowly dissolving as a result of it.

(3) This village is different than the average village in the study above in some unobservable way.

 

I have no sense as to which answer is the most likely. And none of it will stop me from donating to GiveDirectly again. That said, while the charity should be praised for putting these interviews up on their website, they could take a step further and link to the paper on negative spillovers.

 

Update: GiveDirectly’s Max Chapnick has a helpful reply/explanation in the comments below, rightly pointing out that the academic paper I cited relies on within-village randomization (rather than GD’s method of targeting poor households), so the Haushofer effect might be primarily driven by the unfairness inherent in that lottery mechanism. This is a pretty plausible reason for the differences between the empirical study and the informal interviews.