Despite the masses of negative publicity heaped on the continent by the famed Nigerian spam industry, Africa is actually one of the world’s safest places to go online in—featuring seven of the ten nations least attacked by malware.

Virus-checker company AVG surveyed 127 million computers in 144 countries and calculated the average rate of attacks—with the African nation of Sierra Leone emerging as the least assaulted, with only one virus event logged per 692 web users.

Really? My experience (admittedly primarily limited to the offices of Malawian government) is that you should treat any internet-capable computer south of the Sahara as a instant death.

Since internet and e-mail access on the continent tends to be a less reliable and more expensive, a lot of information transfer is done using memory sticks. Even if computers aren’t subjected to very many attacks from the outside,  it just takes one infected stick and a few marginally motivated employees to spread a virus to every other computer in the office. Many of these are the nasty, older viruses/trojans/worms which knock out the antivirus program’s ability to function, which means that AVG can’t see them.

This can happen astonishingly quickly. Tired of spending five minutes scanning my colleagues’ USB drives every time I wanted to get an Excel table from them, I once tried to quarantine and clear every computer in my department,  installing new (trial) antivirus on each cleared system. Unfortunately I missed a couple of computers, and within month when the trial software stopped working, the entire department had been reinfected.

It’s possible that AVG’s results are due to pretty extreme selection bias  on two fronts:

1. AVG users are probably a little more concerned and careful than those who don’t bother to update (as most don’t).
2. As I mentioned before, many attacks can knock out AVG, which means no reporting.
3. Many don’t bother to update AVG’s virus definitions, leaving the program incapable of detecting or reporting new viruses.

So yes, Africa might be a safe continent to go online by yourself in a locked room with tape over your USB drives, but any file-swapping outside the net should be handled with extreme caution.

Hat tip to Chris Blattman’s Google Reader shared items.

Where would people end up if there were no barriers to movement?

The folks at Gallup, who recently produced some interesting figures on the large number of people from developing countries who  would like to permanently emigrate, have followed up with new data on where people would like to move to.

Using their survey data to predict the proportion of the population who would move if all barriers were dropped, they constructed net migration indices, basically showing the increase/decrease in adult population which would result if everyone got their wish. For example, below we have the top gainers (in percentage terms):

The US and UK aren’t far behind with 62% and 60% predicted increases, respectively. The biggest losers aren’t terribly surprising:

The biggest weakness in these indices has to do with the assumptions about movement: each person responding to the survey is conditioning their choice on current conditions in the country they move to. So if I am one of the Haitians that has ‘voted’ to move to the US, my response is conditioned on the current US population, before anyone else moves there. My answer (or, in the aggregate, the probability of choosing the US) may change after a 60% foreign-born increase in the adult population .

While other research has shown that people are quite keen to move to countries with a larger co-national diaspora, it’s not clear that increased net migration from a third country will increase the chances I’ll choose to move somewhere (it’s more likely that my preferences would shift elsewhere).

Similarly, my decision to move at all might be affected, either positively or negatively, by the number of people who have left before me. Neither of these conditions is taken into account when Gallup constructs these estimates.

It would be very difficult to actually get at the equilibrium result – one would have to constantly condition the survey questions, and make some assumptions about strategic behaviour (i.e. where would I move if I thought X number of people from another country are moving there, and vice versa).

I’ve already written about how I think migration is a key part of improving welfare in developing countries. Ranil has rightly pointed out that it isn’t always analogous with development, as it isn’t clear that there are any direct benefits to the sender nation as a whole (although some work has shown that remittances can have some beneficial aggregate effects).

However, from an individual-welfare perspective it’s a possible win. The question we desperately need to be asking ourselves is what happens to welfare in the long run under our two scenarios: do we have a greater change at improving welfare by importing poverty, or exporting assistance? I’m not sure.

I was 16 when I first saw a slum tour. I was outside my 100-square-foot house washing dishes, looking at the utensils with longing because I hadn’t eaten in two days. Suddenly a white woman was taking my picture. I felt like a tiger in a cage. Before I could say anything, she had moved on.

On the educational value of these trips:

To be fair, many foreigners come to the slums wanting to understand poverty, and they leave with what they believe is a better grasp of our desperately poor conditions. The expectation, among the visitors and the tour organizers, is that the experience may lead the tourists to action once they get home.

But it’s just as likely that a tour will come to nothing. After all, looking at conditions like those in Kibera is overwhelming, and I imagine many visitors think that merely bearing witness to such poverty is enough.

A few months ago Ravi Kanbur wrote an interesting paper suggesting that development workers should have to go on routine ‘exposure’ trips, where they spend a few days staying in a rural village to get a better perspective on poverty. Several others thought this would be a good idea, but I remain concerned that this would be nothing more than a glorified poverty safari, akin to earning a merit badge in the Boy Scouts.

The very first post on this blog was on poverty safaris. What do you think of them?

Hat tip to Aid Watch for the link.

In fact, Peirce and others in the social sciences invented the RFT decades before the technique was widely used for therapeutics. By the 1930s, dozens of American universities offered courses in experimental sociology, and the English-speaking world soon saw a flowering of large-scale randomized social experiments and the widely expressed confidence that these experiments would resolve public policy debates. RFTs from the late 1960s through the early 1980s often attempted to evaluate entirely new programs or large-scale changes to existing ones, considering such topics as the negative income tax, employment programs, housing allowances, and health insurance.

So the randomistas aren’t so much as a “new wave” as the “next wave.” More interesting though, are Manzi’s thoughts on external validity:

By about a quarter-century ago, however, it had become obvious to sophisticated experimentalists that the idea that we could settle a given policy debate with a sufficiently robust experiment was naive. The reason had to do with generalization, which is the Achilles’ heel of any experiment, whether randomized or not. In medicine, for example, what we really know from a given clinical trial is that this particular list of patients who received this exact treatment delivered in these specific clinics on these dates by these doctors had these outcomes, as compared with a specific control group. But when we want to use the trial’s results to guide future action, we must generalize them into a reliable predictive rule for as-yet-unseen situations. Even if the experiment was correctly executed, how do we know that our generalization is correct?

One example he discusses the frequent experimentation used in crime-prevention, and how the (very few) subsequent attempts:

Criminologists at the University of Cambridge have done the yeoman’s work of cataloging all 122 known criminology RFTs with at least 100 test subjects executed between 1957 and 2004. By my count, about 20 percent of these demonstrated positive results—that is, a statistically significant reduction in crime for the test group versus the control group. That may sound reasonably encouraging at first. But only four of the programs that showed encouraging results in the initial RFT were then formally replicated by independent research groups. All failed to show consistent positive results.

My biggest fear about the current trend in social science RCT work is not only the failure to confirm positive results, but the failure to confirm negative results. While there is a small, but real incentive to repeat a ‘proven’ randomized study in a new setting, there isn’t much being done to confirm that a negligible treatment effect doesn’t improve elsewhere. While big RCT research groups do care about external validity, it is the initial findings that get seared into the mind of the policymakers. Flashy graphs which generalize without concern don’t help.

Here’s part of the closing to Manzi’s piece, which is a must-read if you’re interested or involved in this type of work:

It is tempting to argue that we are at the beginning of an experimental revolution in social science that will ultimately lead to unimaginable discoveries. But we should be skeptical of that argument. The experimental revolution is like a huge wave that has lost power as it has moved through topics of increasing complexity. Physics was entirely transformed. Therapeutic biology had higher causal density, but it could often rely on the assumption of uniform biological response to generalize findings reliably from randomized trials. The even higher causal densities in social sciences make generalization from even properly randomized experiments hazardous. It would likely require the reduction of social science to biology to accomplish a true revolution in our understanding of human society—and that remains, as yet, beyond the grasp of science.

Gabriel Demombynes over at the World Bank blog has s0me more interesting things to say about the Multidimensional Poverty Index (MPI). There’s one claim he makes a claim which I find particularly interesting:

The MPI is a descendant of the earlier Human Development Index and is similar to the various Unsatisfied Basic Needs indices long used in many countries.

Several others, including Duncan Green, have also stated that the MPI is a natural follow-on from the Human Development Index (HDI), which I’m not sure is correct, as the two have a very different conceptual basis.

As its name implies, the MPI falls into a class of indices known as poverty measures. While they can get quite complex and opaque, the more basic of these have a similar approach: First we have to pick a welfare measure. This could really be anything that is measurable, but is most commonly income, consumption or asset wealth. Then comes the surprisingly contentious task of choosing a threshold, under which people will be classified as being poor if they do not meet it. These poverty lines can be absolute or relative, the latter indicating a greater concern for inequality than absolute deprivation. Counting the poor gives us a final tally of those living below the poverty line.

The MPI is an extension of this approach, instead using a range of indicators wrangled together a multidimensional poverty line. While single-dimension poverty lines make very precise statements about people along one dimension (Person i can only be not-poor if their income Xi exceeds the poverty threshold P (Xi >P), multidimensional lines can classify two households as being poor even when they face vastly different circumstances. For example: two people might be equally unhealthy, but one has enough asset wealth to be classified as “not-poor”. The MPI also tries to include information on the severity of poverty, for those that face many different deprivations all at once, a conceptually similar approach to the poverty gap and squared poverty gap indices.

The MPI, like the other poverty measures that came before it, focuses on a particular segment of the population, discarding all information about the non-poor. Because it is derived by counting individuals whole fall into a pre-specified condition, it is best thought of as a way to describe the state of this sub-population, rather than as a comprehensive indicator.

In contrast, the Human Development Index was intended to be used to make statements about the overall progress of a country’s development. While all of its components are aggregated from individual or household information, or from counting those in a certain condition (i.e. those that are literate, or who have died this year), they do not give the same type of insight. The education component is similar (we are just counting those who are in the state of literacy or who are enrolled in school), but with GNI and life expectancy, we aren’t really counting anything, we’re expressing moments and expectations from interesting country-wide distributions. We cannot say “X number of people have an HDI of Y.”

The HDI was initially introduced as an alternative to just relying on income as a measure of human welfare. This way of looking at the world, which became very popular following Sen’s work on the capabilities approach, also motivates the MPI as an alternative to only considering poverty in income. The weakness in both the indices is in their method with dealing with multidimensionality – by using ad hoc methods of averaging different dimensions together to come up with a single number.

So, when describing the MPI to someone new, one might refer to it as “an extension of traditional income-based poverty measures, taking into account the multidimensional nature of poverty, much as the Human Development Index considers the multidimensional nature of development. Both consider just measuring income, or consumption, to be insufficient,” rather than as a natural follow on from the HDI.

No one else could possibly think of the same thing

There is a wonderful moment in the 1979 film “Life of Brian,” where the People’s Front of Judea, an anti-Roman revolutionary group, embarks on a mission to kidnap the wife of Pontius Pilate to force him to make political concessions. As they sneak through the palace, the group bumps into the Campaign for a Free Galilee, another separatist movement which is also planning to capture Pilate’s wife. The two groups argue over who gets to do this and end up killing each other before Pilate’s guards even get a chance to intervene. You can watch the scene here (fast-forward to 5:00).

Monty Python’s comical vision of a fracture resistance, comprising dozens of similarly-named groups with redundant objectives, is strikingly familiar in the world of aid. While the NGO community suffers from these problems the most, it is official donor fragmentation and duplication which is particularly disheartening as its relative size (a few dozen approaching 100 donors versus hundreds of international NGOs) means coordination and communication ought to be easier.

It is in this muddled context that USAID, has just announced its own “plan” for achieving the 2015 targets for the Millennium Development Goals, soon to be followed by USAID’s overall strategy for development assistance, all ahead of next month’s UN summit on the MDGs.

Let’s simplify things for a minute: Imagine a world where the US was the only donor. In this context, an individual strategy seems quite sensible – the solo donor just needs to decide on what its objective function is (i.e poverty reduction, growth, reaching the MDGs) and allocate aid flows accordingly to best achieve those objectives.

Now let’s move to a world where there are two donors and make the rather strict assumption that they have the same objective (perhaps achieving the MDGs). If each of those donors continues to operate as if they are in a vacuum, without knowledge of or concern over each other’s movements, they will both tend to spend money on the same programmes in the same places. This results in “donor darlings,” countries and programmes that have too many donors and probably receive too much aid money.

The are two ways to back out of this corner: one is to reduce the number of agents dolling out cash. While some donors already implicitly do this by ceding money to larger multilaterals like the World Bank or the UN. However most governments are less keen: the is a perception by the public that aid spent by proxy is less reliable (this view also extends to recipient governments), despite the general lack of concern for analytical oversight of their own aid agencies.

The other, more depressing reality is that while helping people makes us feel good, being identified as the helpers makes us feel even better. As a result we get stickers that read “From the American People” and road signs detailing exactly who paid for this 500m stretch of highway. It’s more difficult to claim the credit when aid is given fungibly to a third organisation.

The other answer to duplication is coordination, which happens in a limited sense in some countries. Again, incentives often get in the way; some methods of helping are less costly, more popular or more photogenic than others. Even when we have a world of perfect information, where DFID and USAID can communicate in real time on their future plans, they are likely to continue, for political reasons, to clump their efforts in the same areas.

The US is renowned for eschewing coordination whenever possible. Much of the electorate practically sits in the same vacuum I described earlier – for most of them the US is the only official donor that really matters, which creates enormous incentives for USAID to act as if this were true. In this context, a grand strategy might actually be a good thing. By announcing “this is what we are going to do regardless of what anyone else is doing,” USAID might be allowing more flexible, more thoughtful donors to pursue complementary strategies. In essence, by declaring openly that they are going to kidnap Pilate’s wife, they nudge other groups to focus their efforts elsewhere. Let’s hope they respond to that nudge.

For a more detailed discussion of the US’s strategy, see Duncan Green (who I owe a hat-tip) or Laura Freschi at Aidwatch.

Duncan Green introduces us to the new Multi-Dimensional Poverty Index (MPI), developed by the Oxford Poverty and Human Development Initiative (OPHI):

The MPI brings together 10 indicators of health (child mortality and nutrition), education (years of schooling and child enrolment) and standard of living (access to electricity, drinking water, sanitation, flooring, cooking fuel and basic assets like a radio or bicycle). It’s thus a logical extension of its predecessor, UNDP’s pioneering Human Development Index, launched in the first Human Development Report back in 1990, which combined life expectancy, education (literacy + enrolment rates) and GDP per capita.

The measure, like the HDI, is part of an attempt to get a “better measure” of poverty, by including many non-income indicators. While I think most would agree that policymakers and researchers should always consider non-income indicators of welfare, does it make sense to average them out into a single index?

What precisely are we measuring when the HDI for a given country increases by .01? These questions always seem to lead back to the original indicators: “A advanced in rank because of education improvements” or  “B is lower than C despite being richer, because life expectancy in B is much lower.” Given that we need to unpack these indices to figure out what’s going on, why do we bother to pack them in the first place?

Duncan, always open for a healthy debate, has already posted a criticism of the MPI by Martin Ravallion of the World Bank, which questions the implicit values placed on different indicators when they are weighted:

The index is essentially adding up “apples and oranges” without knowing their relative price. When one measures aggregate consumption from household-survey data for the purpose of measuring poverty, as in the World Bank’s “$1 a day” measures, one relies on economic theory, which says that (under certain conditions) market prices provide the correct weights for aggregation. We have no such theory for an index like the MPI. A decision has to be taken, and no consensus exists on how the multiple dimensions should be weighted to form the composite index.

On closer scrutiny, the embedded trade-offs (stemming from the weights chosen by the analyst) can be questioned, and may be unacceptable to many people.  In the context of the HDI, I pointed out 15 years ago that by aggregating GDP per capita with life expectancy the HDI implicitly put a value on an extra year of life, and I showed that this value rises from a very low level in poor countries to a remarkably high level in rich ones (4-5 times GDP per capita).   If it was made clearer to users, I expect that they would question this trade-off embedded in the HDI.

The MPI index faces the same problem. How can one contend (as the MPI does implicitly) that the death of a child is equivalent to having a dirt floor, cooking with wood, and not having a radio, TV, telephone, bike or car?  Or that attaining these material conditions is equivalent to an extra year of schooling (such that someone has at least 5 years) or to not having any malnourished family member?  These are highly questionable value judgments. Sometimes such judgments are needed in policy making at country level, but we would not want to have them buried in some aggregate index.  Rather, they should be brought out explicitly in the specific country and policy context, which will determine what trade off is considered appropriate; any given dimension of poverty will have higher priority in some countries and for some policy problems than others.

One could continue to argue about the weights – but Ravallion’s argument will still stand. I fail to see why these indices amount to anything more than intellectual exercises – while the HDI has got us all thinking about other things than income, has it really been useful as a method of actually measuring development? Is the MPI likely to do any better with poverty?

Preference aggregation can be tricky business

A key assumption behind the Global Burden of Disease project is that it is possible to come up with a “Disability Weight” for each health state.  Diseases conditions that are considered worse than other carry higher disability weights than others.  A very important issue in the development of such weights is the question of who should define these conditions?  Should those who have the conditions be the best judge or are they biased?  Should healthy people who have never experienced these conditions be the judge?  Should doctors decide?  Should policy makers?  Should health economists (gasp!!)?

In the past, the GBD has relied upon “expert opinion” to make such decisions.  Well, it seems for the next update of the GBD, which is currently underway, you can also be an expert.  I came across a link to the following survey earlier today that allows you to have some input in these weights.

That’s Karen Grepin discussing an attempt to aggregate beliefs over disease burdens to better define the weights given to different ailments. This is a very similar exercise to preference aggregation, where we attempt to construct a unified set of beliefs that will govern public policy. The result is something approaching a social welfare function, which allows us to make statements like “Society strictly prefers A to B.” One way of doing this is to get a sample of individuals to compare different states and to try and tease out an overall ordinal ranking of these states. Using Grepin’s example, each person has to make a pairwise comparison:

The first person has swelling and tenderness in the testicles and pain during urination.

The second person has lost part of both legs, leaving pain, tingling, and frequent sores in the stumps. The person has great difficulty moving around and has episodes of depression, anxiety and flashbacks to the injury.

By asking enough people to compare different states with different combinations of symptoms, we can tease out their overall ranking of those symptoms – how this is done can sometimes be contentious and quite technical. That ranking then represents the best approximation of everyone’s relative rankings of disease burden.

This isn’t a new method of trying to better calibrate policy to the wishes of many. As the US Army drew closer to victory in the European theatre during WWII, there was some concern over which soldiers would be allowed to return home first. Instead of making an arbitrary decision as to what characteristics led to early demobilisation or imposing a lottery, the Army actually surveyed several thousand active servicemen, asking them to rank soldiers with different circumstances (taken from Peyton Young’s book Equity: in theory and practice):

After the war when the Army starts releasing soldiers back to civilian life, which of each of these two groups of men do you think should be release first?

1. Men with dependents OR men over 30  years of age
2. Men who have been in the army longest OR men with dependents?
3. Men over 30 years of age OR men who have served overseas
4. Men who have served overseas OR men who have been in the army longest?

Based on how respondents ranked the various ‘states’, the Army was able to aggregate those preferences and derive a point system for which each circumstance (having dependents, being older, etc) was weighted. Servicemen with the most points were given priority for early release. The fairness in the system was inherent in the way is was derived: through aggregation of the beliefs and preferences of those that would have to live with its consequences.

This attempt to re-calibrate the weights placed on disability is similar in its approach, save for a significant flaw: the sampled respondents are self-selecting their way in (anyone can go and contribute here, as I already have). The responses will be primarily from the rich, healthy, internet-using population. The survey seems to ask questions which could be used to weight responses, (the age, location and socioeconomic status of the respondent), but I can’t imagine there will be enough of a response from areas of the world which actually have the higher disease burdens to overcome this.

At least it is a step in the right direction. Imagine a future where the priorities areas for aid were not decided by experts and technocrats at a global level, but instead were the result of careful preference aggregation from the populations who stand to benefit? I long for the day where DFID, before deciding where to spend its money in Zambia, runs a representative survey which asks:

Rank these three outcomes:

20 new public schools OR 5 new hospitals OR 3,000 new jobs

So I watched Conan the Barbarian tonight, and I was particularly struck by the scene where the young Conan is forced into slave labour, condemned to push a wheel round and round for years. You can watch the scene here (embedding is disabled so you’ll have to click through to youtube:

As I watched it I suddenly had a really familiar feeling. Why was that? Oh, that’s right: Playpumps!

You can read about the Playpump controversy along with some pointed criticism over at Barefoot Economics and Aid Watch. I am also deeply disappointed the pumps didn’t succeed in churning out out a few million Arnold-sized children.

Honestly, folks, we don't know what the impact will be.

Over at Aidwatch, Alanna Shaikh, citing a few others, considers the limits of impact analysis. At one point she cites a post by Steven Lawry:

“Many real-world problems are not easily described with the kind of precision that professional mathematicians insist upon. This is due to the limitations of data, the costs of collecting and analyzing data, and the inherent difficulties of giving mathematical expression to the complexity of human behavior.” This strikes me as very true. At what point are we expecting too much from our impact assessments?

While the more rigorous impact assessments certainly require some statistical knowhow and reliable data, they don’t necessarily require giving “mathematical expression” to human behaviour. Even though the resulting academic publications might have some calculus window-dressing, an impact measurement is generally about as atheoretical as they come: what was the impact of X on Y? When academics move on and start asking why X impacts Y, they then often retreat to the black box of mathematical modeling (usually in a desperate attempt to avoid qualitative methods, which Chris Blattman writes an excellent post about here).

Alanna also discusses a point made by Andrew Natsios:

Natsios points out that USAID has begun to favor health programs over democracy strengthening or governance programs because health programs can be more easily measured for impact. Rule of law efforts, on the other hand, are vital to development but hard to measure and therefore get less funding.

I think this is the most important criticism of over-reliance on empirical assessment – donors will prefer to fund causes that can easily signal an impact that can be touted back home. A reasonable counter is that those that swim in murkier waters just have to work harder to show their impacts, but in reality they are more likely to either let effort collapse, or just migrate over to programs that do get the funding.

While I’m partially sympathetic to doubts about impact-analysis, I think that much of (but not Alanna’s) the criticism is self-serving: let us continue using the same methods we’ve always used, which happen to always show an impact despite the never-ending micro-macro paradox.

That’s fine if you choose to reject statistical rigour, but please don’t pop up five years later and claim that your project/aid flow is responsible for all sorts of wonderful things you can’t really prove. Some may be content with photos, anecdotes and correlations, but don’t be surprised if the rest of us aren’t.

This doesn’t mean that everything should (or could) be judged by a hardcore RCT starting tomorrow, but when the evidence is less direct, the onus is on the presenter to be more modest and careful with their assertions.