The Proven Mean Initiative (external validity edition)

What's the sex ratio in Malawi? We're not sure, but we *know* the sex ratio in Mozambique

Welcome to the Proven Mean Initiative (PMI), where we strive to bring you only the proven, true expected values of your variables of interest.

Want to know the true mean value of something? Wondering whether or not your assumptions about household consumption are correct?  Here at the Proven Mean Initiative, we have generated evidence for these important values through rigorous, randomized sampling in key locations of the world.

You might ask yourself “what is the mean income of the poor in Zambia?” Look below at our chart of results from our completed randomized surveys:

Want to know the life expectancy where you are? Well, we’ve got the life expectancy in Turkana (43) and Kathmandu (59.3)!

Here are PMI, we ask you to stop making potential hazardous policy decisions without knowing whether or not you know your assumptions about pertinent values have been put to the test. If no one has performed a rigorous, randomized survey on your variable of interest, then you should consider any estimates unproven, and shy away from any relevant policy decisions.

Latest news: two new studies undertaken PMI researchers has discovered that the female adult literacy rate in Eastern Ouagadougou is higher than the infant mortality rate.

7 thoughts on “The Proven Mean Initiative (external validity edition)

  1. Lee

    November 19, 2010 at 6:54pm

    I don’t get it.

  2. Matt

    November 19, 2010 at 7:21pm

    The only difference between an RCT and a regular, observation survey, is that the former aims to give you some estimate of d[E[Y]]/dX, with the added bonus of exogenous variation in X, where the latter is an estimate of just E[Y].

    My point is, you would never accept a variety of different estimates of E[Y] from different contexts, locations, and times as strong evidence for the E[Y] in a new context, location, or time. If you told me that your estimate of the E[income] in India was $1,000, it would be silly for me to use that to infer E[income] in Nepal. Even if you sampled estimates of E[income] in a variety of different places, unless you also did it in a systematic (hopefully random) fashion, you still can’t make very sound inference on E[income] on average, or in any location you haven’t done a randomized survey in.

    The only differene between observational studies and RCTs is that we are estimating something different, yet suddenly, we are expected to treat a few estimates of d[E[Y]]dX as `proven evidence’ of an impact. While researchers are a little more careful with their words, many take the ball and run with it. A proven estimate of d[E[Y]]/dX in Uganda or Kenya (say, of the impact of circumcision on HIV/AIDS) is used to justify a much larger focus rollout of a policy.

    Yet it seems more reasonable when you are estimating an impact, not just a mean, but it’s all just inference.

  3. Lee

    November 19, 2010 at 8:22pm

    You’re missing the point.

    People always make implicit assumptions about d[E[Y]}/dx when they do things. Most of the time these assumptions are not based on evidence. Isn’t it better to base assumptions on some evidence?

    Take microfinance. Billions spent already on subsidizing loans, whilst the evaluation debate went nowhere. A couple of RCTs and suddenly we actually know something. And now that we do there are signs that enthusiasm for lending is waning (e.g. Gates moving completely from loans to micro-savings, pledging $500m).

    Would it be better for us to shut up and keep those results quiet because we aren’t 100% sure about their relevance in different contexts?

    Why not question the way that most aid is hyped despite approximately zero evidence? Because that is the benchmark.

  4. Matt

    November 19, 2010 at 9:55pm

    Lee,

    Of course – I’m not doubting their causal cred – in that realm I consider RCTs to be the gold standard. I have spent a year and a half on this blog questioning the way that most aid is hyped.

    But that doesn’t give RCT research houses a free pass. We shouldn’t use words like “proven,” and be careful when we make grand statements about what interventions will work. It also means we should work, really really hard to continue to replicate those results.

    I’m not saying research houses should “shut up and keep those results quiet,” but they should be more careful in how they present and aggregate those results, because like *all* research methods, there are limitations (external validity being the largest elephant in the room), and far too often those limitations get swept under the rug.

  5. Matt

    November 19, 2010 at 10:02pm

    And for the record, I don’t consider this a problem with the way most of the research is originally published, just the way it is aggregated and promoted.

  6. Lee

    November 19, 2010 at 11:16pm

    I wish that marketing didn’t matter and everyone was perfectly rational, and that everyone read economics journals. However in the real world….

  7. David Week

    December 5, 2010 at 3:51pm

    My problem with the RCTs is that they are full of marketing hype–including the original papers. One day I’ll pick one of those papers apart: my intended target is that original microfinance RCT.

    But if peruse physical science journals, you won’t find them referring to particular experimental techniques as “the gold standard.”

    And who whoever thought up that monicker seems to have forgotten what happened to the original gold standard… and the fact that gold is no longer the standard.

Comments are closed.