Why predictions fail

If *only* we had included institutions in our prediction model

Over at the Why Nations Fail blog, Daron Acemoglu and James Robinson’s discuss a set of growth predictions made by Paul Rosenstein-Rodan, the father of  the Big Push model, illustrating just how wrong they were:

Acemoglu and Robinson argue that the these predictions were off primarily because the Big Push model ignored politics and institutions:

Of course, things didn’t quite work out that way. In fact, many of the economies about which Rosenstein-Rodan was bullish are not much richer today than they were in 1961. Liberia and Haiti’s economies contracted since then. Angola, Kenya, Nigeria and Uganda haven’t done so well either. We of course know that Afghanistan, India and Pakistan grew more slowly than South Korea, Taiwan, Thailand and Singapore. Argentina and Haiti were no match for Costa Rica, the Dominican Republic and Panama.

The main reason why Rosenstein-Rodan got it so wrong is because he completely ignored the role of institutions and politics.

It’s hard to disagree that Rosenstein-Rodan should have taken these into account – but are they the primary drivers? What about geography, natural resources, export commodity prices, health and the myriad other factors which might drive a country’s growth rate? Without a little more effort, the models lack of effectiveness doesn’t tell us anything about why it is ineffective. I understand that Acemoglu and Robinson consider institutions to be the chief determinant of everything since the beginning of time, but arguing that the Rosenstein-Rodan prediction is wrong because it ignored institutions is a little like arguing that a car missing all four wheels won’t drive because – damn it – it’s also missing four tires.

Slightly more disconcerting: A&J are only displaying a subset of predictions from Rodan’s original paper. Why? My guess is that eye-balling the full dataset doesn’t reveal as much. This calls our for a slightly more rigorous approach than pointing to a few bad predictions. Even better, does someone have the time to crunch the numbers and see if Rodan’s predictions are less useful than predictions being made today?

Famine prediction bleg

Over at The Guardian, Oxfam’s Max Lawson makes this assertion:

…. we have been banging our head against a brick wall from the beginning of the year, warning that an emergency was on its way. We have been talking for months about the complex combination of conflict, entrenched poverty, political marginalisation and the worst drought in 60 years – but commentators haven’t been interested, and governments don’t engage until the TV crews are on the ground and a disaster has been declared.

Really? I’ve had a (cursory) look, and can’t really find  bold predictions from any NGOs about the coming crisis prior to this Reuters report, which itself was based on Famine Early Warning System report two weeks earlier. It seems to me that Oxfam threw most of its clout behind the  GROW campaign this spring. This in itself isn’t a criticism – Oxfam believes that GROW is part of the solution to preventing future crisises – but would they have really chosen to spend several million dollars on a new campaign if they thought a famine was right around the corner?

So readers, this one is up to you: can any of you find an NGO which used the f-word (famine) prior to the release of the FEWS report? Not a general “we’re going to see more famine in the future” but something akin to “We’re going to see famine or something close to famine this year.”

I’m happy to be proven wrong on this one, but I think everyone was equally caught with their pants down.

Update: that was quick, Oxfam is vindicated (although still no f-word).

Food prices and reliable predictions

Trust me kid, you can't bet wrong with my numbers

I’ve been trying to find some time to put down my thoughts about Oxfam’s new campaign (as well as post the contest results, review two books I was meant to review a long time ago, etc). In the meantime, I’m going to piggyback on Marc Bellemare’s thoughts on Oxfam’s claim that food prices will double by 2030:

This means that you can be reasonably confident in a forecast if it’s two or three time periods (e.g., months for monthly data, years for annual data) in the future. For anything beyond five time periods, however, you are in the dark. In other words, the further into the future you try to forecast, the likelier you are to be wrong.

If Oxfam has been using the same monthly food data I have been using — annual data would mask a considerable amount of heterogeneity; in 2010 alone, food prices have increased by 23 percent — they are forecasting food prices about 228 months into the future. Even using annual data, Oxfam forecasting food prices in 19 years makes for what I would charitably call a very heroic forecast. There are about a dozen more reasons why I think Oxfam’s forecast is wrong, which I might get into if there is a demand for it.

But if you truly believe food prices will have doubled by 2030, there’s a derivative contract I might like to sell you…

For more reasons why these predictions might be dubious see a post I wrote a post a while ago on the unreliability of big reports.

Bellemare’s last point is the most important though. If Oxfam is convinced of this result, why not put some money down on it? The benefits would be obvious: Oxfam will have more money to help the poor with, who would then be facing extremely high prices.

Of course, while Oxfam may believe this prediction now, they are trying to change behaviour so it doesn’t come true – this means that the claims will be incredibly difficult to (dis)prove. If food prices happen to double in 20 years, Oxfam will say “Look! We told you so.” If they don’t, they will say “Look at the disaster we averted!” Either way, the winning narrative will be constructed ex-post.