Trust me kid, you can't bet wrong with my numbers
I’ve been trying to find some time to put down my thoughts about Oxfam’s new campaign (as well as post the contest results, review two books I was meant to review a long time ago, etc). In the meantime, I’m going to piggyback on Marc Bellemare’s thoughts on Oxfam’s claim that food prices will double by 2030:
This means that you can be reasonably confident in a forecast if it’s two or three time periods (e.g., months for monthly data, years for annual data) in the future. For anything beyond five time periods, however, you are in the dark. In other words, the further into the future you try to forecast, the likelier you are to be wrong.
If Oxfam has been using the same monthly food data I have been using — annual data would mask a considerable amount of heterogeneity; in 2010 alone, food prices have increased by 23 percent — they are forecasting food prices about 228 months into the future. Even using annual data, Oxfam forecasting food prices in 19 years makes for what I would charitably call a very heroic forecast. There are about a dozen more reasons why I think Oxfam’s forecast is wrong, which I might get into if there is a demand for it.
But if you truly believe food prices will have doubled by 2030, there’s a derivative contract I might like to sell you…
For more reasons why these predictions might be dubious see a post I wrote a post a while ago on the unreliability of big reports.
Bellemare’s last point is the most important though. If Oxfam is convinced of this result, why not put some money down on it? The benefits would be obvious: Oxfam will have more money to help the poor with, who would then be facing extremely high prices.
Of course, while Oxfam may believe this prediction now, they are trying to change behaviour so it doesn’t come true – this means that the claims will be incredibly difficult to (dis)prove. If food prices happen to double in 20 years, Oxfam will say “Look! We told you so.” If they don’t, they will say “Look at the disaster we averted!” Either way, the winning narrative will be constructed ex-post.
#1 by Marc on June 2, 2011 - 3:30 pm
Thanks for the link, Matt.
That said, there is a book about the foolishness of making such predictions that came out recently: “Future Babble,” by the Ottawa Citizen’s Dan Gardner. I have not yet had a chance to read it (and I don’t know if it’s out in the UK yet), but I had really enjoyed his previous book, which was about how our societies perceives and deals with risk.
#2 by terence on June 4, 2011 - 1:41 am
Hi Matt,
Interesting post, but wrong on a number of accounts I think.
First, the argument advanced by you and Marc Bellemare appears to rest on very similar logic to that which underpins the claims of people who state that ‘we can’t even predict the weather 2 weeks from now, how can those scientists be so confident that the globe will be warmer 20 years into the future?’ Accurately predicting short and mid-term fluctuations in food prices is likely considerably harder than the task of predicting long run trends, based on fundamentals such as demand exceeding supply, and from this deducing that food will be more expensive in 30 years time.
This isn’t the same as saying Oxfam are definitely right, but rather that you two are comparing apples with oranges here to an extent.
Second, you argue that Oxfam’s claims aren’t falsifiable, but they are. If food prices don’t rise, any reasonable attempt at process tracing making use of historical data (as it will then be) will ought to be able to determine whether food prices didn’t rise because campaigning led to changed behaviours or because something else took place. A more likely escape route for the campaigners of today is that 30 years from now very few people will remember all those old reports or YouTube videos…
#3 by Matt on June 4, 2011 - 10:56 am
Hi Terence,
Marc might be better served to respond, given that he spends a lot more time studying food prices, but here we go.
“First, the argument advanced by you and Marc Bellemare appears to rest on very similar logic to that which underpins the claims of people who state that ‘we can’t even predict the weather 2 weeks from now, how can those scientists be so confident that the globe will be warmer 20 years into the future?’ ”
Uh, no – that’s not the logic I’m using – you’re building a straw man here. The fundamentals behind global warming, greenhouse gases, are generally less difficult to predict, even if the system they operate in (the global climate) is incredibly complex. I wouldn’t put any money on the specific claims of climate change predictions, other than the temperature is going up – our uncertainty about climate change should probably scare us more than our certainty – we don’t really know how bad it can be.
If you look at Oxfam’s report, there are many, many, many different factors going into these predictions – income growth, agricultural productivity, estimates of demand for different commodities, etc, etc, etc. These are being done on a country, by country basis. Each could be wrong, and each gets less precise as you project into the future. Throw these together, and you have an incredibly, incredibly noisy estimate. Oxfam knows this, which is why their estimate has a huge confidence interval – but they don’t report the confidence intervals to the news, they always report the means.
This doesn’t mean that we shouldn’t be worried – but it does mean we shouldn’t make shoddy claims about how bad it will be.
“Second, you argue that Oxfam’s claims aren’t falsifiable, but they are. If food prices don’t rise, any reasonable attempt at process tracing making use of historical data (as it will then be) will ought to be able to determine whether food prices didn’t rise because campaigning led to changed behaviours or because something else took place.”
Good luck with that – unless Oxfam’s campaign has an obvious, immediate effect on something tangible which feeds into food prices (i.e., basically a structural break in prices), there’s little chance that any empirical tests are going to come up with much here. Even if you find that some fundamentals change slowly over time (and where’s the counterfactual in all this? That’s right, in an alternative universe) – you’ve got to first connect these fundamentals to Oxfam’s campaign, then connect the fundamentals to changes in food prices.
Sure, you might be able to use historical evidence to say “we have a pretty good idea why food prices have gone up”, but you’re never going to be ale to say something as precise as “Food prices are only half of what Oxfam predicted BECAUSE of Oxfam’s campaign”.
#4 by terence on June 4, 2011 - 11:07 pm
Hi Matt,
Thanks, that’s a good reply. I take your point regarding the methods for predicting food prices. And tip my hat to both you and Marc for having taken the time to read the actual report. The one quibble I’d have is that, if the confidence intervals are large then, as with climate change, uncertainty is hardly a cause for complacency.
With regards to our second point of contention. Here I still definitely disagree. It’s true, using the methods I suggest we’d never be able to say, “Food prices are only half of what Oxfam predicted BECAUSE of Oxfam’s campaign”, but through qualitative work involving process-tracing we ought to be able to say with a reasonable degree of confidence whether the campaign worked and through what avenues. Such work is never going to produce results as definitive as the best RCT might. But these methods are all we’ve got for answering some types of important question. And they’re better than nothing by quite some margin.