Over at Cheap Talk, Jeff Ely discusses the Reproducibility project, an attempt by a group of psychologists to replicate every study published in 2008 from three journals:
We should do this in economics. But there is a less confrontational way to do it. Top departments in experimental economics attract PhD students who want hands on experience in the lab. These are departments like NYU and CalTech. They would benefit the profession, their students, and the reputation of their PhD programs, i.e. everybody concerned, if they were to add as a requirement that every student receiving a PhD must pick one recently published experimental article and attempt to replicate it.
I agree – and I don’t think this should be limited to lab experiments. Grad school is a great time to set norms for replication and to take advantage of cheap PhD labour. Before getting stuck into their own research, students could be required to replicate a study (preferably one that hasn’t been replicated before), with extra points the more citations that study has. Students should the be required present that replication attempt, with the results subsequently being published somewhere public.
This might put more pressure on researchers to be careful. There have already been a couple of high profile conflicts over replication problems (see the Hoxby-Rothstein dust up and the Acemoglu, Johnson and Robinson cage match with David Albouy). The knowledge that a high-profile results is likely to attract the attention of a grad student with a prerequisite to conquer might make us all a bit more careful.
Such a system would also go a long way to knock down some well-accepted results: when I first started my PhD, the data I first started working on had been used to produce a fairly well known (within the sub-field) result. My supervisor suggested I try to replicate it, and I couldn’t (I’d tell you more, but I didn’t take it to the next level of asking the authors for help).
More more caveat: while replication of lab experiments and econometrics results would be relatively easy, field experiments are probably too expensive and time consuming to be included. All the more reason for us to start funding RCT replications independently.