In which your lab experiment might not be entirely representative


Self-selection can be a bitch when you want a representative sample

A new paper by Elaine Liu, Paul Frijters and Tao Kong:

We compare the characteristics and regression coefficients between the participants in a field experiment in China and the survey population from which they were recruited. The experimental participants were more educated, younger, more likely to be male, more risk-loving and work fewer hours than the more general population. The estimates of their regression coefficients in the standard analyses of wages, happiness and entrepreneurship differed significantly from non-participants, indicating that inferences drawn from experimental samples may not hold for more representative groups of the population.

Lab experiments have always caught flak for non-representative participant groups, which more often than not still comprise Western university students. The rise of lab experiments in the field, where populations more suitable for the context of the study are targeted, led to somewhat robust claims of external validity. Maybe these claims were hasty.

Hat tip to Andres Marroquin.

Comments are closed.