Statistical Conclusion Validity: Measurement in Economic Experiments

Chapter 4 — Statistical Conclusion Validity: Measurement in Economic Experiments Recovering a causal effect is only half the battle—measuring it precisely and drawing valid inferences is the other. This chapter covers the superpopulation and finite-population frameworks for estimating average treatment effects, single and multiple hypothesis testing, and the family-wise error rate. It introduces the difference-in-means and difference-in-differences estimators, Fisher Randomization Inference, and three corrections for multiple testing: Bonferroni, Holm, and the List, Shaikh, and Xu approach.

Key Ideas

1. Causal inference is challenging because the same unit cannot be observed in more than one state of the world simultaneously.
2. Experimentation is a vital tool used to solve the causal inference problem, allowing us to lend insights into two Experimental Problems.
3. Experimental Problem 1 (EP1) pertains to measuring the causal impact of treatments and determining relevant mediators and moderators in an ethically responsible manner.
4. Experimental Problem 2 (EP2) relates to whether the causal impacts of treatments implemented in one environment transfer to other environments, be them spatially, temporally, or scale differentiated.