Optimal Experimental Design

Chapter 5 — Optimal Experimental Design Good experiments don’t just happen—they are engineered. This chapter provides a rigorous framework for calculating optimal sample sizes, trading off statistical power against cost, and making design choices that maximize the likelihood of detecting true effects. Topics include the minimum detectable effect, binary and continuous outcomes, heterogeneity in participant costs, clustered designs, factorial designs, and how pre-treatment covariates and measurement choices can sharpen inference without adding observations.

1. Statistical power provides an indication of the likelihood of detecting a true effect when there is one.
2. There are several approaches to improving statistical power of an experiment, from initial design to final data analysis.
3. Sample size, variation in treatment, cost per observation, treatment response variability, covariates collected pre-treatment, and the modeling of data all play a key role in optimal design.
4. When the unit of randomization is different from the unit of observation, clustering is an important consideration.