Building Confidence in (and Knowledge from) Experimental Results

Chapter 15 — Building Confidence in (and Knowledge from) Experimental Results A single finding is a data point. A body of reliable, replicated evidence is scientific knowledge. This chapter examines the post-study probability that a research finding is true, the role of priors, significance, and power in updating beliefs, and how the experimental approach—through selective data generation and replication—is uniquely positioned to build a cumulative stock of knowledge. It covers the spectrum of replication types (pure, same-population, similar-population, conceptual), pre-registration, registered reports, and pre-analysis plans as tools for promoting transparency and reducing publication bias.

1. Scientific research aims to create a stock of knowledge. Optimally adding to this stock requires confidence in the received estimates to realize EP1.
2. A key question concerning confidence revolves around the following query: After a research finding has been claimed, what is the post-study probability that it is true?
3. Two unique features of the experimental approach situate it well to deepen the stock of scientific knowledge: selective data generation and the ability to enhance the notion, and role, of replications.
4. Combining these two features, the analyst can maximize inferential power across the experimental spectrum and efficiently update knowledge.