Heterogeneity of Research Results: A New Perspective From Which to Assess and Promote Progress in Psychological Science

Audrey Linden, Johannes Honekopp

Research output: Contribution to journalArticlepeer-review

25 Downloads (Pure)

Abstract

Heterogeneity emerges when multiple close or conceptual replications on the same subject produce results that vary more than expected from the sampling error. Here we argue that unexplained heterogeneity reflects a lack of coherence between the concepts applied and data observed and therefore a lack of understanding of the subject matter. Typical levels of heterogeneity thus offer a useful but neglected perspective on the levels of understanding achieved in psychological science. Focusing on continuous outcome variables, we surveyed heterogeneity in 150 meta-analyses from cognitive, organizational, and social psychology and 57 multiple close replications. Heterogeneity proved to be very high in meta-analyses, with powerful moderators being conspicuously absent. Population effects in the average meta-analysis vary from small to very large for reasons that are typically not understood. In contrast, heterogeneity was moderate in close replications. A newly identified relationship between heterogeneity and effect size allowed us to make predictions about expected heterogeneity levels. We discuss important implications for the formulation and evaluation of theories in psychology. On the basis of insights from the history and philosophy of science, we argue that the reduction of heterogeneity is important for progress in psychology and its practical applications, and we suggest changes to our collective research practice toward this end.
Original languageEnglish
Pages (from-to)358-376
Number of pages19
JournalPerspectives on Psychological Science
Volume16
Issue number2
Early online date5 Jan 2021
DOIs
Publication statusPublished - 1 Mar 2021

Fingerprint Dive into the research topics of 'Heterogeneity of Research Results: A New Perspective From Which to Assess and Promote Progress in Psychological Science'. Together they form a unique fingerprint.

Cite this