Many Labs 2: Investigating Variation in Replicability Across Samples and Settings

Research output: Contribution to journalArticle

DOI

Authors

  • Richard A. Klein
  • Michelangelo Vianello
  • Fred Hasselman
  • Byron G. Adams
  • Reginald B. Adams
  • Sinan Alper
  • Mark Aveyard
  • Jordan R. Axt
  • Mayowa T. Babalola
  • Štěpán Bahník
  • Rishtee Batra
  • Mihály Berkics
  • Michael J. Bernstein
  • Daniel R. Berry
  • Olga Bialobrzeska
  • Evans Dami Binan
  • Konrad Bocian
  • Mark J. Brandt
  • Robert Busching
  • Anna Cabak Rédei
  • Huajian Cai
  • Fanny Cambier
  • Katarzyna Cantarero
  • Cheryl L. Carmichael
  • Francisco Ceric
  • Jesse Chandler
  • Jen-ho Chang
  • Armand Chatard
  • Eva E. Chen
  • Winnee Cheong
  • David C. Cicero
  • Sharon Coen
  • Jennifer A. Coleman
  • Brian Collisson
  • Morgan A. Conway
  • Katherine S. Corker
  • Paul G. Curran
  • Fiery Cushman
  • Zubairu K. Dagona
  • Ilker Dalgar
  • Anna Dalla Rosa
  • William E. Davis
  • Maaike De Bruijn
  • Leander De Schutter
  • Thierry Devos
  • Marieke De Vries
  • Canay Doğulu
  • Nerisa Dozo
  • Kristin Nicole Dukes
  • Yarrow Dunham
  • Kevin Durrheim
  • Charles R. Ebersole
  • John E. Edlund
  • Anja Eller
  • Alexander Scott English
  • Carolyn Finck
  • Natalia Frankowska
  • Miguel-ángel Freyre
  • Mike Friedman
  • Elisa Maria Galliani
  • Joshua C. Gandi
  • Tanuka Ghoshal
  • Steffen R. Giessner
  • Tripat Gill
  • Timo Gnambs
  • Ángel Gómez
  • Roberto González
  • Jesse Graham
  • Jon E. Grahe
  • Ivan Grahek
  • Eva G. T. Green
  • Kakul Hai
  • Elizabeth L. Haines
  • Michael P. Hall
  • Marie E. Heffernan
  • Joshua A. Hicks
  • Petr Houdek
  • Jeffrey R. Huntsinger
  • Ho Phi Huynh
  • Hans Ijzerman
  • Yoel Inbar
  • Åse H. Innes-ker
  • William Jiménez-leal
  • Melissa-sue John
  • Jennifer A. Joy-gaba
  • Roza G. Kamiloğlu
  • Heather Barry Kappes
  • Serdar Karabati
  • Haruna Karick
  • Victor N. Keller
  • Anna Kende
  • Nicolas Kervyn
  • Goran Knežević
  • Carrie Kovacs
  • Lacy E. Krueger
  • German Kurapov
  • Jamie Kurtz
  • Daniël Lakens
  • Ljiljana B. Lazarević
  • Carmel A. Levitan
  • Neil A. Lewis
  • Samuel Lins
  • Nikolette P. Lipsey
  • Joy E. Losee
  • Esther Maassen
  • Angela T. Maitner
  • Winfrida Malingumu
  • Robyn K. Mallett
  • Satia A. Marotta
  • Janko Međedović
  • Fernando Mena-pacheco
  • Taciano L. Milfont
  • Wendy L. Morris
  • Sean C. Murphy
  • Koen Neijenhuijs
  • Anthony J. Nelson
  • Félix Neto
  • Austin Lee Nichols
  • Aaron Ocampo
  • Susan L. O’donnell
  • Haruka Oikawa
  • Masanori Oikawa
  • Elsie Ong
  • Gábor Orosz
  • Malgorzata Osowiecka
  • Grant Packard
  • Rolando Pérez-sánchez
  • Boban Petrović
  • Ronaldo Pilati
  • Brad Pinter
  • Lysandra Podesta
  • Gabrielle Pogge
  • Monique M. H. Pollmann
  • Abraham M. Rutchick
  • Patricio Saavedra
  • Alexander K. Saeri
  • Erika Salomon
  • Kathleen Schmidt
  • Felix D. Schönbrodt
  • Maciej B. Sekerdej
  • David Sirlopú
  • Jeanine L. M. Skorinko
  • Vanessa Smith-castro
  • Karin C. H. J. Smolders
  • Agata Sobkow
  • Walter Sowden
  • Philipp Spachtholz
  • Manini Srivastava
  • Troy G. Steiner
  • Jeroen Stouten
  • Chris N. H. Street
  • Oskar K. Sundfelt
  • Stephanie Szeto
  • Ewa Szumowska
  • Andrew C. W. Tang
  • Norbert Tanzer
  • Morgan J. Tear
  • Jordan Theriault
  • Manuela Thomae
  • David Torres
  • Jakub Traczyk
  • Joshua M. Tybur
  • Adrienn Ujhelyi
  • Robbie C. M. Van Aert
  • Marcel A. L. M. Van Assen
  • Marije Van Der Hulst
  • Paul A. M. Van Lange
  • Anna Elisabeth Van ’t Veer
  • Alejandro Vásquez- Echeverría
  • Leigh Ann Vaughn
  • Alexandra Vázquez
  • Luis Diego Vega
  • Catherine Verniers
  • Mark Verschoor
  • Ingrid P. J. Voermans
  • Marek A. Vranka
  • Cheryl Welch
  • Aaron L. Wichman
  • Lisa A. Williams
  • Michael Wood
  • Julie A. Woodzicka
  • Marta K. Wronska
  • Liane Young
  • John M. Zelenski
  • Zeng Zhijia
  • Brian A. Nosek

Departments

External departments

  • Laboratoire Inter-universitaire de Psychologie, Personnalité, Cognition, Changement Social (LIP/PC2S), Université Grenoble Alpes
  • University of Padua
  • Radboud University Nijmegen
  • University of Johannesburg
  • Pennsylvania State University
  • Yasar University
  • American University of Sharjah
  • Duke University
  • United Arab Emirates University
  • University of Economics
  • Saint Joseph’s University
  • Eötvös Loránd University
  • California State University San Marcos
  • SWPS University of Social Sciences and Humanities
  • University of Jos
  • Tilburg University
  • University of Potsdam
  • Lund University
  • CAS - Institute of Psychology
  • Universite Catholique de Louvain
  • City University of New York
  • Universidad del Desarrollo
  • Mathematica Policy Research, Princeton, New Jersey
  • National Taiwan University
  • CNRS Unité Mixte de Recherche 7295, Poitiers, France
  • Hong Kong University of Science and Technology
  • HELP University
  • University of Hawaii at Mānoa
  • University of Salford
  • Virginia Commonwealth University
  • Azusa Pacific University
  • University of Florida
  • Grand Valley State University
  • Harvard University
  • Middle East Technical University
  • Wittenberg University
  • Leadership and Human Resource Management, WHU – Otto Beisheim School of Management
  • San Diego State University
  • Başkent University
  • University of Queensland
  • Office of Institutional Diversity, Allegheny College
  • Yale University
  • University of KwaZulu-Natal
  • University of Virginia
  • Rochester Institute of Technology
  • Facultad de Psicología, Universidad Nacional Autónoma de México
  • Shanghai International Studies University
  • Departamento de Psicología, Universidad de los Andes, Colombia
  • Erasmus University
  • Wilfrid Laurier University
  • Johannes Kepler University Linz
  • Departamento de Psicología Social y de las Organizaciones, Universidad Nacional de Educación a Distancia
  • Escuela de Psicología, Pontificia Universidad Católica de Chile
  • University of Utah
  • Pacific Lutheran University
  • Ghent University
  • University of Lausanne
  • Amity University
  • William Paterson University
  • University of Michigan
  • Smith Child Health Research, Outreach, and Advocacy Center, Ann & Robert H. Lurie Children’s Hospital of Chicago, Chicago, Illinois
  • Texas A&M University
  • Jan Evangelista Purkyne University
  • Loyola University Chicago
  • University of Toronto Scarborough
  • Worcester Polytechnic Institute
  • University of Amsterdam
  • London School of Economics and Political Science
  • Istanbul Bilgi University
  • University of Brasília
  • University of Belgrade
  • Johannes Kepler University
  • James Madison University
  • Eindhoven University of Technology
  • Occidental College
  • Cornell University
  • University of Porto
  • Open University of Tanzania
  • Tufts University
  • Institute of Criminological and Sociological Research, Belgrade, Serbia
  • Universidad Latina de Costa Rica
  • Victoria University of Wellington
  • McDaniel College
  • University of Melbourne
  • VU University Medical Center
  • University of Central Florida
  • George Fox University
  • Doshisha University
  • Open University of Hong Kong
  • Stanford University
  • University of Costa Rica
  • California State University
  • University of Sussex
  • Monash University
  • The University of Chicago
  • Southern Illinois University
  • Ludwig-Maximilians-Universität München
  • Jagiellonian University in Kraków
  • Center for Military Psychiatry and Neuroscience, Walter Reed Army Institute of Research, Silver Spring, Maryland
  • University of Regensburg
  • Occupational & Organisational Psychology and Professional Learning, KU Leuven
  • University of Huddersfield
  • University of Graz
  • Northeastern University
  • University of Winchester
  • Universidad de Iberoamerica
  • Vrije Universiteit Amsterdam
  • Erasmus University Medical Center
  • Leiden University
  • Centro de Investigación Básica en Psicología, Universidad de la República
  • Ithaca College
  • Paris Descartes University
  • University of Groningen
  • Charles University
  • Western Kentucky University
  • University of New South Wales
  • Washington and Lee University
  • Boston College
  • Carleton University
  • Zhejiang University of Finance and Economics
  • Center for Open Science, Charlottesville, Virginia

Details

Original languageEnglish
Pages (from-to)443-490
JournalAdvances in Methods and Practices in Psychological Science
Volume1
Issue number4
DOIs
Publication statusPublished - 1 Dec 2018
Publication type

Research output: Contribution to journalArticle

Abstract

We conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance (p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion (p < .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely high-powered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen’s ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relatively high and low WEIRDness scores, respectively). Cumulatively, variability in the observed effect sizes was attributable more to the effect being studied than to the sample or setting in which it was studied.