Examining the generalizability of research findings from archival data

Generalizability Tests Forecasting Collaboration; Andrew Delios; Elena Giulia Clemente; Tao Wu; Hongbin Tan; Yong Wang; Michael Gordon; Domenico Viganola; Zhaowei Chen; Anna Dreber; Magnus Johannesson; Thomas Pfeiffer; Eric Luis Uhlmann; Ahmad M.Abd Al-Aziz; Ajay T. Abraham; Jais Trojan; Matus Adamkovic; Elena Agadullina; Jungsoo Ahn; Cinla Akinci; Handan Akkas; David Albrecht; Shilaan Alzahawi; Marcio Amaral-Baptista; Rahul Anand; Kevin Francis U. Ang; Frederik Anseel; John Jamir Benzon R. Aruta; Mujeeba Ashraf; Bradley J. Baker; Xueqi Bao; Ernest Baskin; Hanoku Bathula; Christopher W. Bauman; Jozef Bavolar; Secil Bayraktar; Stephanie E. Beckman; Aaron S. Benjamin; Stephanie E.V. Brown; Jeffrey Buckley; Ricardo E. Buitrago; Jefferson L. Bution; Nick Byrd; Clara Carrera; Eugene M. Caruso; Minxia Chen; Lin Chen; Eyyub Ensari Cicerali; Eric D. Cohen; Marcus Crede; Amanda Rotella

doi:10.1073/pnas.2120377119

Examining the generalizability of research findings from archival data

Generalizability Tests Forecasting Collaboration, Andrew Delios^*, Elena Giulia Clemente, Tao Wu, Hongbin Tan, Yong Wang, Michael Gordon, Domenico Viganola, Zhaowei Chen, Anna Dreber, Magnus Johannesson, Thomas Pfeiffer, Eric Luis Uhlmann^*, Ahmad M.Abd Al-Aziz, Ajay T. Abraham, Jais Trojan, Matus Adamkovic, Elena Agadullina, Jungsoo Ahn, Cinla AkinciHandan Akkas, David Albrecht, Shilaan Alzahawi, Marcio Amaral-Baptista, Rahul Anand, Kevin Francis U. Ang, Frederik Anseel, John Jamir Benzon R. Aruta, Mujeeba Ashraf, Bradley J. Baker, Xueqi Bao, Ernest Baskin, Hanoku Bathula, Christopher W. Bauman, Jozef Bavolar, Secil Bayraktar, Stephanie E. Beckman, Aaron S. Benjamin, Stephanie E.V. Brown, Jeffrey Buckley, Ricardo E. Buitrago, Jefferson L. Bution, Nick Byrd, Clara Carrera, Eugene M. Caruso, Minxia Chen, Lin Chen, Eyyub Ensari Cicerali, Eric D. Cohen, Marcus Crede, Amanda Rotella

^*Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

17 Citations (Scopus)

19 Downloads (Pure)

Abstract

This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability-for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.

Original language	English
Article number	e2120377119
Number of pages	9
Journal	Proceedings of the National Academy of Sciences of the United States of America
Volume	119
Issue number	30
DOIs	https://doi.org/10.1073/pnas.2120377119
Publication status	Published - 26 Jul 2022
Externally published	Yes

Keywords

archival data
context sensitivity
generalizability
reproducibility
research reliability

Access to Document

10.1073/pnas.2120377119Licence: CC BY-NC-ND

delios-et-al-2022-examining-the-generalizability-of-research-findings-from-archival-dataFinal published version, 843 KBLicence: CC BY-NC-ND

Cite this

Generalizability Tests Forecasting Collaboration, Delios, A., Clemente, E. G., Wu, T., Tan, H., Wang, Y., Gordon, M., Viganola, D., Chen, Z., Dreber, A., Johannesson, M., Pfeiffer, T., Uhlmann, E. L., Al-Aziz, A. M. A., Abraham, A. T., Trojan, J., Adamkovic, M., Agadullina, E., Ahn, J., ... Rotella, A. (2022). Examining the generalizability of research findings from archival data. Proceedings of the National Academy of Sciences of the United States of America, 119(30), Article e2120377119. https://doi.org/10.1073/pnas.2120377119

@article{0635d7916eb6451fa61a9933beec7f77,

title = "Examining the generalizability of research findings from archival data",

abstract = "This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability-for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.",

keywords = "archival data, context sensitivity, generalizability, reproducibility, research reliability",

author = "{Generalizability Tests Forecasting Collaboration} and Andrew Delios and Clemente, {Elena Giulia} and Tao Wu and Hongbin Tan and Yong Wang and Michael Gordon and Domenico Viganola and Zhaowei Chen and Anna Dreber and Magnus Johannesson and Thomas Pfeiffer and Uhlmann, {Eric Luis} and Al-Aziz, {Ahmad M.Abd} and Abraham, {Ajay T.} and Jais Trojan and Matus Adamkovic and Elena Agadullina and Jungsoo Ahn and Cinla Akinci and Handan Akkas and David Albrecht and Shilaan Alzahawi and Marcio Amaral-Baptista and Rahul Anand and Ang, {Kevin Francis U.} and Frederik Anseel and Aruta, {John Jamir Benzon R.} and Mujeeba Ashraf and Baker, {Bradley J.} and Xueqi Bao and Ernest Baskin and Hanoku Bathula and Bauman, {Christopher W.} and Jozef Bavolar and Secil Bayraktar and Beckman, {Stephanie E.} and Benjamin, {Aaron S.} and Brown, {Stephanie E.V.} and Jeffrey Buckley and Buitrago, {Ricardo E.} and Bution, {Jefferson L.} and Nick Byrd and Clara Carrera and Caruso, {Eugene M.} and Minxia Chen and Lin Chen and Cicerali, {Eyyub Ensari} and Cohen, {Eric D.} and Marcus Crede and Amanda Rotella",

note = "Funding information: This research project benefitted from Ministry of Education (Singapore) Tier 1 Grant R-313-000-131-115 (to A. Delios), National Science Foundation of China Grants 72002158 (to H.T.) and 71810107002 (to H.T.), grants from the Knut and Alice Wallenberg Foundation (to A. Dreber) and the Marianne and Marcus Wallenberg Foundation (through a Wallenberg Scholar grant; to A. Dreber), Austrian Science Fund (FWF) Grant SFB F63 (to A. Dreber), grants from the Jan Wallander and Tom Hedelius Foundation (Svenska Handelsbankens Forskningsstiftelser; to A. Dreber), and an Research & Development (R&D) research grant from Institut Europ{\'e}en d{\textquoteright}Administration des Affaires (INSEAD) (to E.L.U.). Dmitrii Dubrov, of the G.T.F.C., was supported by the National Research University Higher School of Economics (HSE University) Basic Research Program.",

year = "2022",

month = jul,

day = "26",

doi = "10.1073/pnas.2120377119",

language = "English",

volume = "119",

journal = "Proceedings of the National Academy of Sciences of the United States of America",

issn = "0027-8424",

publisher = "National Academy of Sciences",

number = "30",

}

Generalizability Tests Forecasting Collaboration, Delios, A, Clemente, EG, Wu, T, Tan, H, Wang, Y, Gordon, M, Viganola, D, Chen, Z, Dreber, A, Johannesson, M, Pfeiffer, T, Uhlmann, EL, Al-Aziz, AMA, Abraham, AT, Trojan, J, Adamkovic, M, Agadullina, E, Ahn, J, Akinci, C, Akkas, H, Albrecht, D, Alzahawi, S, Amaral-Baptista, M, Anand, R, Ang, KFU, Anseel, F, Aruta, JJBR, Ashraf, M, Baker, BJ, Bao, X, Baskin, E, Bathula, H, Bauman, CW, Bavolar, J, Bayraktar, S, Beckman, SE, Benjamin, AS, Brown, SEV, Buckley, J, Buitrago, RE, Bution, JL, Byrd, N, Carrera, C, Caruso, EM, Chen, M, Chen, L, Cicerali, EE, Cohen, ED, Crede, M & Rotella, A 2022, 'Examining the generalizability of research findings from archival data', Proceedings of the National Academy of Sciences of the United States of America, vol. 119, no. 30, e2120377119. https://doi.org/10.1073/pnas.2120377119

TY - JOUR

T1 - Examining the generalizability of research findings from archival data

AU - Generalizability Tests Forecasting Collaboration

AU - Delios, Andrew

AU - Clemente, Elena Giulia

AU - Wu, Tao

AU - Tan, Hongbin

AU - Wang, Yong

AU - Gordon, Michael

AU - Viganola, Domenico

AU - Chen, Zhaowei

AU - Dreber, Anna

AU - Johannesson, Magnus

AU - Pfeiffer, Thomas

AU - Uhlmann, Eric Luis

AU - Al-Aziz, Ahmad M.Abd

AU - Abraham, Ajay T.

AU - Trojan, Jais

AU - Adamkovic, Matus

AU - Agadullina, Elena

AU - Ahn, Jungsoo

AU - Akinci, Cinla

AU - Akkas, Handan

AU - Albrecht, David

AU - Alzahawi, Shilaan

AU - Amaral-Baptista, Marcio

AU - Anand, Rahul

AU - Ang, Kevin Francis U.

AU - Anseel, Frederik

AU - Aruta, John Jamir Benzon R.

AU - Ashraf, Mujeeba

AU - Baker, Bradley J.

AU - Bao, Xueqi

AU - Baskin, Ernest

AU - Bathula, Hanoku

AU - Bauman, Christopher W.

AU - Bavolar, Jozef

AU - Bayraktar, Secil

AU - Beckman, Stephanie E.

AU - Benjamin, Aaron S.

AU - Brown, Stephanie E.V.

AU - Buckley, Jeffrey

AU - Buitrago, Ricardo E.

AU - Bution, Jefferson L.

AU - Byrd, Nick

AU - Carrera, Clara

AU - Caruso, Eugene M.

AU - Chen, Minxia

AU - Chen, Lin

AU - Cicerali, Eyyub Ensari

AU - Cohen, Eric D.

AU - Crede, Marcus

AU - Rotella, Amanda

N1 - Funding information: This research project benefitted from Ministry of Education (Singapore) Tier 1 Grant R-313-000-131-115 (to A. Delios), National Science Foundation of China Grants 72002158 (to H.T.) and 71810107002 (to H.T.), grants from the Knut and Alice Wallenberg Foundation (to A. Dreber) and the Marianne and Marcus Wallenberg Foundation (through a Wallenberg Scholar grant; to A. Dreber), Austrian Science Fund (FWF) Grant SFB F63 (to A. Dreber), grants from the Jan Wallander and Tom Hedelius Foundation (Svenska Handelsbankens Forskningsstiftelser; to A. Dreber), and an Research & Development (R&D) research grant from Institut Européen d’Administration des Affaires (INSEAD) (to E.L.U.). Dmitrii Dubrov, of the G.T.F.C., was supported by the National Research University Higher School of Economics (HSE University) Basic Research Program.

PY - 2022/7/26

Y1 - 2022/7/26

N2 - This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability-for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.

AB - This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability-for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.

KW - archival data

KW - context sensitivity

KW - generalizability

KW - reproducibility

KW - research reliability

UR - http://www.scopus.com/inward/record.url?scp=85134491120&partnerID=8YFLogxK

U2 - 10.1073/pnas.2120377119

DO - 10.1073/pnas.2120377119

M3 - Article

C2 - 35858443

AN - SCOPUS:85134491120

SN - 0027-8424

VL - 119

JO - Proceedings of the National Academy of Sciences of the United States of America

JF - Proceedings of the National Academy of Sciences of the United States of America

IS - 30

M1 - e2120377119

ER -

Examining the generalizability of research findings from archival data

Abstract

Keywords

Access to Document

Other files and links

Cite this