Insights into the accuracy of social scientists’ forecasts of societal change

Igor Grossmann; Amanda Rotella; Cendri A. Hutcherson; Konstantyn Sharpinskyi; Michael E. W. Varnum; Sebastian Achter; Mandeep K. Dhami; Xinqi Evie Guo; Mane Kara-Yakoubian; David R. Mandel; Louis Raes; Louis Tay; Aymeric Vie; Lisa Wagner; Matus Adamkovic; Arash Arami; Patricia Arriaga; Kasun Bandara; Gabriel Banik; František Bartoš; Ernest Baskin; Christoph Bergmeir; Michal Bialek; Caroline K. Børsting; Dillon T. Browne; Eugene M. Caruso; Rong Chen; Bin-Tzong Chie; William J. Chopik; Robert N. Collins; Chin W. Cong; Lucian G. Conway; Matthew Davis; Martin V. Day; Nathan A. Dhaliwal; Justin D. Durham; Martyna Dziekan; Christian T. Elbaek; Eric Shuman; Marharyta Fabrykant; Mustafa Firat; Geoffrey T. Fong; Jeremy A. Frimer; Jonathan M. Gallegos; Simon B. Goldberg; Anton Gollwitzer; Julia Goyal; Lorenz Graf-Vlachy; Scott D. Gronlund; Sebastian Hafenbrädl; Andree Hartanto; Matthew J. Hirshberg; Matthew J. Hornsey; Piers D. L. Howe; Anoosha Izadi; Bastian Jaeger; Pavol Kacmár; Yeun Joon Kim; Ruslan Krenzler; Daniel G. Lannin; Hung-Wen Lin; Nigel Mantou Lou; Verity Y. Q. Lua; Aaron W. Lukaszewski; Albert L. Ly; Christopher R. Madan; Maximilian Maier; Nadyanna M. Majeed; David S. March; Abigail A. Marsh; Michal Misiak; Kristian Ove R. Myrseth; Jaime M. Napan; Jonathan Nicholas; Konstantinos Nikolopoulos; Jiaqing O; Tobias Otterbring; Mariola Paruzel-Czachura; Shiva Pauer; John Protzko; Quentin Raffaelli; Ivan Ropovik; Robert M. Ross; Yefim Roth; Espen Roysamb; Landon Schnabel; Astrid Schütz; Matthias Seifert; A. Timur Sevincer; Garrick T. Sherman; Otto Simonsson; Ming-Chien Sung; Chung-Ching Tai; Thomas Talhelm; Bethany A. Teachman; Philip E. Tetlock; Dimitrios Thomakos; Dwight C. K. Tse; Oliver J. Twardus; Joshua M. Tybur; Lyle Ungar; Daan Vandermeulen; Leighton Vaughan Williams; Hrag A. Vosgerichian; Qi Wang; Ke Wang; Mark E. Whiting; Conny E. Wollbrant; Tao Yang; Kumar Yogeeswaran; Sangsuk Yoon; Ventura r. Alves; Jessica R. Andrews-Hanna; Paul A. Bloom; Anthony Boyles; Loo Charis; Mingyeong Choi; Sean Darling-Hammond; Zoe E. Ferguson; Cheryl R. Kaiser; Simon T. Karg; Alberto López Ortega; Lori Mahoney; Melvin S. Marsh; Marcellin F. R. C. Martinie; Eli K. Michaels; Philip Millroth; Jeanean B. Naqvi; Weiting Ng; Robb B. Rutledge; Peter Slattery; Adam H. Smiley; Oliver Strijbis; Daniel Sznycer; Eli Tsukayama; Austin van Loon; Jan G. Voelkel; Margaux N. A. Wienk; Tom Wilkening; The Forecasting Collaborative

doi:10.1038/s41562-022-01517-1

Insights into the accuracy of social scientists’ forecasts of societal change

Igor Grossmann^*, Amanda Rotella, Cendri A. Hutcherson, Konstantyn Sharpinskyi, Michael E. W. Varnum, Sebastian Achter, Mandeep K. Dhami, Xinqi Evie Guo, Mane Kara-Yakoubian, David R. Mandel, Louis Raes, Louis Tay, Aymeric Vie, Lisa Wagner, Matus Adamkovic, Arash Arami, Patricia Arriaga, Kasun Bandara, Gabriel Banik, František BartošErnest Baskin, Christoph Bergmeir, Michal Bialek, Caroline K. Børsting, Dillon T. Browne, Eugene M. Caruso, Rong Chen, Bin-Tzong Chie, William J. Chopik, Robert N. Collins, Chin W. Cong, Lucian G. Conway, Matthew Davis, Martin V. Day, Nathan A. Dhaliwal, Justin D. Durham, Martyna Dziekan, Christian T. Elbaek, Eric Shuman, Marharyta Fabrykant, Mustafa Firat, Geoffrey T. Fong, Jeremy A. Frimer, Jonathan M. Gallegos, Simon B. Goldberg, Anton Gollwitzer, Julia Goyal, Lorenz Graf-Vlachy, Scott D. Gronlund, Sebastian Hafenbrädl, Andree Hartanto, Matthew J. Hirshberg, Matthew J. Hornsey, Piers D. L. Howe, Anoosha Izadi, Bastian Jaeger, Pavol Kacmár, Yeun Joon Kim, Ruslan Krenzler, Daniel G. Lannin, Hung-Wen Lin, Nigel Mantou Lou, Verity Y. Q. Lua, Aaron W. Lukaszewski, Albert L. Ly, Christopher R. Madan, Maximilian Maier, Nadyanna M. Majeed, David S. March, Abigail A. Marsh, Michal Misiak, Kristian Ove R. Myrseth, Jaime M. Napan, Jonathan Nicholas, Konstantinos Nikolopoulos, Jiaqing O, Tobias Otterbring, Mariola Paruzel-Czachura, Shiva Pauer, John Protzko, Quentin Raffaelli, Ivan Ropovik, Robert M. Ross, Yefim Roth, Espen Roysamb, Landon Schnabel, Astrid Schütz, Matthias Seifert, A. Timur Sevincer, Garrick T. Sherman, Otto Simonsson, Ming-Chien Sung, Chung-Ching Tai, Thomas Talhelm, Bethany A. Teachman, Philip E. Tetlock, Dimitrios Thomakos, Dwight C. K. Tse, Oliver J. Twardus, Joshua M. Tybur, Lyle Ungar, Daan Vandermeulen, Leighton Vaughan Williams, Hrag A. Vosgerichian, Qi Wang, Ke Wang, Mark E. Whiting, Conny E. Wollbrant, Tao Yang, Kumar Yogeeswaran, Sangsuk Yoon, Ventura r. Alves, Jessica R. Andrews-Hanna, Paul A. Bloom, Anthony Boyles, Loo Charis, Mingyeong Choi, Sean Darling-Hammond, Zoe E. Ferguson, Cheryl R. Kaiser, Simon T. Karg, Alberto López Ortega, Lori Mahoney, Melvin S. Marsh, Marcellin F. R. C. Martinie, Eli K. Michaels, Philip Millroth, Jeanean B. Naqvi, Weiting Ng, Robb B. Rutledge, Peter Slattery, Adam H. Smiley, Oliver Strijbis, Daniel Sznycer, Eli Tsukayama, Austin van Loon, Jan G. Voelkel, Margaux N. A. Wienk, Tom Wilkening, The Forecasting Collaborative

^*Corresponding author for this work

Psychology

Research output: Contribution to journal › Article › peer-review

3 Citations (Scopus)

Abstract

How well can social scientists predict societal change, and what processes underlie their predictions? To answer these questions, we ran two forecasting tournaments testing accuracy of predictions of societal change in domains commonly studied in the social sciences: ideological preferences, political polarization, life satisfaction, sentiment on social media, and gender-career and racial bias. Following provision of historical trend data on the domain, social scientists submitted pre-registered monthly forecasts for a year (Tournament 1; N=86 teams/359 forecasts), with an opportunity to update forecasts based on new data six months later (Tournament 2; N=120 teams/546 forecasts). Benchmarking forecasting accuracy revealed that social scientists? forecasts were on average no more accurate than simple statistical models (historical means, random walk, or linear regressions) or the aggregate forecasts of a sample from the general public (N=802). However, scientists were more accurate if they had scientific expertise in a prediction domain, were interdisciplinary, used simpler models, and based predictions on prior data.

Original language	English
Pages (from-to)	484-501
Number of pages	18
Journal	Nature Human Behaviour
Volume	7
Issue number	4
Early online date	9 Feb 2023
DOIs	https://doi.org/10.1038/s41562-022-01517-1
Publication status	Published - 1 Apr 2023

Access to Document

10.1038/s41562-022-01517-1

Cite this

Grossmann, I., Rotella, A., Hutcherson, C. A., Sharpinskyi, K., Varnum, M. E. W., Achter, S., Dhami, M. K., Guo, X. E., Kara-Yakoubian, M., Mandel, D. R., Raes, L., Tay, L., Vie, A., Wagner, L., Adamkovic, M., Arami, A., Arriaga, P., Bandara, K., Banik, G., ... Collaborative, T. F. (2023). Insights into the accuracy of social scientists’ forecasts of societal change. Nature Human Behaviour, 7(4), 484-501. https://doi.org/10.1038/s41562-022-01517-1

@article{3730cf369a6f4d14be5352467fce0b6d,

title = "Insights into the accuracy of social scientists{\textquoteright} forecasts of societal change",

abstract = "How well can social scientists predict societal change, and what processes underlie their predictions? To answer these questions, we ran two forecasting tournaments testing accuracy of predictions of societal change in domains commonly studied in the social sciences: ideological preferences, political polarization, life satisfaction, sentiment on social media, and gender-career and racial bias. Following provision of historical trend data on the domain, social scientists submitted pre-registered monthly forecasts for a year (Tournament 1; N=86 teams/359 forecasts), with an opportunity to update forecasts based on new data six months later (Tournament 2; N=120 teams/546 forecasts). Benchmarking forecasting accuracy revealed that social scientists? forecasts were on average no more accurate than simple statistical models (historical means, random walk, or linear regressions) or the aggregate forecasts of a sample from the general public (N=802). However, scientists were more accurate if they had scientific expertise in a prediction domain, were interdisciplinary, used simpler models, and based predictions on prior data.",

keywords = "meta-science, forecasting, expert judgement, political polarization, prejudice, well-being",

author = "Igor Grossmann and Amanda Rotella and Hutcherson, {Cendri A.} and Konstantyn Sharpinskyi and Varnum, {Michael E. W.} and Sebastian Achter and Dhami, {Mandeep K.} and Guo, {Xinqi Evie} and Mane Kara-Yakoubian and Mandel, {David R.} and Louis Raes and Louis Tay and Aymeric Vie and Lisa Wagner and Matus Adamkovic and Arash Arami and Patricia Arriaga and Kasun Bandara and Gabriel Banik and Franti{\v s}ek Barto{\v s} and Ernest Baskin and Christoph Bergmeir and Michal Bialek and B{\o}rsting, {Caroline K.} and Browne, {Dillon T.} and Caruso, {Eugene M.} and Rong Chen and Bin-Tzong Chie and Chopik, {William J.} and Collins, {Robert N.} and Cong, {Chin W.} and Conway, {Lucian G.} and Matthew Davis and Day, {Martin V.} and Dhaliwal, {Nathan A.} and Durham, {Justin D.} and Martyna Dziekan and Elbaek, {Christian T.} and Eric Shuman and Marharyta Fabrykant and Mustafa Firat and Fong, {Geoffrey T.} and Frimer, {Jeremy A.} and Gallegos, {Jonathan M.} and Goldberg, {Simon B.} and Anton Gollwitzer and Julia Goyal and Lorenz Graf-Vlachy and Gronlund, {Scott D.} and Sebastian Hafenbr{\"a}dl and Andree Hartanto and Hirshberg, {Matthew J.} and Hornsey, {Matthew J.} and Howe, {Piers D. L.} and Anoosha Izadi and Bastian Jaeger and Pavol Kacm{\'a}r and Kim, {Yeun Joon} and Ruslan Krenzler and Lannin, {Daniel G.} and Hung-Wen Lin and Lou, {Nigel Mantou} and Lua, {Verity Y. Q.} and Lukaszewski, {Aaron W.} and Ly, {Albert L.} and Madan, {Christopher R.} and Maximilian Maier and Majeed, {Nadyanna M.} and March, {David S.} and Marsh, {Abigail A.} and Michal Misiak and Myrseth, {Kristian Ove R.} and Napan, {Jaime M.} and Jonathan Nicholas and Konstantinos Nikolopoulos and Jiaqing O and Tobias Otterbring and Mariola Paruzel-Czachura and Shiva Pauer and John Protzko and Quentin Raffaelli and Ivan Ropovik and Ross, {Robert M.} and Yefim Roth and Espen Roysamb and Landon Schnabel and Astrid Sch{\"u}tz and Matthias Seifert and Sevincer, {A. Timur} and Sherman, {Garrick T.} and Otto Simonsson and Ming-Chien Sung and Chung-Ching Tai and Thomas Talhelm and Teachman, {Bethany A.} and Tetlock, {Philip E.} and Dimitrios Thomakos and Tse, {Dwight C. K.} and Twardus, {Oliver J.} and Tybur, {Joshua M.} and Lyle Ungar and Daan Vandermeulen and Williams, {Leighton Vaughan} and Vosgerichian, {Hrag A.} and Qi Wang and Ke Wang and Whiting, {Mark E.} and Wollbrant, {Conny E.} and Tao Yang and Kumar Yogeeswaran and Sangsuk Yoon and Alves, {Ventura r.} and Andrews-Hanna, {Jessica R.} and Bloom, {Paul A.} and Anthony Boyles and Loo Charis and Mingyeong Choi and Sean Darling-Hammond and Ferguson, {Zoe E.} and Kaiser, {Cheryl R.} and Karg, {Simon T.} and Ortega, {Alberto L{\'o}pez} and Lori Mahoney and Marsh, {Melvin S.} and Martinie, {Marcellin F. R. C.} and Michaels, {Eli K.} and Philip Millroth and Naqvi, {Jeanean B.} and Weiting Ng and Rutledge, {Robb B.} and Peter Slattery and Smiley, {Adam H.} and Oliver Strijbis and Daniel Sznycer and Eli Tsukayama and Loon, {Austin van} and Voelkel, {Jan G.} and Wienk, {Margaux N. A.} and Tom Wilkening and Collaborative, {The Forecasting}",

note = "Funding information: This program of research was supported by Basic Research Program at the National Research University Higher School of Economics (M. Fabrykant), John Templeton Foundation grant 62260 (I.G. and P.T.), Kega 079UK-4/2021 (P.K.), National Center for Complementary & Integrative Health of the National Institutes of Health under Award Number K23AT010879 (Simon B. Goldberg), National Science Foundation RAPID Grant 2026854 (M.E.W.V.), PID2019-111512RB-I00 (M.S.), NPO Systemic Risk Institute (LX22NPO5101) (I.R.), Slovak Research and Development Agency under contract no. APVV-20-0319 (M.A.), Social Sciences and Humanities Research Council of Canada Insight Grant 435-2014-0685 (I.G.), Social Sciences and Humanities Research Council of Canada Connection Grant 611-2020-0190 (I.G.), Swiss National Science Foundation grant PP00P1_170463 (O. Strijbis).",

year = "2023",

month = apr,

day = "1",

doi = "10.1038/s41562-022-01517-1",

language = "English",

volume = "7",

pages = "484--501",

journal = "Nature Human Behaviour",

issn = "2397-3374",

publisher = "Nature Publishing Group",

number = "4",

}

Grossmann, I, Rotella, A, Hutcherson, CA, Sharpinskyi, K, Varnum, MEW, Achter, S, Dhami, MK, Guo, XE, Kara-Yakoubian, M, Mandel, DR, Raes, L, Tay, L, Vie, A, Wagner, L, Adamkovic, M, Arami, A, Arriaga, P, Bandara, K, Banik, G, Bartoš, F, Baskin, E, Bergmeir, C, Bialek, M, Børsting, CK, Browne, DT, Caruso, EM, Chen, R, Chie, B-T, Chopik, WJ, Collins, RN, Cong, CW, Conway, LG, Davis, M, Day, MV, Dhaliwal, NA, Durham, JD, Dziekan, M, Elbaek, CT, Shuman, E, Fabrykant, M, Firat, M, Fong, GT, Frimer, JA, Gallegos, JM, Goldberg, SB, Gollwitzer, A, Goyal, J, Graf-Vlachy, L, Gronlund, SD, Hafenbrädl, S, Hartanto, A, Hirshberg, MJ, Hornsey, MJ, Howe, PDL, Izadi, A, Jaeger, B, Kacmár, P, Kim, YJ, Krenzler, R, Lannin, DG, Lin, H-W, Lou, NM, Lua, VYQ, Lukaszewski, AW, Ly, AL, Madan, CR, Maier, M, Majeed, NM, March, DS, Marsh, AA, Misiak, M, Myrseth, KOR, Napan, JM, Nicholas, J, Nikolopoulos, K, O, J, Otterbring, T, Paruzel-Czachura, M, Pauer, S, Protzko, J, Raffaelli, Q, Ropovik, I, Ross, RM, Roth, Y, Roysamb, E, Schnabel, L, Schütz, A, Seifert, M, Sevincer, AT, Sherman, GT, Simonsson, O, Sung, M-C, Tai, C-C, Talhelm, T, Teachman, BA, Tetlock, PE, Thomakos, D, Tse, DCK, Twardus, OJ, Tybur, JM, Ungar, L, Vandermeulen, D, Williams, LV, Vosgerichian, HA, Wang, Q, Wang, K, Whiting, ME, Wollbrant, CE, Yang, T, Yogeeswaran, K, Yoon, S, Alves, VR, Andrews-Hanna, JR, Bloom, PA, Boyles, A, Charis, L, Choi, M, Darling-Hammond, S, Ferguson, ZE, Kaiser, CR, Karg, ST, Ortega, AL, Mahoney, L, Marsh, MS, Martinie, MFRC, Michaels, EK, Millroth, P, Naqvi, JB, Ng, W, Rutledge, RB, Slattery, P, Smiley, AH, Strijbis, O, Sznycer, D, Tsukayama, E, Loon, AV, Voelkel, JG, Wienk, MNA, Wilkening, T & Collaborative, TF 2023, 'Insights into the accuracy of social scientists’ forecasts of societal change', Nature Human Behaviour, vol. 7, no. 4, pp. 484-501. https://doi.org/10.1038/s41562-022-01517-1

TY - JOUR

T1 - Insights into the accuracy of social scientists’ forecasts of societal change

AU - Grossmann, Igor

AU - Rotella, Amanda

AU - Hutcherson, Cendri A.

AU - Sharpinskyi, Konstantyn

AU - Varnum, Michael E. W.

AU - Achter, Sebastian

AU - Dhami, Mandeep K.

AU - Guo, Xinqi Evie

AU - Kara-Yakoubian, Mane

AU - Mandel, David R.

AU - Raes, Louis

AU - Tay, Louis

AU - Vie, Aymeric

AU - Wagner, Lisa

AU - Adamkovic, Matus

AU - Arami, Arash

AU - Arriaga, Patricia

AU - Bandara, Kasun

AU - Banik, Gabriel

AU - Bartoš, František

AU - Baskin, Ernest

AU - Bergmeir, Christoph

AU - Bialek, Michal

AU - Børsting, Caroline K.

AU - Browne, Dillon T.

AU - Caruso, Eugene M.

AU - Chen, Rong

AU - Chie, Bin-Tzong

AU - Chopik, William J.

AU - Collins, Robert N.

AU - Cong, Chin W.

AU - Conway, Lucian G.

AU - Davis, Matthew

AU - Day, Martin V.

AU - Dhaliwal, Nathan A.

AU - Durham, Justin D.

AU - Dziekan, Martyna

AU - Elbaek, Christian T.

AU - Shuman, Eric

AU - Fabrykant, Marharyta

AU - Firat, Mustafa

AU - Fong, Geoffrey T.

AU - Frimer, Jeremy A.

AU - Gallegos, Jonathan M.

AU - Goldberg, Simon B.

AU - Gollwitzer, Anton

AU - Goyal, Julia

AU - Graf-Vlachy, Lorenz

AU - Gronlund, Scott D.

AU - Hafenbrädl, Sebastian

AU - Hartanto, Andree

AU - Hirshberg, Matthew J.

AU - Hornsey, Matthew J.

AU - Howe, Piers D. L.

AU - Izadi, Anoosha

AU - Jaeger, Bastian

AU - Kacmár, Pavol

AU - Kim, Yeun Joon

AU - Krenzler, Ruslan

AU - Lannin, Daniel G.

AU - Lin, Hung-Wen

AU - Lou, Nigel Mantou

AU - Lua, Verity Y. Q.

AU - Lukaszewski, Aaron W.

AU - Ly, Albert L.

AU - Madan, Christopher R.

AU - Maier, Maximilian

AU - Majeed, Nadyanna M.

AU - March, David S.

AU - Marsh, Abigail A.

AU - Misiak, Michal

AU - Myrseth, Kristian Ove R.

AU - Napan, Jaime M.

AU - Nicholas, Jonathan

AU - Nikolopoulos, Konstantinos

AU - O, Jiaqing

AU - Otterbring, Tobias

AU - Paruzel-Czachura, Mariola

AU - Pauer, Shiva

AU - Protzko, John

AU - Raffaelli, Quentin

AU - Ropovik, Ivan

AU - Ross, Robert M.

AU - Roth, Yefim

AU - Roysamb, Espen

AU - Schnabel, Landon

AU - Schütz, Astrid

AU - Seifert, Matthias

AU - Sevincer, A. Timur

AU - Sherman, Garrick T.

AU - Simonsson, Otto

AU - Sung, Ming-Chien

AU - Tai, Chung-Ching

AU - Talhelm, Thomas

AU - Teachman, Bethany A.

AU - Tetlock, Philip E.

AU - Thomakos, Dimitrios

AU - Tse, Dwight C. K.

AU - Twardus, Oliver J.

AU - Tybur, Joshua M.

AU - Ungar, Lyle

AU - Vandermeulen, Daan

AU - Williams, Leighton Vaughan

AU - Vosgerichian, Hrag A.

AU - Wang, Qi

AU - Wang, Ke

AU - Whiting, Mark E.

AU - Wollbrant, Conny E.

AU - Yang, Tao

AU - Yogeeswaran, Kumar

AU - Yoon, Sangsuk

AU - Alves, Ventura r.

AU - Andrews-Hanna, Jessica R.

AU - Bloom, Paul A.

AU - Boyles, Anthony

AU - Charis, Loo

AU - Choi, Mingyeong

AU - Darling-Hammond, Sean

AU - Ferguson, Zoe E.

AU - Kaiser, Cheryl R.

AU - Karg, Simon T.

AU - Ortega, Alberto López

AU - Mahoney, Lori

AU - Marsh, Melvin S.

AU - Martinie, Marcellin F. R. C.

AU - Michaels, Eli K.

AU - Millroth, Philip

AU - Naqvi, Jeanean B.

AU - Ng, Weiting

AU - Rutledge, Robb B.

AU - Slattery, Peter

AU - Smiley, Adam H.

AU - Strijbis, Oliver

AU - Sznycer, Daniel

AU - Tsukayama, Eli

AU - Loon, Austin van

AU - Voelkel, Jan G.

AU - Wienk, Margaux N. A.

AU - Wilkening, Tom

AU - Collaborative, The Forecasting

N1 - Funding information: This program of research was supported by Basic Research Program at the National Research University Higher School of Economics (M. Fabrykant), John Templeton Foundation grant 62260 (I.G. and P.T.), Kega 079UK-4/2021 (P.K.), National Center for Complementary & Integrative Health of the National Institutes of Health under Award Number K23AT010879 (Simon B. Goldberg), National Science Foundation RAPID Grant 2026854 (M.E.W.V.), PID2019-111512RB-I00 (M.S.), NPO Systemic Risk Institute (LX22NPO5101) (I.R.), Slovak Research and Development Agency under contract no. APVV-20-0319 (M.A.), Social Sciences and Humanities Research Council of Canada Insight Grant 435-2014-0685 (I.G.), Social Sciences and Humanities Research Council of Canada Connection Grant 611-2020-0190 (I.G.), Swiss National Science Foundation grant PP00P1_170463 (O. Strijbis).

PY - 2023/4/1

Y1 - 2023/4/1

N2 - How well can social scientists predict societal change, and what processes underlie their predictions? To answer these questions, we ran two forecasting tournaments testing accuracy of predictions of societal change in domains commonly studied in the social sciences: ideological preferences, political polarization, life satisfaction, sentiment on social media, and gender-career and racial bias. Following provision of historical trend data on the domain, social scientists submitted pre-registered monthly forecasts for a year (Tournament 1; N=86 teams/359 forecasts), with an opportunity to update forecasts based on new data six months later (Tournament 2; N=120 teams/546 forecasts). Benchmarking forecasting accuracy revealed that social scientists? forecasts were on average no more accurate than simple statistical models (historical means, random walk, or linear regressions) or the aggregate forecasts of a sample from the general public (N=802). However, scientists were more accurate if they had scientific expertise in a prediction domain, were interdisciplinary, used simpler models, and based predictions on prior data.

AB - How well can social scientists predict societal change, and what processes underlie their predictions? To answer these questions, we ran two forecasting tournaments testing accuracy of predictions of societal change in domains commonly studied in the social sciences: ideological preferences, political polarization, life satisfaction, sentiment on social media, and gender-career and racial bias. Following provision of historical trend data on the domain, social scientists submitted pre-registered monthly forecasts for a year (Tournament 1; N=86 teams/359 forecasts), with an opportunity to update forecasts based on new data six months later (Tournament 2; N=120 teams/546 forecasts). Benchmarking forecasting accuracy revealed that social scientists? forecasts were on average no more accurate than simple statistical models (historical means, random walk, or linear regressions) or the aggregate forecasts of a sample from the general public (N=802). However, scientists were more accurate if they had scientific expertise in a prediction domain, were interdisciplinary, used simpler models, and based predictions on prior data.

KW - meta-science

KW - forecasting

KW - expert judgement

KW - political polarization

KW - prejudice

KW - well-being

UR - http://www.scopus.com/inward/record.url?scp=85147648888&partnerID=8YFLogxK

U2 - 10.1038/s41562-022-01517-1

DO - 10.1038/s41562-022-01517-1

M3 - Article

VL - 7

SP - 484

EP - 501

JO - Nature Human Behaviour

JF - Nature Human Behaviour

SN - 2397-3374

IS - 4

ER -

Insights into the accuracy of social scientists’ forecasts of societal change

Abstract

Access to Document

Other files and links

Embargoed Document

Fingerprint

Cite this