Статья

Many Labs 2: Investigating Variation in Replicability Across Samples and Settings

Richard KleinLaboratoire Inter-universitaire de Psychologie, Personnalité, Cognition, Changement Social (LIP/PC2S), Université Grenoble AlpesMichelangelo VianelloDepartment of Philosophy, Sociology, Education and Applied Psychology, University of PaduaFred HasselmanBehavioural Science Institute, Radboud University NijmegenByron G. AdamsDepartment of Industrial Psychology and People Management, University of JohannesburgReginald B. AdamsDepartment of Psychology, The Pennsylvania State UniversitySinan AlperDepartment of Psychology, Yasar UniversityMark AveyardDepartment of International Studies, American University of SharjahJordan AxtCenter for Advanced Hindsight, Duke UniversityMayowa T. BabalolaCollege of Business and Economics, United Arab Emirates UniversityŠtěpán BahníkDepartment of Management, Faculty of Business Administration, University of Economics, PragueRishtee BatraErivan K. Haub School of Business, Saint Joseph’s UniversityMihály BerkicsInstitute of Psychology, ELTE Eötvös Loránd UniversityMichael J. BernsteinPsychological and Social Sciences Program, Pennsylvania State University AbingtonDaniel R. BerryDepartment of Psychology, California State University San MarcosOlga BiałobrzeskaWarsaw Faculty of Psychology, SWPS University of Social Sciences and HumanitiesEvans Dami BinanDepartment of General and Applied Psychology, University of JosKonrad BocianSopot Faculty of Psychology, SWPS University of Social Sciences and HumanitiesMark J. BrandtDepartment of Social Psychology, Tilburg UniversityRobert BuschingDepartment of Psychology, University of PotsdamAnna Cabak RédeiCentre for Languages and Literature, Lund UniversityHuajian CaiInstitute of Psychology, Chinese Academy of SciencesFanny CambierCenter on Consumers and Marketing Strategy (CCMS), Université catholique de LouvainKatarzyna CantareroSocial Behavior Research Centre, Wroclaw Faculty of Psychology, SWPS University of Social Sciences and HumanitiesCheryl L. CarmichaelDepartment of Psychology, Brooklyn College & Graduate Center, CUNYFrancisco CerićCentro de Apego y Regulacion Emocional, Universidad del DesarrolloJesse ChandlerInstitute for Social Research, University of MichiganJen‐Ho ChangDepartment of Psychology, National Taiwan UniversityArmand ChatardCNRS Unité Mixte de Recherche 7295, Poitiers, FranceEva E. ChenDivision of Social Science, The Hong Kong University of Science and TechnologyWinnee CheongDepartment of Psychology, HELP UniversityDavid C. CiceroDepartment of Psychology, University of Hawaii at ManoaSharon CoenDirectorate of Psychology and Public Health, University of SalfordJennifer A. ColemanDepartment of Psychology, Virginia Commonwealth UniversityBrian CollissonDepartment of Psychology, Azusa Pacific UniversityMorgan ConwayDepartment of Psychology, University of FloridaKatherine S. CorkerDepartment of Psychology, Grand Valley State UniversityPaul CurranDepartment of Psychology, Grand Valley State UniversityFiery CushmanDepartment of Psychology, Harvard UniversityZubairu Kwambo DagonaDepartment of General and Applied Psychology, University of Josİlker DalḡarDepartment of Psychology, Middle East Technical UniversityAnna Dalla RosaDepartment of Philosophy, Sociology, Education and Applied Psychology, University of PaduaWilliam E. DavisDepartment of Psychology, Wittenberg UniversityMaaike de BruijnDepartment of Social Psychology, Tilburg UniversityLeander De SchutterLeadership and Human Resource Management, WHU – Otto Beisheim School of ManagementThierry DevosDepartment of Psychology, San Diego State UniversityMarieke de VriesBehavioural Science Institute, Radboud University NijmegenCanay DoğuluDepartment of Psychology, Başkent UniversityNerisa DozoSchool of Psychology, The University of QueenslandKristin Nicole DukesOffice of Institutional Diversity, Allegheny CollegeYarrow DunhamDepartment of Psychology, Yale UniversityKevin DurrheimSchool of Applied Human Sciences, University of KwaZulu-NatalCharles R. EbersoleDepartment of Psychology, University of VirginiaJohn E. EdlundDepartment of Psychology, Rochester Institute of TechnologyAnja EllerFacultad de Psicología, Universidad Nacional Autónoma de MéxicoAlexander Scott EnglishShanghai Intercultural Institute, Shanghai International Studies UniversityCarolyn FinckDepartamento de Psicología, Universidad de los Andes, ColombiaNatalia FrankowskaWarsaw Faculty of Psychology, SWPS University of Social Sciences and HumanitiesMiguel-Ángel FreyreFacultad de Psicología, Universidad Nacional Autónoma de MéxicoMike FriedmanCenter on Consumers and Marketing Strategy (CCMS), Université catholique de LouvainElisa Maria GallianiDepartment of Political and Juridical Sciences and International Studies, University of PaduaJoshua C. GandiDepartment of General and Applied Psychology, University of JosTanuka GhoshalDepartment of Marketing and International Business, Baruch College, CUNYSteffen R. GiessnerDepartment of Organisation and Personnel Management, Rotterdam School of Management, Erasmus UniversityTripat GillLazaridis School of Business and Economics, Wilfrid Laurier UniversityTimo GnambsEducational Measurement, Leibniz Institute for Educational Trajectories, Bamberg, GermanyÁngel GómezDepartamento de Psicología Social y de las Organizaciones, Universidad Nacional de Educación a DistanciaRoberto GonzálezEscuela de Psicología, Pontificia Universidad Católica de ChileJesse GrahamEccles School of Business, University of UtahJon GrahePsychology, Pacific Lutheran UniversityIvan GrahekDepartment of Experimental Clinical and Health Psychology, Ghent UniversityEva G. T. GreenInstitute of Psychology, Faculty of Social and Political Sciences, University of LausanneKakul HaiAmity Institute of Psychology and Allied Sciences, Amity UniversityMatthew HaighDepartment of Psychology, Northumbria UniversityElizabeth L. HainesDepartment of Psychology, William Paterson UniversityMichael P. HallDepartment of Psychology, University of MichiganMarie E. HeffernanSmith Child Health Research, Outreach, and Advocacy Center, Ann & Robert H. Lurie Children’s Hospital of Chicago, Chicago, IllinoisJoshua A. HicksDepartment of Psychological & Brain Sciences, Texas A&M UniversityPetr HoudekJeffrey R. HuntsingerDepartment of Psychology, Loyola University ChicagoHo Phi HuynhDepartment of Science and Mathematics, Texas A&M University-San AntonioHans IJzermanLaboratoire Inter-universitaire de Psychologie, Personnalité, Cognition, Changement Social (LIP/PC2S), Université Grenoble AlpesYoel InbarDepartment of Psychology, University of Toronto ScarboroughÅse Innes-KerDepartment of Psychology, Lund UniversityWilliam Jiménez‐LealDepartamento de Psicología, Universidad de los Andes, ColombiaMelissa-Sue JohnDepartment of Social Science and Policy Studies, Worcester Polytechnic InstituteJennifer A. Joy-GabaDepartment of Psychology, Virginia Commonwealth UniversityRoza Gizem KamilogluDepartment of Psychology, University of AmsterdamHeather Barry KappesDepartment of Management, London School of Economics and Political ScienceSerdar KarabatıDepartment of Business Administration, Istanbul Bilgi UniversityHaruna KarickDepartment of General and Applied Psychology, University of JosVictor N. KellerDepartment of Social and Work Psychology, Institute of Psychology, University of BrasiliaAnna KendeDepartment of Social Psychology, ELTE Eötvös Loránd UniversityNicolas KervynCenter on Consumers and Marketing Strategy (CCMS), Université catholique de LouvainGoran KneževićDepartment of Psychology, Faculty of Philosophy, University of BelgradeCarrie KovacsDepartment of Work, Organizational and Media Psychology, Johannes Kepler University LinzLacy E. KruegerDepartment of Psychology & Special Education, Texas A&M University-CommerceGerman KurapovInternational Victimology Institute Tilburg, Tilburg UniversityJamie KurtzDepartment of Psychology, James Madison UniversityDaniël LakensSchool of Innovation Science, Eindhoven University of TechnologyLjiljana B. LazarevićInstitute of Psychology, Faculty of Philosophy, University of Belgrade

2018en

ABI

Аннотация

We conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance ( p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion ( p < .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely high-powered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen’s ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relatively high and low WEIRDness scores, respectively). Cumulatively, variability in the observed effect sizes was attributable more to the effect being studied than to the sample or setting in which it was studied.

Перевод пока недоступен

Идентификаторы

DOI: 10.1177/2515245918810225

Цитирования и источники

Цитирований: 2Использованных источников: 0

Показатели — AkademScholar