Breznau, Nate ORCID: 0000-0003-4983-3137, Rinke, Eike Mark ORCID: 0000-0002-5330-7634, Wuttke, Alexander ORCID: 0000-0002-9579-5357, Nguyen, Hung H. V. ORCID: 0000-0001-9496-6217, Adem, Muna, Adriaans, Jule ORCID: 0000-0001-7782-505X, Alvarez-Benjumea, Amalia ORCID: 0000-0002-5829-2099, Andersen, Henrik K., Auer, Daniel ORCID: 0000-0003-4454-2365, Azevedo, Flavio ORCID: 0000-0001-9000-8513, Bahnsen, Oke ORCID: 0000-0003-3198-2804, Balzer, Dave, Bauer, Gerrit, Bauer, Paul C., Baumann, Markus, Baute, Sharon, Benoit, Verena ORCID: 0000-0002-8596-9202, Bernauer, Julian, Berning, Carl, Berthold, Anna, Bethke, Felix S., Biegert, Thomas, Blinzler, Katharina, Blumenberg, Johannes N., Bobzien, Licia, Bohman, Andrea, Bol, Thijs, Bostic, Amie, Brzozowska, Zuzanna, Burgdorf, Katharina, Burger, Kaspar ORCID: 0000-0001-5582-7062, Busch, Kathrin B., Carlos-Castillo, Juan, Chan, Nathan, Christmann, Pablo, Connelly, Roxanne, Czymara, Christian S. ORCID: 0000-0002-9535-3559, Damian, Elena, Ecker, Alejandro, Edelmann, Achim, Eger, Maureen A., Ellerbrock, Simon, Forke, Anna, Forster, Andrea, Gaasendam, Chris, Gavras, Konstantin, Gayle, Vernon, Gessler, Theresa, Gnambs, Timo, Godefroidt, Amelie, Groemping, Max, Gross, Martin, Gruber, Stefan, Gummer, Tobias, Hadjar, Andreas, Heisig, Jan Paul, Hellmeier, Sebastian, Heyne, Stefanie, Hirsch, Magdalena, Hjerm, Mikael, Hochman, Oshrat, Hovermann, Andreas, Hunger, Sophia, Hunkler, Christian ORCID: 0000-0002-1632-9834, Huth, Nora, Ignacz, Zsofia S., Jacobs, Laura, Jacobsen, Jannes, Jaeger, Bastian ORCID: 0000-0002-4398-9731, Jungkunz, Sebastian, Jungmann, Nils, Kauff, Mathias, Kleinert, Manuel, Klinger, Julia, Kolb, Jan-Philipp, Kolczynska, Marta, Kuk, John, Kunissen, Katharina, Sinatra, Dafina Kurti, Langenkamp, Alexander, Lersch, Philipp M., Lobel, Lea-Maria, Lutscher, Philipp, Mader, Matthias, Madia, Joan E., Malancu, Natalia, Maldonado, Luis, Marahrens, Helge, Martin, Nicole, Martinez, Paul, Mayerl, Jochen, Mayorga, Oscar J., McManus, Patricia, McWagner, Kyle, Meeusen, Cecil, Meierrieks, Daniel, Mellon, Jonathan, Merhout, Friedolin, Merk, Samuel, Meyer, Daniel, Micheli, Leticia, Mijs, Jonathan, Moya, Cristobal ORCID: 0000-0002-7176-4775, Neunhoeffer, Marcel, Nust, Daniel, Nygard, Olav, Ochsenfeld, Fabian, Otte, Gunnar, Pechenkina, Anna O., Prosser, Christopher, Raes, Louis, Ralston, Kevin, Ramos, Miguel R., Roets, Arne, Rogers, Jonathan, Ropers, Guido, Samuel, Robin, Sand, Gregor, Schachter, Ariela, Schaeffer, Merlin ORCID: 0000-0003-1969-8974, Schieferdecker, David, Schlueter, Elmar, Schmidt, Regine, Schmidt, Katja M., Schmidt-Catran, Alexander, Schmiedeberg, Claudia, Schneider, J. Urgen, Schoonvelde, Martijn, Schulte-Cloos, Julia ORCID: 0000-0001-7223-3602, Schumann, Sandy, Schunck, Reinhard, Schupp, J. Urgen, Seuring, Julian, Silber, Henning, Sleegers, Willem, Sonntag, Nico, Staudt, Alexander, Steiber, Nadia ORCID: 0000-0002-9425-8840, Steiner, Nils, Sternberg, Sebastian, Stiers, Dieter ORCID: 0000-0001-7242-8477, Stojmenovska, Dragana, Storz, Nora, Striessnig, Erich ORCID: 0000-0001-5419-9498, Stroppe, Anne-Kathrin, Teltemann, Janna, Tibajev, Andrey ORCID: 0000-0001-7348-1632, Tung, Brian, Vagni, Giacomo, Van Assche, Jasper, van der Linden, Meta, van der Noll, Jolanda, Van Hootegem, Arno ORCID: 0000-0002-9559-8038, Vogtenhuber, Stefan ORCID: 0000-0003-0839-4481, Voicu, Bogdan, Wagemans, Fieke, Wehl, Nadja, Werner, Hannah ORCID: 0000-0003-4248-5037, Wiernik, Brenton M., Winter, Fabian, Wolf, Christof ORCID: 0000-0002-9364-9524, Yamada, Yuki, Zhang, Nan, Ziller, Conrad, Zins, Stefan and Zoltak, Tomasz (2022). Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty. Proc. Natl. Acad. Sci. U. S. A., 119 (44). WASHINGTON: NATL ACAD SCIENCES. ISSN 1091-6490

Full text not available from this repository.

Abstract

This study explores how researchers' analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers' expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team's workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers' results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.

Item Type: Journal Article
Creators:
CreatorsEmailORCIDORCID Put Code
Breznau, NateUNSPECIFIEDorcid.org/0000-0003-4983-3137UNSPECIFIED
Rinke, Eike MarkUNSPECIFIEDorcid.org/0000-0002-5330-7634UNSPECIFIED
Wuttke, AlexanderUNSPECIFIEDorcid.org/0000-0002-9579-5357UNSPECIFIED
Nguyen, Hung H. V.UNSPECIFIEDorcid.org/0000-0001-9496-6217UNSPECIFIED
Adem, MunaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Adriaans, JuleUNSPECIFIEDorcid.org/0000-0001-7782-505XUNSPECIFIED
Alvarez-Benjumea, AmaliaUNSPECIFIEDorcid.org/0000-0002-5829-2099UNSPECIFIED
Andersen, Henrik K.UNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Auer, DanielUNSPECIFIEDorcid.org/0000-0003-4454-2365UNSPECIFIED
Azevedo, FlavioUNSPECIFIEDorcid.org/0000-0001-9000-8513UNSPECIFIED
Bahnsen, OkeUNSPECIFIEDorcid.org/0000-0003-3198-2804UNSPECIFIED
Balzer, DaveUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Bauer, GerritUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Bauer, Paul C.UNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Baumann, MarkusUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Baute, SharonUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Benoit, VerenaUNSPECIFIEDorcid.org/0000-0002-8596-9202UNSPECIFIED
Bernauer, JulianUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Berning, CarlUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Berthold, AnnaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Bethke, Felix S.UNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Biegert, ThomasUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Blinzler, KatharinaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Blumenberg, Johannes N.UNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Bobzien, LiciaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Bohman, AndreaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Bol, ThijsUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Bostic, AmieUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Brzozowska, ZuzannaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Burgdorf, KatharinaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Burger, KasparUNSPECIFIEDorcid.org/0000-0001-5582-7062UNSPECIFIED
Busch, Kathrin B.UNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Carlos-Castillo, JuanUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Chan, NathanUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Christmann, PabloUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Connelly, RoxanneUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Czymara, Christian S.UNSPECIFIEDorcid.org/0000-0002-9535-3559UNSPECIFIED
Damian, ElenaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Ecker, AlejandroUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Edelmann, AchimUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Eger, Maureen A.UNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Ellerbrock, SimonUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Forke, AnnaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Forster, AndreaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Gaasendam, ChrisUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Gavras, KonstantinUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Gayle, VernonUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Gessler, TheresaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Gnambs, TimoUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Godefroidt, AmelieUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Groemping, MaxUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Gross, MartinUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Gruber, StefanUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Gummer, TobiasUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Hadjar, AndreasUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Heisig, Jan PaulUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Hellmeier, SebastianUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Heyne, StefanieUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Hirsch, MagdalenaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Hjerm, MikaelUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Hochman, OshratUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Hovermann, AndreasUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Hunger, SophiaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Hunkler, ChristianUNSPECIFIEDorcid.org/0000-0002-1632-9834UNSPECIFIED
Huth, NoraUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Ignacz, Zsofia S.UNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Jacobs, LauraUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Jacobsen, JannesUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Jaeger, BastianUNSPECIFIEDorcid.org/0000-0002-4398-9731UNSPECIFIED
Jungkunz, SebastianUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Jungmann, NilsUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Kauff, MathiasUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Kleinert, ManuelUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Klinger, JuliaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Kolb, Jan-PhilippUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Kolczynska, MartaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Kuk, JohnUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Kunissen, KatharinaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Sinatra, Dafina KurtiUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Langenkamp, AlexanderUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Lersch, Philipp M.UNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Lobel, Lea-MariaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Lutscher, PhilippUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Mader, MatthiasUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Madia, Joan E.UNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Malancu, NataliaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Maldonado, LuisUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Marahrens, HelgeUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Martin, NicoleUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Martinez, PaulUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Mayerl, JochenUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Mayorga, Oscar J.UNSPECIFIEDUNSPECIFIEDUNSPECIFIED
McManus, PatriciaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
McWagner, KyleUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Meeusen, CecilUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Meierrieks, DanielUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Mellon, JonathanUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Merhout, FriedolinUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Merk, SamuelUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Meyer, DanielUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Micheli, LeticiaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Mijs, JonathanUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Moya, CristobalUNSPECIFIEDorcid.org/0000-0002-7176-4775UNSPECIFIED
Neunhoeffer, MarcelUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Nust, DanielUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Nygard, OlavUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Ochsenfeld, FabianUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Otte, GunnarUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Pechenkina, Anna O.UNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Prosser, ChristopherUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Raes, LouisUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Ralston, KevinUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Ramos, Miguel R.UNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Roets, ArneUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Rogers, JonathanUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Ropers, GuidoUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Samuel, RobinUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Sand, GregorUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Schachter, ArielaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Schaeffer, MerlinUNSPECIFIEDorcid.org/0000-0003-1969-8974UNSPECIFIED
Schieferdecker, DavidUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Schlueter, ElmarUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Schmidt, RegineUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Schmidt, Katja M.UNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Schmidt-Catran, AlexanderUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Schmiedeberg, ClaudiaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Schneider, J. UrgenUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Schoonvelde, MartijnUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Schulte-Cloos, JuliaUNSPECIFIEDorcid.org/0000-0001-7223-3602UNSPECIFIED
Schumann, SandyUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Schunck, ReinhardUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Schupp, J. UrgenUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Seuring, JulianUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Silber, HenningUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Sleegers, WillemUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Sonntag, NicoUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Staudt, AlexanderUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Steiber, NadiaUNSPECIFIEDorcid.org/0000-0002-9425-8840UNSPECIFIED
Steiner, NilsUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Sternberg, SebastianUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Stiers, DieterUNSPECIFIEDorcid.org/0000-0001-7242-8477UNSPECIFIED
Stojmenovska, DraganaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Storz, NoraUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Striessnig, ErichUNSPECIFIEDorcid.org/0000-0001-5419-9498UNSPECIFIED
Stroppe, Anne-KathrinUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Teltemann, JannaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Tibajev, AndreyUNSPECIFIEDorcid.org/0000-0001-7348-1632UNSPECIFIED
Tung, BrianUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Vagni, GiacomoUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Van Assche, JasperUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
van der Linden, MetaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
van der Noll, JolandaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Van Hootegem, ArnoUNSPECIFIEDorcid.org/0000-0002-9559-8038UNSPECIFIED
Vogtenhuber, StefanUNSPECIFIEDorcid.org/0000-0003-0839-4481UNSPECIFIED
Voicu, BogdanUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Wagemans, FiekeUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Wehl, NadjaUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Werner, HannahUNSPECIFIEDorcid.org/0000-0003-4248-5037UNSPECIFIED
Wiernik, Brenton M.UNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Winter, FabianUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Wolf, ChristofUNSPECIFIEDorcid.org/0000-0002-9364-9524UNSPECIFIED
Yamada, YukiUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Zhang, NanUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Ziller, ConradUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Zins, StefanUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Zoltak, TomaszUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
URN: urn:nbn:de:hbz:38-699555
DOI: 10.1073/pnas.2203150119
Journal or Publication Title: Proc. Natl. Acad. Sci. U. S. A.
Volume: 119
Number: 44
Date: 2022
Publisher: NATL ACAD SCIENCES
Place of Publication: WASHINGTON
ISSN: 1091-6490
Language: English
Faculty: Unspecified
Divisions: Unspecified
Subjects: no entry
Uncontrolled Keywords:
KeywordsLanguage
WELFARE-STATE; IMMIGRATION; SUPPORT; REDISTRIBUTION; PREFERENCES; ANALYSTS; IDEASMultiple languages
Multidisciplinary SciencesMultiple languages
URI: http://kups.ub.uni-koeln.de/id/eprint/69955

Downloads

Downloads per month over past year

Altmetric

Export

Actions (login required)

View Item View Item