Is social research really not better than alchemy?

Organised by

• Department of Political and Social Sciences

Starting with Silberzahn and Uhlmann (2015), many-analysts studies have become a popular design to examine the robustness and credibility of scientific research. We argue that the current design of many-analysts studies tends to exaggerate the unreliability of science. This is shown by reassessing the finding of a recent, widely discussed many-analysts study done by Breznau, Rinke, Wuttke et al. [Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty; PNAS 119 (2022)]. This study found that for unidentifiable ("hidden") reasons, the many analysts' results varied widely even when testing the same social science hypothesis (that immigration reduces social policy support) with the same data.

In our re-analysis we found several pitfalls, including re-scaling errors and overlooked effect size moderators. The general take-away from our investigation is that for producing valid results on research certainty, many-analysts projects must adhere to some meta-analytical guidelines. We also discuss some alternative methods to assess the credibility of social science research.