The statistical significance filter leads to overoptimistic expectations of replicability

Keywords

Filter (signal processing), Econometrics, Computer science, Artificial intelligence, Mathematics

References

• An Activation-Based Model of Sentence Processing as Skilled Memory RetrievalDennis B. Downey — 2005
• The dependency locality theory: A distance-based theory of linguistic complexity.Edward Gibson — 2000
• Doing Bayesian Data Analysis: A Tutorial with R, JAGS, and StanJohn K. Kruschke — 2014
• Das Digitale Wörterbuch der Deutschen Sprache (DWDS)Wolfgang Klein, Alexander Geyken — 2010
• Bayesian Adaptive Methods for Clinical TrialsScott Berry, Bradley P. Carlin, J. Jack Lee — 2010
• Argument-Head Distance and Processing Complexity: Explaining both Locality and Antilocality EffectsShravan Vasishth, Richard L. Lewis — 2006
• Parsimonious Mixed ModelsDouglas M. Bates, Reinhold Kliegl, Shravan Vasishth — 2015
• Locality and Parsing ComplexityLars Konieczny — 2000
• Uncertainty and Expectation in Sentence Processing: Evidence From Subcategorization DistributionsTal Linzen, T. Florian Jaeger — 2016
• Estimating the reproducibility of psychological scienceAlexander A. Aarts — 2015
• Fitting Linear Mixed-Effects Models Using lme4Douglas M. Bates, Martin Mächler, Benjamin M. Bolker — 2014
• The behavior of the P-value when the alternative hypothesis is true.Hung Hung, Robert T. O’Neill, Péter Bauer — 1997
• Processing Chinese Relative Clauses: Evidence for the Subject-Relative AdvantageShravan Vasishth, Zhong Chen, Li Qiang — 2013
• Reading processes during syntactic analysis and reanalysisFernanda Ferreira, John M. Henderson — 1993
• Bayesian Approaches to Randomized TrialsDavid J. Spiegelhalter, Laurence S. Freedman, Mahesh Parmar — 1994
• The subject-relative advantage in Chinese: Evidence for expectation-based processingLena A. Jäger, Zhong Chen, Li Qiang — 2015
• Making and Correcting Errors during Sentence Comprehension: Eye Movements in the Analysis of Structurally Ambiguous SentencesLyn Frazier, Keith Rayner — 1982
• Estimating effect size: Bias resulting from the significance criterion in editorial decisionsDavid M. Lane, William P. Dunlap — 1978
• The meaning of "significance" for different types of research [translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas]. 1969.Adriaan D. De Groot — 2014
• Sequential Trials, Sequential Analysis and the Likelihood PrincipleJerome Cornfield — 1966

Cited by

• Conditional equivalence testing: An alternative remedy for publication biasHarlan Campbell, Paul Gustafson — 2017
• Using meta-analysis for evidence synthesis: The case of incomplete neutralization in GermanBruno Nicenboim, Timo B. Roettger, Shravan Vasishth — 2018
• The Effect of Prominence and Cue Association on Retrieval Processes: A Computational AccountFelix Engelmann, Lena A. Jäger, Shravan Vasishth — 2019
• Large-scale replication study reveals a limit on probabilistic prediction in language comprehensionMante S. Nieuwland, Stephen Politzer‐Ahles, Evelien Heyselaar — 2018
• Bayesian data analysis in the phonetic sciences: A tutorial introductionShravan Vasishth, Bruno Nicenboim, Mary E. Beckman — 2018
• A Theory of Timing Effects in a Self-Organizing Model of Sentence ProcessingGarrett Smith — 2018
• Emergent data analysis in phonetic sciences: Towards pluralism and reproducibilityTimo B. Roettger, Bodo Winter, R. Harald Baayen — 2019
• Researcher degrees of freedom in phonetic researchTimo B. Roettger — 2019
• Reading Shakespearean tropes in a foreign tongue: Age of L2 acquisition modulates neural responses to functional shiftsMartina G. Vilas, Micaela Santilli, Ezequiel Mikulan — 2019
• The posterior probability of a null hypothesis given a statistically significant resultDaniel J. Schad, Shravan Vasishth — 2019
• The grammaticality asymmetry in agreement attraction reflects response bias: Experimental and modeling evidence.Christopher Hammerly, Adrian Staub, Brian Dillon — 2019
• Social Smartphone Apps Do Not Capture Attention Despite Their Perceived High Reward ValueNiklas Johannes, Jonas Dora, Dorottya Rusz — 2018
• The effects of high versus low talker variability and individual aptitude on phonetic training of Mandarin lexical tonesHanyu Dong, Meghan Clayards, Helen Brown — 2018
• Exact replication: Foundation of science or game of chance?Sophie K. Piper, Ulrike Grittner, André Rex — 2019
• Toward a principled Bayesian workflow in cognitive science.Daniel J. Schad, Michael Betancourt, Shravan Vasishth — 2019
• Attachment and Concord of Temporal Adverbs: Evidence From Eye MovementsNicoletta Biondo, Francesco Vespignani, Brian Dillon — 2019
• A large-scale study of the effects of word frequency and predictability in naturalistic readingCory Shain — 2019
• Biases in fiscal multiplier estimatesZareh Asatryan, Annika Havlik, Friedrich Heinemann — 2020
• Are confidence intervals better termed “uncertainty intervals”?Andrew Gelman, Sander Greenland — 2019
• Capitalization interacts with syntactic complexity.Michael G. Cutter, Andrea E. Martin, Patrick Sturt — 2020

Related papers

• Revisiting the Gelman–Rubin DiagnosticDootika Vats, Christina Knudson — 2018
• Hypothesis Probability or Statistical SignificanceEmilia I. De la Fuente, Gustavo R. Cañadas, Joan Guàrdia‐Olmos — 2009
• Bayesian Inference and ComputationChristian P. Robert, Jean‐Michel Marin, Judith Rousseau — 2011
• Replication, p rep, and confidence intervals: comment prompted by Iverson, Wagenmakers, and Lee (2010); Lecoutre, Lecoutre, and Poitevineau (2010); and Maraun and Gabriel (2010).Geoff Cumming — 2010