The unsafe dependence of meta-analyses in behavioral medicine on Failsafe N

Citation

Ozakinci G, Heene M & Coyne J (2014) The unsafe dependence of meta-analyses in behavioral medicine on Failsafe N. International Congress of Behavioural Medicine (ICBM 2014), Brainerd, Minnesota. International Journal of Behavioral Medicine, 21 (Supplement 1), pp. S121-S122. https://doi.org/10.1007/s12529-014-9418-2

Abstract
Meta-analyses of health-behavior change and psychosocial interventions uniformly conclude that they are effective and ready for dissemination. Concerns about publication bias and the reliance on small, underpowered studies of only low to medium quality are frequently waived off with calculations of failsafe N most associated with Robert Rosenthal. Such calculations estimate the many null studies lying in desk drawers that would be needed to unseat the conclusions of the meta-analyses. They consistently exceed the number of available studies many times over. However, outside of psychology, the failsafe N of Rosenthal and others has fallen into disfavor. The Cochrane collaboration and many epidemiologists recommend against its use. The Rosenthal’s (Rosenthal & Rubin, 1978) Fail-Safe-Number (FSN) aimed to estimate the number of unpublished studies in meta-analyses required to bring the meta-analytic mean effect size down to a statistically insignificant level is widely-used in order to determine the stability of meta-analytic results against potential publication bias threats. However, the FSN is invalid because it treats the file drawer as unbiased and almost always severely overestimates the number of unpublished studies (Elsahoff, 1978; Iyengar & Greenhouse, 1988).

Journal
International Journal of Behavioral Medicine: Volume 21, Issue Supplement 1