Biostatistics Advance Access originally published online on September 12, 2006
Biostatistics 2007 8(2):500-504; doi:10.1093/biostatistics/kxl025
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Published by Oxford University Press 2006.
Multiple comparisons distortions of parameter estimates
MSC 1430, 10 Center Drive, National Institute of Neurological Disorders and Stroke, National Institutes of Health, Bethesda, MD 20892, USA neal.jeffries{at}nih.gov
In experiments involving many variables, investigators typically use multiple comparisons procedures to determine differences that are unlikely to be the result of chance. However, investigators rarely consider how the magnitude of the greatest observed effect sizes may have been subject to bias resulting from multiple testing. These questions of bias become important to the extent investigators focus on the magnitude of the observed effects. As an example, such bias can lead to problems in attempting to validate results, if a biased effect size is used to power a follow-up study. An associated important consequence is that confidence intervals constructed using standard distributions may be badly biased. A bootstrap approach is used to estimate and adjust for the bias in the effect sizes of those variables showing strongest differences. This bias is not always present; some principles showing what factors may lead to greater bias are given and a proof of the convergence of the bootstrap distribution is provided.
Keywords: Bootstrap; Effect size; Multiple comparisons
Received February 3, 2006; revised September 1, 2006; accepted for publication September 8, 2006.