Biostatistics Advance Access originally published online on February 21, 2009
Biostatistics 2009 10(3):424-435; doi:10.1093/biostatistics/kxp001
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
A simulation–approximation approach to sample size planning for high-dimensional classification studies
Department of Environmental Science, Policy, & Management, University of California, 137 Hilgard Hall No. 3114, Berkeley, CA 94720-3114, USA
pdevalpine{at}berkeley.edu
Roche Palo Alto, 3431 Hillview Avenue, Palo Alto, CA 94304, USA
XDx Expression Diagnostics, 3260 Bayshore Boulevard, Brisbane, CA 94005, USA
Predicant Biosciences, 2 N. First Street, San Jose, CA 95113, USA
* To whom correspondence should be addressed.
Classification studies with high-dimensional measurements and relatively small sample sizes are increasingly common. Prospective analysis of the role of sample sizes in the performance of such studies is important for study design and interpretation of results, but the complexity of typical pattern discovery methods makes this problem challenging. The approach developed here combines Monte Carlo methods and new approximations for linear discriminant analysis, assuming multivariate normal distributions. Monte Carlo methods are used to sample the distribution of which features are selected for a classifier and the mean and variance of features given that they are selected. Given selected features, the linear discriminant problem involves different distributions of training data and generalization data, for which 2 approximations are compared: one based on Taylor series approximation of the generalization error and the other on approximating the discriminant scores as normally distributed. Combining the Monte Carlo and approximation approaches to different aspects of the problem allows efficient estimation of expected generalization error without full simulations of the entire sampling and analysis process. To evaluate the method and investigate realistic study design questions, full simulations are used to ask how validation error rate depends on the strength and number of informative features, the number of noninformative features, the sample size, and the number of features allowed into the pattern. Both approximation methods perform well for most cases but only the normal discriminant score approximation performs well for cases of very many weakly informative or uninformative dimensions. The simulated cases show that many realistic study designs will typically estimate substantially suboptimal patterns and may have low probability of statistically significant validation results.
Keywords: Biomarker discovery; Experimental design; Generalization error; Genomic; Pattern recognition; Proteomic
Received May 29, 2007; revised August 19, 2008; revised November 24, 2008; accepted for publication January 20, 2009.