B T Bates, J S Dufek, H P Davis

Many research studies produce results that falsely support a null hypothesis due to a lack of statistical power. The purpose of this research was to demonstrate selected relationships between single subject (SS) and group analyses and the importance of data reliability (trial size) on results. A computer model was developed and used in conjunction with Monte Carlo procedures to study the effects of sample size (subjects and trials), within- and between-subject variability, and subject performance strategies on selected statistical evaluation procedures. The inherent advantages of the approach are control and replication. Selected results are presented in this paper. Group analyses on subjects using similar performance strategies identified 10, 5, and 3 trials for sample sizes of 5, 10, and 20, respectively, as necessary to achieve statistical power values greater than 90% for effect sizes equal to one standard deviation of the condition distribution. SS analyses produced results exhibiting considerably less power than the group results for corresponding trial sizes, indicating how much more difficult it is to detect significant differences using a SS design. These results should be of concern to all investigators especially when interpreting nonsignificant findings.