Comparability, stability, and reliability of internet-based mental chronometry in domestic and laboratory settings

R. Miller, K. Schmidt, C. Kirschbaum, S. Enge

The internet-based assessment of response time (RT) and error rate (ERR) has recently become a well-validated alternative to traditional laboratory-based assessment, because methodological research has provided evidence for negligible setting- and setup-related differences in RT and ERR measures of central tendency. However, corresponding data on potential differences in the variability of such performance measures are still lacking, to date. Hence, the aim of this study was to conduct internet-based mental chronometry in both poorly standardized domestic and highly standardized laboratory environments and to compare the variabilities of the corresponding performance measures. Using the Millisecond Inquisit4Web software, 127 men and women completed three different RT-based cognitive paradigms (i.e., go/no-go, two-back, and number-letter). Each participant completed all paradigms in two environments (i.e., at home and in the laboratory), with a time lag of seven days and in a counterbalanced order. Mixed-effects modeling was employed to estimate the between-setting variability across a comprehensive set of performance measures, including conventional measures of central tendency (i.e., mean RT and ERR) and further measures characterizing the joint distribution of RT/ERR. The latter measures were estimated using the diffusion model. The results suggested negligible differences between the domestic and laboratory settings. Thus, this study provides novel evidence suggesting that the statistical power of internet-based mental chronometry is commonly not compromised by increased environmental variance. The within- and between-session reliabilities were in a satisfactory range-that is, comparable to performance measures collected offline in laboratory settings. In consequence, our results support the broad applicability, robustness, and cost efficiency of mental chronometry assessment using the internet.