Peer-reviewed article

The double-pass correlation: A reliability metric for subjective judgments in developmental research.

Ryoichi Watanabe, Yusuke Moriguchi · 9 Jul 2026

Abstract

Ensuring reliable psychological experiments with young children is challenging, especially when outcomes depend on subjective judgments that vary across individuals and tasks. The double-pass design, in which identical trials are presented twice to assess within-participant consistency, is useful for evaluating reliability. However, the number of repeated trials required for reliable estimation in developmental samples remains unclear. We reanalyzed the data from 30 children aged 3-6 years and 31 adults who completed two blocks of color pair similarity judgments. In the observed order analysis, adults reached stable correlations with approximately 31 trials under a strict criterion and 14 trials under a lenient criterion, whereas children required approximately 50 and 32 trials, respectively. Children reached ρ = .30 after approximately 49 repetitions, whereas stronger thresholds were less consistent: 83.3% exceeded ρ = .50 and 63.3% exceeded ρ = .70 in the full 81-trial estimates. These findings suggest that approximately 50 repeated trials may be appropriate for young children when feasible, whereas at least 32 repeated trials may be needed as a minimum shortened procedure to obtain reliable results. For adults, approximately 31 trials may suffice. These results provide empirical guidance for designing shorter, developmentally appropriate tasks that maintain reliable interpretability.