Assessing the Written Communication Skills of Medical School Graduates

John R. Boulet, Thomas A. Rebbecchi, Elizabeth C. Denton, Danette W. McKinley, Gerald P. Whelan

The ECFMG Clinical Skills Assessment (CSA) was developed to evaluate whether graduates of international medical schools (IMGs) are ready to enter graduate training programs in the United States. The patient note (PN) exercise, conducted after a 15-minute interview with a standardized patient (SP), is specifically used to assess a candidate's ability to summarize and synthesize the data collected. On a yearly basis, approximately 75,000 patient notes are reviewed and scored by physician raters. Recent changes to the PN scoring rubric, combined with enhancements to quality assurance procedures, mandate that additional evidence be provided to support the intended use of PN scores. The purpose of this study was to further investigate the psychometric adequacy of PN scores. Generalizability analyses suggest that while variability in PN ratings can be attributed to the choice of rater, candidate scores are reproducible over the 10-encounter CSA. The relationship of PN scores with other related ability measures and select candidate characteristics provides additional evidence to support the validity of the written exercise.