Constructed-response items are commonly used in educational and psychological testing, and the answers to those items are typically scored by human raters. In the current rater monitoring processes, validity scoring is used to ensure that the scores assigned by raters do not deviate severely from the standards of rating quality. In this article, an adaptive rater monitoring approach that may potentially improve the efficiency of current rater monitoring practice is proposed. Based on the Rasch partial credit model and known development in multidimensional computerized adaptive testing, two essay selection methods-namely, the D-optimal method and the Single Fisher information method-are proposed. These two methods intend to select the most appropriate essays based on what is already known about a rater's performance. Simulation studies, using a simulated essay bank and a cloned real essay bank, show that the proposed adaptive rater monitoring methods can recover rater parameters with much fewer essay questions. Future challenges and potential solutions are discussed in the end.
This research output has an Altmetric Attention Score of 3. This is our high-level measure of the quality and quantity of online attention that it has received. This Attention Score, as well as the ranking and number of research outputs shown below, was calculated when the research output was last mentioned on 07 January 2017.
All research outputs
#12,971,559
of 22,896,955 outputs
Outputs from Applied Psychological Measurement
#174
of 467 outputs
Outputs of similar age
#155,765
of 313,870 outputs
Outputs of similar age from Applied Psychological Measurement
#2
of 20 outputs
Altmetric has tracked 22,896,955 research outputs across all sources so far. This one is in the 42nd percentile – i.e., 42% of other outputs scored the same or lower than it.
So far Altmetric has tracked 467 research outputs from this source. They receive a mean Attention Score of 2.3. This one has gotten more attention than average, scoring higher than 61% of its peers.
Older research outputs will score higher simply because they've had more time to accumulate mentions. To account for age we can compare this Altmetric Attention Score to the 313,870 tracked outputs that were published within six weeks on either side of this one in any source. This one is in the 49th percentile – i.e., 49% of its contemporaries scored the same or lower than it.
We're also able to compare this research output to 20 others from the same source and published within six weeks on either side of this one. This one has done well, scoring higher than 85% of its contemporaries.
