Developmental–Confirmatory Psychometric Evaluation of a Digital Peer-Assessment Microteaching Rubric Using Many-Facet Rasch Measurement

Abstract

Digital peer-assessment systems can scale performance-based evaluation, but psychometric quality depends on how rubric criteria and rating categories function under rater-mediated use. This study used Many-Facet Rasch Measurement (MFRM) to evaluate and refine a microteaching rubric implemented in the Mobile Audience Response System (MOARS) and assess it in a subsequent cohort. In Cohort 1 (2024–2025; 27 preservice teachers, 331 peer rater–performance records), the baseline six-criterion, four-level rubric was examined through four MFRM specifications. The lowest category was used only five times, while the holistic overall-impression criterion showed comparatively low mean-square fit values and conceptual overlap with the analytic criteria. In this context, collapsing the two lowest categories and excluding the holistic criterion yielded a five-criterion, three-category analytical specification with ordered thresholds and performance separation of 2.98 (reliability = 0.90). In Cohort 2 (2025–2026; 18 preservice teachers, 209 peer rater–performance records), the refined rubric, with an integrated Developing level, was administered unchanged. Model 2 showed a connected peer-rating network, ordered and well-used categories, criterion fit within the prespecified range, and performance separation of 3.37 (reliability = 0.92). Rating-scale functioning was more consistent across cohorts than criterion-difficulty ordering, providing context-bound psychometric evidence for scoring processes, internal structure, and local cross-cohort reproducibility.