Evaluating the effectiveness of ChatGPT in rubric development

Abstract

This mixed-methods study investigates the effectiveness of ChatGPT in supporting rubric development among preservice English language teachers. Thirty participants created two analytic writing rubrics—one independently and one with ChatGPT assistance. Ten expert raters evaluated rubric quality using Many-Facet Rasch Measurement, focusing on clarity, fairness, and comprehensiveness. AI-assisted rubrics were rated significantly higher than those created independently. While raters applied criteria consistently, variation in severity and certain rubric dimensions (e.g., objectiveness) was noted. Qualitative data was obtained from prompts, focus groups, and surveys. Preservice teachers described ChatGPT as a resource that could help refine descriptors, generate ideas, and improve linguistic precision. However, they also reported limitations such as generic content and misalignment with instructional goals, requiring critical review and revision. Effective use of ChatGPT was positioned as a learned practice that involves iterative formulation and refinement of prompts. Integration of quantitative and qualitative results suggests that ChatGPT can enhance rubric quality and assessment literacy when used reflectively, with due consideration to contextual needs and pedagogical alignment. Findings indicate that AI tools can support teacher education when paired with step-by-step training in writing assessment design and opportunities for teachers to reflect on their decisions during rubric development. © 2026 Elsevier Inc.