How Well Does AI-Generated Feedback Work? Intrinsic and Extrinsic Evaluation Across More Than 20,000 EFL Essay Drafts
Coyne, Steven Galvan-Sosa, Diana Spring, Ryan Shimmei, Machi Zock, Michael Sakaguchi, Keisuke Inui, Kentaro
Coyne, Steven
Galvan-Sosa, Diana
Spring, Ryan
Shimmei, Machi
Zock, Michael
Sakaguchi, Keisuke
Inui, Kentaro
Supervisor
Department
Natural Language Processing
Embargo End Date
Type
Conference proceeding
Date
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
This study examines feedback in English as a Foreign Language (EFL) writing contexts, focusing on written corrective feedback (WCF). Large language models (LLMs) can provide WCF at scale, but aligning them with pedagogical best practices remains an ongoing challenge. WCF meeting criteria like factuality or relevance may still be unsuitable for learning contexts, highlighting the need for extrinsic evaluation based on the learner’s perspective. We deployed WCF systems in a university-level EFL class with nearly 2,000 students, collecting over 20,000 drafts. We evaluated the generated WCF from two perspectives: intrinsic evaluation by experienced English teachers using a rubric, and extrinsic evaluation via student feedback and engagement metrics. Results revealed low alignment between teacher expert ratings and student feedback. These findings suggest that traditional expert evaluation alone may not fully capture WCF’s usability or helpfulness from the learner’s perspective, highlighting the importance of learner-centered evaluation frameworks for AI-based applications in language education.
Citation
S. Coyne, D. Galvan-Sosa, R. Spring, M. Shimmei, M. Zock, K. Sakaguchi , et al., "How Well Does AI-Generated Feedback Work? Intrinsic and Extrinsic Evaluation Across More Than 20,000 EFL Essay Drafts," 2026, pp. 247-253.
Source
Communications in Computer and Information Science
Conference
27th International Conference, AIED 2026
Keywords
39 Education, 47 Language, Communication and Culture, 4703 Language Studies, 4704 Linguistics
Subjects
Source
27th International Conference, AIED 2026
Publisher
Springer Nature