Pedagogy-Driven Evaluation of Generative AI-Powered Intelligent Tutoring Systems
Maurya, Kaushal Kumar Kochmar, Ekaterina
Maurya, Kaushal Kumar
Kochmar, Ekaterina
Supervisor
Department
Natural Language Processing
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
The interdisciplinary research domain of Artificial Intelligence in Education (AIED) has a long history of developing Intelligent Tutoring Systems (ITSs) by integrating insights from technological advancements, educational theories, and cognitive psychology. The remarkable success of generative AI (GenAI) models has accelerated the development of large language model (LLM)-powered ITSs, which have potential to imitate human-like, pedagogically rich, and cognitively demanding tutoring. However, the progress and impact of these systems remain largely untraceable due to the absence of reliable, universally accepted, and pedagogy-driven evaluation frameworks and benchmarks. Most existing educational dialogue-based ITS evaluations rely on subjective protocols and non-standardized benchmarks, leading to inconsistencies and limited generalizability. In this work, we take a step back from mainstream ITS development and provide comprehensive state-of-the-art evaluation practices, highlighting associated challenges through real-world case studies from careful and caring AIED research. Finally, building on insights from previous interdisciplinary AIED research, we propose three practical, feasible, and theoretically grounded research directions, rooted in learning science principles and aimed at establishing fair, unified, and scalable evaluation methodologies for ITSs.
Citation
K. K. Maurya and E. Kochmar, “Pedagogy-Driven Evaluation of Generative AI-Powered Intelligent Tutoring Systems,” pp. 32–46, 2025, doi: 10.1007/978-3-031-99261-2_3
Source
Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium, Blue Sky, and WideAIED
(AIED 2025)
Conference
International Conference on Artificial Intelligence in Education
Keywords
Subjects
Source
International Conference on Artificial Intelligence in Education
Publisher
Springer Nature