Can LLMs Reliably Simulate Real Students’ Abilities in Mathematics and Reading Comprehension?
Srivatsa, KV Aditya Maurya, Kaushal Kochmar, Ekaterina
Srivatsa, KV Aditya
Maurya, Kaushal
Kochmar, Ekaterina
Supervisor
Department
Natural Language Processing
Embargo End Date
Type
Conference proceeding
Date
2025
License
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Large Language Models (LLMs) are increasingly used as proxy students in the development of Intelligent Tutoring Systems (ITSs) and in piloting test questions. However, to what extent these proxy students accurately emulate the behavior and characteristics of real students remains an open question. To investigate this, we collected a dataset of 489 items from the National Assessment of Educational Progress (NAEP), covering mathematics and reading comprehension in grades 4, 8, and 12. We then apply an Item Response Theory (IRT) model to position 11 diverse and state-of-the-art LLMs on the same ability scale as real student populations. Our findings reveal that, without guidance, strong general-purpose models consistently outperform the average student at every grade, while weaker or domain-mismatched models may align incidentally. Using grade-enforcement prompts changes models’ performance, but whether they align with the average grade-level student remains highly model- and prompt-specific: no evaluated model–prompt pair fits the bill across subjects and grades, underscoring the need for new training and evaluation strategies. We conclude by providing guidelines for the selection of viable proxies based on our findings. All related code and data have been made available (https://github.com/kvadityasrivatsa/IRT-for-LLMs-as-Students).
Citation
K. A. Srivatsa, K. Maurya, and E. Kochmar, “Can LLMs Reliably Simulate Real Students’ Abilities in Mathematics and Reading Comprehension?,” pp. 988–1001, Aug. 2025, doi: 10.18653/V1/2025.BEA-1.75
Source
Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications
Conference
20th Workshop on Innovative Use of NLP for Building Educational Applications
Keywords
Subjects
Source
20th Workshop on Innovative Use of NLP for Building Educational Applications
Publisher
Association for Computational Linguistics