Loading...
Misconception-Aware Prompting Can Generate Authentic Analytical Multiple-Choice Questions
Shimmei, Machi Uto, Masaki Matsubayashi, Yuichiroh Inui, Kentaro Mallavarapu, Aditi Matsuda, Noboru
Shimmei, Machi
Uto, Masaki
Matsubayashi, Yuichiroh
Inui, Kentaro
Mallavarapu, Aditi
Matsuda, Noboru
Supervisor
Department
Natural Language Processing
Embargo End Date
Type
Journal article
Date
License
http://creativecommons.org/licenses/by/4.0/
Language
English
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Although automatic multiple-choice question (MCQ) generation has been extensively studied, generating MCQs that require deep reasoning remains underexplored. This study aims to develop a framework for automatically constructing Analytical Multiple-Choice Questions (Analytical MCQs) using a pre-trained large language model. Analytical MCQs present sentence-level assertions as choice items and are designed to evaluate students’ higher-order reasoning about complex subject matter. One of the main challenges in this task lies in generating plausible yet incorrect assertions (i.e., distractors) that reflect students’ misunderstandings. To address this challenge, we utilize students’ written responses to open-ended questions, which offer insights into their conceptual understanding of a topic. Building on this foundation, we introduce AnaQuest, a prompting technique that integrates these open-ended responses into the generation of Analytical MCQs. AnaQuest analyzes these responses, identifies common errors, and generates incorrect assertions alongside correct ones. We evaluated the quality of AI-generated assertions by comparing their Item Response Theory (IRT) parameters, collected from exams where the generated questions were used, with those of certified human-crafted assertions. The analysis showed that assertions generated by AnaQuest more closely resembled human-crafted ones in both discrimination power and difficulty level, compared to those generated by a baseline ChatGPT model that does not incorporate student misunderstandings. To our knowledge, this is the first study to demonstrate the importance of using knowledge of students’ misconceptions to generate high-quality Multiple-Choice questions.
Citation
M. Shimmei, M. Uto, Y. Matsubayashi, K. Inui, A. Mallavarapu, N. Matsuda, "Misconception-Aware Prompting Can Generate Authentic Analytical Multiple-Choice Questions," IEEE Access, vol. 14, pp. 53513-53528, 2026, https://doi.org/10.1109/access.2026.3681235.
Source
IEEE Access
Conference
Keywords
46 Information and Computing Sciences, 4602 Artificial Intelligence
Subjects
Source
Publisher
IEEE