Although automatic multiple-choice question (MCQ) generation has been extensively studied, generating MCQs that require deep reasoning remains underexplored. This study aims to develop a framework for automatically constructing Analytical Multiple-Choice Questions (Analytical MCQs) using a pre-trained large language model. Analytical MCQs present sentence-level assertions as choice items and are designed to evaluate students’ higher-order reasoning about complex subject matter. One of the main challenges in this task lies in generating plausible yet incorrect assertions (i.e., distractors) that reflect students’ misunderstandings. To address this challenge, we utilize students’ written responses to open-ended questions, which offer insights into their conceptual understanding of a topic. Building on this foundation, we introduce AnaQuest, a prompting technique that integrates these open-ended responses into the generation of Analytical MCQs. AnaQuest analyzes these responses, identifies common errors, and generates incorrect assertions alongside correct ones. We evaluated the quality of AI-generated assertions by comparing their Item Response Theory (IRT) parameters, collected from exams where the generated questions were used, with those of certified human-crafted assertions. The analysis showed that assertions generated by AnaQuest more closely resembled human-crafted ones in both discrimination power and difficulty level, compared to those generated by a baseline ChatGPT model that does not incorporate student misunderstandings. To our knowledge, this is the first study to demonstrate the importance of using knowledge of students’ misconceptions to generate high-quality Multiple-Choice questions.
