From Evaluated Models to Evaluation Aids: A Multi-Evidence Study of LLM-Based Difficulty Calibration for Programming Examinations
Hongfei Yan, Jiangkai Xiong, Yiqing Li, Chong Chen
Why It Matters
What makes this one worth your time
Understanding and ensuring the fairness of exam difficulty is crucial for educational assessment, and leveraging LLMs could provide a novel approach to achieving this.
The study explores LLMs as tools for calibrating programming exam difficulty, linking AI and student performance data.
Summary
The paper investigates the use of large language models (LLMs) as auxiliary tools for evaluating the difficulty of programming examinations, correlating AI performance with student outcomes and other metrics to assess exam fairness and quality over time.
Key contributions
- Demonstrates the correlation between AI and student performance in programming exams.
- Proposes using LLMs as auxiliary evidence sources for interpreting exam difficulty.
- Provides a methodology for combining AI evidence with other data sources to assess exam fairness.
Notable insights
- AI pass rates and solving-based difficulty indices correlate with student performance, suggesting LLMs can provide valuable insights into exam difficulty.
- The study highlights the potential of using AI as an external reference for problem validation and fairness discussions in educational settings.
Possible limitations
- Model-identity boundary and review-output instability limit the use of AI difficulty scales for individual evaluation.
- Not stated in the abstract
Abstract
arXiv:2608.07523v1 Announce Type: cross Abstract: Difficulty differences across parallel-class programming examinations affect the fairness of course assessment. This study repositions large language models from benchmark evaluation targets to auxiliary evidence sources for interpreting exam difficulty, combining AI evidence with aggregated student performance, item exposure, online-judge process data, and teacher interpretation. First, ten models solved an eight-problem final exam synchronously with 120 students: AI pass rate correlated positively with student pass rate (Spearman rho = 0.866, exact p = 0.0119), and a solving-based composite difficulty index correlated negatively with it (rho = -0.905, exact p = 0.0046). A single structured reviewer was then run via auditable API calls on a third-party OpenAI-compatible endpoint whose model label (gpt-5.6-sol) cannot authenticate an official OpenAI upstream model; call metadata and raw responses are archived. Across 79 problems from 11 parallel-class final exams, AI overall difficulty correlated with problem-level pass rate at rho = -0.871 and with non-attempt rate at rho = 0.800; in a 26-problem longitudinal Data Structures and Algorithms B sample, the correlations were -0.829 and 0.883. A 106-problem introductory-course (CS101) sample marks the boundary: the problem-level correlation weakened to rho = -0.552, and the exam-level correlation across 16 exams was near zero, with cohort composition dominating exam-level outcomes. Exposure-discount (0-0.40) and duplicate-problem perturbation tests did not change these directions. AI evidence can thus serve as an external reference for problem validation, parallel-class fairness discussion, and longitudinal quality tracking, while the model-identity boundary, single-reviewer design, and review-output instability set explicit limits: AI difficulty scales must not be used for individual student evaluation or automatic grade adjustment.