Automated item evaluation: Predicting item acceptance and rejection using LLM-generated critiques
Hotaka Maeda, Yikai Lu
Why It Matters
What makes this one worth your time
Automating item evaluation can significantly reduce the workload of experts in educational assessment, making the process more efficient and scalable.
This research demonstrates a promising approach to automate the evaluation of educational items using LLM-generated critiques.
Summary
The paper presents a model for automated item evaluation (AIE) that predicts the acceptance or rejection of educational items based on historical data, utilizing both raw item text and critiques generated by a language model.
Key contributions
- Development of a near-comprehensive AIE model for predicting item acceptance and rejection.
- Implementation of a fusion model that combines representations from raw item text and LLM-generated critiques.
- Empirical evaluation demonstrating the model's performance metrics across different item types and subjects.
Notable insights
- The fusion model combining raw item text and LLM-generated critiques outperformed individual models, highlighting the value of integrating diverse data sources.
- The model's performance varied significantly between subjects, indicating that domain-specific factors may influence the effectiveness of automated evaluations.
Possible limitations
- The model struggled with identifying items flagged for bias and sensitivity, particularly in English language arts.
- Potential over-reliance on automated methods for items with fairness concerns, which may necessitate human review.
- Not stated in the abstract.
Abstract
arXiv:2608.06609v1 Announce Type: new Abstract: Automated item evaluation (AIE) refers to the use of computational methods to assess item quality without requiring manual expert review or field testing of the items under evaluation. We aimed to build a near-comprehensive AIE model by predicting item acceptance and rejection from item text using historical rejection data from a large-scale standardized testing program. The dataset contained 52,759 English language arts (ELA) and mathematics items with 34% permanently rejected from future operational use. Rejection reasons included poor psychometric properties, content issues, bias and sensitivity concerns, and non-content issues. We fine-tuned a DeBERTaV3-large classifier on raw item text, a second DeBERTa classifier on Qwen3-generated item critiques, and a fusion model combining representations from both. The fusion model achieved the strongest overall performance (Accuracy = .75, F1 = .64, AUC = .80, Sensitivity = .64, Specificity = .81). Prediction for math (F1 = .73, AUC = .86) was considerably more accurate than ELA (F1 = .51, AUC = .72). Lowering the decision threshold from .5 to .25 raised average sensitivity for ELA and math to .88 and .91, while reducing specificity to .31 and .56, respectively, which may be preferable in automated item generation contexts where generating items is cheaper than evaluating them. Incorporating item critiques alongside raw item text improved performance across most rejection reasons. The model assigned higher rejection probabilities to more difficult items. However, the fusion model struggled to identify items flagged for bias, sensitivity, fairness, or accessibility, especially for ELA. These findings suggest that text-based AIE is feasible in some areas and may offer a practical tool for reducing the burden of manual review and field testing, while also underscoring the importance of human review for items with fairness concerns.