CSTutorBench: Benchmarking Small Language Models as Tutors for Block-Based Programming
H. Chad Lane, Bryson Kageler
Why It Matters
What makes this one worth your time
This research addresses the practical need for effective AI tutors in educational settings, particularly in K-12 environments, where privacy and cost are significant concerns.
CSTutorBench benchmarks small language models for tutoring in block-based programming.
Summary
The paper introduces CSTutorBench, a benchmark designed to evaluate small language models as tutors in block-based programming environments, specifically VEX VR, and presents preliminary findings on model performance based on pedagogical criteria.
Key contributions
- Development of CSTutorBench, a novel benchmark for evaluating small language models in educational contexts.
- Identification of specific pedagogical behaviors where models struggle, such as avoiding answer leakage and engaging with debugging histories.
- Demonstration of improved model performance through targeted prompt revisions based on educational prompt engineering.
Notable insights
- The study highlights that model family and instruction-tuning approaches are more indicative of tutoring quality than parameter count, suggesting a shift in evaluation criteria for educational AI.
- The use of a human-in-the-loop evaluation pipeline for assessing model performance adds a layer of rigor to the benchmarking process.
Possible limitations
- The small number of models evaluated limits the generalizability of the findings.
- Not stated in the abstract.
Abstract
arXiv:2607.05571v1 Announce Type: new Abstract: Large language models are increasingly explored as AI tutors, yet deploying them in K-12 settings raises concerns around privacy, cost, and reliance on proprietary models. Small language models (SLMs) offer a promising alternative, but selecting the right model for a specific educational context remains difficult, particularly when the target domain, such as block-based programming, is largely absent from model training data. We introduce CSTutorBench, a benchmark for evaluating language models as CS tutors in VEX VR, a block-based robotics environment. The benchmark comprises 17 scenario-based questions scored against a pedagogical rubric grounded in established tutoring and feedback research, with a human-in-the-loop LLM-as-judge pipeline for evaluation. Preliminary findings across 11 models (4B-120B parameters) reveal that models perform well on surface-level criteria such as vocabulary and tone but struggle with deeper pedagogical behaviors, particularly avoiding answer leakage and engaging with student debugging histories. In our sample, model family and instruction-tuning approach appear to be better predictors of tutoring quality than parameter count alone, though the small number of models limits the strength of this conclusion. A targeted prompt revision grounded in recent educational prompt engineering research improved scores for 10 of 11 models. These results underscore the value of context-specific, pedagogically grounded benchmarks for SLM selection in educational deployment.